Visual Programming for Hybrid User Interfaces

Proceedings of the 2nd International Workshop on Mixed Reality User Interfaces (MRUI'07)

IEEE Virtual Reality 2007 Conference

This paper presents a novel software system called Thekla that facilitates the integration of 2D and 3D user interfaces into hybrid, distributed mixed reality applications. Combining heterogenous display and interaction devices allows mixing and matching of interaction styles for a convenient user experience that is often absent from pure virtual reality applications. To this aim, Thekla enables an application developer to include standard 2D user interface components based on the popular Qt framework with a convenient visual programming environment, thereby significantly easing the introduction of 2D components into hybrid user interfaces, compared to the effort typically necessary in previous virtual and mixed reality frameworks. Thekla consists of a 2D component which synchronizes Qt-based user interfaces with 3D scene graphs, and a 3D component which translates 3D tracker input into 2D events consumable by desktop applications, allowing to turn any screen or surface into a touch screen. The rapid prototyping of 2D user interfaces is supported by Qt Designer, a professional visual programming tool bundled with Qt. Overall, Thekla provides the necessary software components for the seamless integration of distributed hybrid 2D-3D user interfaces. Several examples including a complex surgery planning application demonstrate Thekla's abilities.

PDF