Research Projects

This project aims to develop portable measurement tools with in-situ visualization for the construction industry. A future measurement tool will provide a direct augmented reality view of measured properties over the real environment together with instructions as to where and how a certain task can be completed. For example, a metal detection tool should be able to provide direct visual feedback on the location of hidden metallic structures over a live video view of the inspected wall area. Furthermore it can guide a construction engineer to the optimal position for drilling a hole, avoiding any damage to existing structures.

Thus the tools should combine information from several sources to provide interactive and contextaware

guidance: Measurements from built-in sensors; location-aware through online tracking and registration; spatial, semantic information retrieved from a building information system (BIM). At the same time, future tools need to be simple to be used by non-expert users; therefore the system needs to be intuitive and guide users in the correct operation to fulfil their tasks. To accomplish this goal, The project addresses the following challenges:

• Tracking for mobile devices in changing and unknown environments for correct visual overlays. We will investigate the combination of visual online reconstruction methods with range finders and coarse models for absolute registration.
• X-Ray visualization of hidden and abstract information in unknown environments. Here we will investigate automatic approaches that take the environment’s appearance and the virtual information into account to select the best visualization method.
• User guidance based on measured and plan information requires automatic analysis of the spatial arrangements and automatic visualization.

Critical decisions involving a lot of data are rarely made by a single person, but are rather discussed and evaluated by a team of experts. Examples are doctors deciding for treatment of severe illness, emergency services having to react to ongoing crises, or engineers collaborating to make technical decisions concerning expensive products. These activities can be assisted by information visualization tools. However, traditional information visualization rarely considers the collaborative nature of data analysis tasks. The foundation of our research proposal is the extension of a multiple view visualization system to a multi-display environment. Multiple view visualization shows data in different representations and thereby accommodates for different knowledge backgrounds and user preferences. Multi-display environments turn unused wall and table spaces into interactive surfaces using off-the-shelf projection technology and integrate private workstations smoothly into this shared interactive workspace. Our research aim is the design and creation of a co-located collaborative information visualization workspace dealing with two principal challenges: display space management and collaborative interaction techniques. Intelligent display space management adopts information visualizations and placement of views automatically to the physical display properties and supports the users interacting with the environment. Combined with visual linking of related data entities distributed across the environment, it will help to establish a common knowledge ground. Collaborative interaction techniques are required to organize such a rich, but potentially complex environment. We will investigate high-level activity support for typical tasks in shared information workspaces and how users can maintain awareness of each other’s activities. The proposed research benefits from two ongoing projects at Graz University of Technology: Deskotheque delivers the basic technology necessary for collaborative work in multi-display environments, while Caleydo, a visualization project from the biomedical domain, provides an excellent use case, including the necessary experts willing to collaborate in studies. Using these frameworks, we plan to conduct several usability studies, with prototypes of different levels of sophistication. This research is part of the project Caleydo.

The market for mobile media services will expand significantly in the next years. The explosion in the usage of smartphones and the growth of the application store model to sell individual services to smartphone users opens a new and attractive market for developers of simple, useful applications. New revenue streams can be created by in-application one-click purchasing. Aggregation of a camera on Internet-connected smartphones leads to the possibility of having a live video stream of the user's reality augmented by content and services from the Web. Location-based services and augmented reality are seen as potential killer applications of the mobile Internet because users are enabled to access additional information related to where they are, what they are seeing, or what they are doing, as well as instantly purchase related services and content.

For example, instead of just seeing a street poster for a club night and passing by, this new paradigm opens up instant access via the Internet-enabled smartphone to the club‘s location, purchasing an entrance ticket or listening to/buying the DJ mixes. A new co-operation net-work – the Innovation Network for Smart Applications and Media - will bring key Austrian R&D and innovative SMEs together to make real this new paradigm for smart mobile and media applications which we call Smart Reality, and be the first to benefit commercially from it.

SMART VIDENTE focuses on research on the next-generation field information system for utility companies, providing mobile workforces with capabilities for on-site inspection and planning, data capture and as-built surveying. For achieving this aim, handheld Augmented Reality technology is used for on-site modification and surveying of geometric and semantic attributes of geospatial 3D models on the user’s handheld device. The project aims at providing a fully functional handheld Augmented Reality device for utility field workers. To achieve this goal, we require a software solution that can visualize registered three-dimensional underground models in real time. Registration in 3D requires being able to perform accurate global localization and posing tracking in real time without relying on unrealistic assumptions concerning prior scene knowledge. We will address this issue through fusion of vision, inertial and GPS sensors. Visualization requires the rendering of complex 3D models of underground infrastructure in a way that is easily comprehensible and useful to the mobile worker. This requires visualization techniques for geometric as well as non-geometric information from the geo-database, in particular of hidden objects through so-called “X-ray vision”. These visualization techniques need to be adaptive to scene complexity and environmental conditions. The three-dimensional geometry to be shown is not available per default, but must be extracted from a conventional database system and interpreted on-the-fly as a 3D visualization using procedural modeling techniques. We want to support annotation and even surveying tasks in the field, so the system must also allow to write information back to the geo-database. Finally, we will work with three large Austrian infrastructure companies to assess the usability of our solutions.

Die klinische Rechtsmedizin gewann in den letzten Jahren aufgrund einer Sensibilisierung der Öffentlichkeit gegenüber häuslicher und sexueller Gewalt, Gewalt gegenüber Kindern und Verdachtsfällen von medizinischen Behandlungsfehlern stark an Bedeutung. Die forensische Untersuchung von Lebenden ist bis heute jedoch auf eine äussere Besichtigung des Körpers beschränkt.

Das neue Ludwig-Boltzmann-Institut (LBI) für klinisch-forensische Bildgebung hat zum Ziel, Verfahren zur Erfassung von inneren Verletzungsbefunden als Grundlage für forensische Gutachten zu entwickeln. Mittels Computertomographie (CT) und Magnetresonanztomographie (MRT), welche in der Klinik etabliert sind, können zusätzliche, objektiv nachweisbare innere Verletzungsbefunde erhoben werden, die eine verbesserte Einschätzung der ausgeübten Gewalt gegen die untersuchte Person ermöglichen. Die Methoden sind jedoch auf klinische Diagnostik ausgerichtet, während forensisch wichtige Befunde nicht oder nicht optimal dargestellt werden.

Das Institut fuer Maschinelles Sehen und Darstellen kooperiert mit dem LBI zur Entwicklung neuer Methoden der Bildverarbeitung und Computergrafik zum Zwecke der Bildgebung.

The research aim of the project is to provide a system infrastructure to support teams of users in the on-site monitoring of events and analysis of natural resources. The project will introduce the innovative concept of event-driven campaigns using handheld devices, potentially supported by an unmanned aerial vehicle (UAV). Event-driven campaigns provide users the capacity to analyse and predict environmental changes on-site, supporting the process of taking appropriate countermeasures to avoid environmental degradation. During these campaigns, users will be able to setup and retrieve data from mobile sensorstations, the UAV and external sources (such as permanent sensor networks) in order to generate dense information on a small area. The whole sensor network system will gather and store sensor data, and process simulations based on physical process models. Hence, a shared information system fusing heterogeneous data sources will be provided that supports teams of stakeholders to monitor environmental processes on-site, complementing remote monitoring and management. To enrich the data sets from a specific location, additional remotely controlled cameras will be deployed, mounted on sensorstations and below the UAV. Users will be able to analyse the environment using mobile phones and handheld computers, supported by advanced user interface techniques.

The project will improve monitoring and management for environmental scientists, institutions, service providers, engineering companies and municipalities through its strong integration of handhelds and sensor networks. The project will progress well beyond the current state in the art, by dealing with short-term events and detailed analysis of small sites. The analysis of such events is hardly supported by current methods, but has a large impact on environmental degradation. Furthermore, information is made available to citizens by providing mechanisms to access top-level environmental data. Within the project, cutting edge inter-disciplinary research will be performed to develop user-centered solutions. When the data is integrated with analytical tools in a shared information space it will also aid a wide range of managers and planners pursuing more environmentally sensitive solutions to engineering problems. To aid the process, the research is steered by considerable end-user involvement in all its phases.

The use of augmented reality in medicine is an important field, especially in teaching and training of sensitive tasks. To support teaching and training of neonatal cranial sonography, an augmented reality simulator was developed. Physical models of a newborn and an ultrasound probe were tracked and their movements displayed in their virtual representation. The head of the newborn model was augmented with a 3D volume, reconstructed from ultrasound images of a real patient. Reconstructing a 3D volume from irregular source data takes a special focus on positioning the images and the subsequent interpolation. Moving the physical model towards each other, the according slices are generated in realtime.

VIPEM ist ein System zur hypothesengesteuerten Analyse multidimensionaler Datenräume im Gebiet der personalisierten Medizin. Ein multidimensionaler Datenraum, bestehend aus molekularen und klinischen Daten, wird unter gleichzeitiger Anwendung algorithmischer Verfahren und direkter Benutzerinteraktion gefiltert und hierarchisch strukturiert. Ein zentrales Forschungsproblem der personalisierten Medizin ist die Frage, wie die Verknüpfungen zwischen genetischen Variationen und Krankheiten, bzw. dem Ansprechen auf bestimmte Medikamente, gefunden werden können. Dazu gilt es, z.B. Gendaten mit klinischen Daten zu verknüpfen und in Folge spezifische Patientengruppen zu identifizieren. Die großen Datenmengen der molekularen Analyseverfahren (genetische Polymorphismen, Genexpressionsdaten, Proteomics) können nur mehr mit Methoden der Bioinformatik und Statistik bewältigt werden. Aber auch Standardmethoden der Statistik und der Bioinformatik versagen, wenn die Daten sehr inhomogen strukturiert sind dies ist bei den klinischen Daten der Fall und wenn Strukturen in den Daten durch Rauschen bzw. dominante Muster verdeckt werden. VIPEM soll mit Hilfe von Visualierungsmethoden die Struktur in den Datenräumen sichtbar machen und eine interaktive Navigation und Strukturierung sowohl der molekularen, als auch der klinischen Daten erlauben. VIPEM baut auf Grundlagenergebnissen in den Bereichen Informations-Visualisierung und multimodale Benutzerschittstellen auf. Durch eine enge Verknüpfung mehrerer gleichzeitig wirksamer Eingabekanäle und die sofortige Sichtbarkeit der Analyseschritte in der Visualisierung steht dem Experten ein Werkzeug zu interaktiven Erkundung von komplexen Datenräumen zur Verfügung. Als Eingabeparameter für Analysealgorithmen nutzt VIPEM hierbei die menschliche Fähigkeit, komplexe Muster und Zusammenhänge visuell bereits in Ansätzen zu erfassen, und erlaubt dadurch das Freilegen sonst verdeckter Strukturen. VIPEM fokussiert auf die hohe Nachfrage nach visualisierter Analytik im Bereich der Bioinformatik. Der innovative Zugang von VIPEM versteht sich als einmaliges Verkaufsargument, zumal sich mit VIPEM ein viel versprechendes Produkt abzeichnet, welches sicher innerhalb der nächsten zwei bis drei Jahre seinen Stellenwert als verwertbares Produkt am Markt behaupten könnte. Diese Forschungsarbeit wird als Teil des Projekts Caleydo durchgeführt.

Genoptikum is an interactive data exploration system for the visualization of and navigation in molecular and clinical data in the field of personalized medicine. Genoptikum addresses the essential but to date unsolved problem of how to identify connections between genetic variants and their corresponding diseases or the response to certain drugs and treatments, respectively. It is, therefore, necessary to connect gene data and clinical data in order to categorise specific subgroups of patients with certain disease features. The huge amount of data provided by molecular analytical methods (genetic polymorphisms, gene expression data, proteomics) can only be analysed by applying statistical methods and bioinformatics. However, even standard methods of statistics and bioinformatics fail when the data are inhomogeneous as is the case with clinical data and when data structures are obscured by noise and dominant patterns. Genoptikum should make the structure of the data spaces visible by using innovative methods of visualisation based on multiple high resolution displays in combination with data projection technologies. Genoptikum is bases on fundamental results in the fields of visualisation of information and multimodal user interfaces which enable an interactive navigation and structuring of both molecular and clinical data. Through a close link between several input channels, which are simultaneously active, and by immediate visualisation of the steps of the analysis, the expert is provides with a tool for the interactive exploration of complex data spaces. As input parameter for analysis algorithms Genoptikum makes use of the human visual capacity to grasp complex patterns to reveal hidden structures and correlations in large data spaces. This research is part of the project Caleydo.

This Integrated Project will undertake a research programme that has as its major goal the understanding and exploitation of brain mechanisms for the enhancement of presence and interaction in mixed and virtual reality. The project is highly interdisciplinary, combining neuroscience, computer science, psychiatry, psychology, psychophysics, mechanical engineering, philosophy and drama. By presence we mean the propensity of humans to respond to fake stimuli as if they were real, something observed daily in every virtual reality laboratory. Understanding the neural basis of this ‘presence response’ its enhancement and its application is the fundamental object of study within the IP from many different points of view, and including visual, haptic and auditory modalities. The most interesting, challenging and useful mixed environments are social. The types of interaction we plan in mixed reality are those supporting interactions between real people and other remote real people, real people and virtual people, and between virtual people and virtual people. The aim is to make people’s responses real even if the perception of the reality is based on virtual stimuli. The project will carry out fundamental research adopting a neuroscience methodology and theoretical standpoint combined with research in the delivery of presence through multisensory modelling and rendering, and wide area tracking and display systems. A substantial part of the project is concerned with interaction through brain-computer interfaces. The whole will be brought together through a number of applications, in particular a persistent virtual community that represents the project itself, methods for projecting sensations of ownership to virtual representations of self, and the exploitation of neurofeedback for the enhancement of creativity through mixed reality environments.