Project Overview

The liver surgery planning system (LSPS) is addressing obstacles associated with surgical planning of segment oriented liver resections based on CT data. The main goal was to support radiologist's during data preparation and to give surgeons precise information for making optimal decisions. This was achieved by combining the fields of medical image analysis and computer graphics, which allows for innovative problem solutions, especially when user interaction with complex 3D objects is needed¹. Our approach to virtual liver surgery planning (Fig. 1) was presented at the European Congress of Radiology's ``The Matrix'' exhibit and was distinguished with ``magna cum laude'' twice. It is to the best of our knowledge the only one of its kind reported in literature so far. In the next sections we will give an overview of the progress made within this research project.

The Augmented Reality System

The AR hardware setup of the LSPS consists of the following components: stereoscopic see-through head-mounted displays (HMDs), (optical) tracking system, tracked input devices, rendering workstation(s) and tracking workstation. The LSPS is based on a visually coupled AR system, thus it uses head tracking for correct stereoscopic visualization (see Figure 2). Surgeons and/or radiologists wear see-through HMDs that display virtual objects e.g. the liver surface, vessels and tumors, while the surrounding world can still be seen and interacted with. Virtual objects can be observed as if they were real. Surgeons may walk around, take a closer look and directly move objects for instance. A natural way of interaction with virtual objects is provided due to the use of tracked input devices, consisting of a tracked pencil (PEN) and a transparent plexiglass Personal Interaction Panel (PIP) shown in Figure 2. The PEN is equipped with buttons, to trigger input events. The capabilities of the AR environment are utilized throughout the system. One example is the 3D segmentation result inspection/editing (see Figure 4), where 3D visualization and interaction are needed at the same time. By utilizing the AR system, user interaction can be done very effectively and also allows for new problem solutions. An extension of the developed core system for intraoperative applications is possible and provides the advantage of having one platform for planning and support during surgery. The system is also suitable for other purposes in the medical field including telemedicine and educational applications.

System Overview

The different building blocks of the LSPS are shown in Figure 3. The system provides methods for processing/preparation of the input CT data, 3D visualization and resection planning. The output consists of quantitative indices (e.g. volume of liver tumors or segments) and an elaborated plan for the surgical procedure including information like the number of liver segments subject to resection and their spatial arrangement or the volume of the remaining/removed liver tissue. Basically the LSPS splits up into two main components, a medical image analysis and an AR system. The medical image analysis part is responsible for segmentation of liver, vasculature, tumors and for liver segment approximation. The AR system is utilized for visualization and various kinds of user interaction. A detailed description of the individual function blocks follows in the next sections.

Liver and Tumor Segmentation

The goal for liver and tumor segmentation was to develop a robust and highly automated algorithm. A main obstacle here is, that in some cases local borders/edges of the liver to neighboring structures (e.g. between liver and heart) are virtually not present in the image data. In the case of the liver high level shape and appearance knowledge is utilized to tackle this problem. Neighboring anatomical structures are of special interest, since (diseased) livers are subject to a quite substantial variation in 3D shape and gray-value appearance. Thus the following three step approach to liver segmentation has been taken: (I) separate surrounding structures from the liver using shape knowledge, (II) segment the liver tissue mainly based on gray-value information (contrast agent is used) and (III) fix the remaining problems in an interactive segmentation editing step in 3D. The main idea behind the third step is, that high level shape knowledge can be efficiently contributed in an interactive 3D segment refinement/editing stage if needed (see Section 1.6 for more details). For step (I) a new type of 3D Active Appearance Model (AAM) has been developed for diaphragm dome surface segmentation. By segmenting the diaphragm dome in 3D, liver and heart can be separated and less segmentation editing is needed in this region prone to partial volume effects. An algorithm for automated model initialization is in preparation and will be reported in the near future. The second step consists of preprocessing and a histogram based segmentation method. Different filtering methods based on wavelets, bilateral filtering and the Mean Shift (MS) algorithm have been developed. The mean shift algorithm can also be used for a direct segmentation and thus to replace the subsequent segmentation process. Therefore several extensions based on different approximation schemes have been developed in order to reduce the runtime of this algorithm. For tumor segmentation steps (II) and (III) of the liver segmentation approach are applied with different parameter settings.

Portal Vein Segmentation and Liver Segment Partitioning

The applicability of the filters as a preprocessing step for vessel segmentation has also been studied and showed considerable improvements compared to standard filters (e.g. 3D median filter). However, some smaller vessel branches are still missing in the segmentation result due to partial volume effects. These smaller branches are of importance for an accurate liver segment approximation. Therefore research in this area focused on model based filter approaches that offer a higher degree of robustness. For segmentation a fuzzy connectivity and an adaptive region growing approach have been investigated. Experiments with segment partitioning using a fast skeletonization algorithm are in progress and results will be published by the end of 2003.

Visual Inspection

Once the initial segmentation is available, the radiologist may load the data sets into the AR environment for visual exploration and evaluation. To do so, the segmented objects are converted to surface representations using surface reconstruction techniques, which have to meet the constraints on complexity imposed by the AR system, while still delivering high perceptual quality. To fulfill these requirements a two stage surface reconstruction method has been developed, which first produces a rigid mesh and then converts this mesh into a deformable simplex mesh. Surface reconstruction is done for the most important structures only, while context information is displayed using direct volume rendering. The radiologist may observe the organ from different viewpoints and distances by walking around it or directly moving it using the tracked input devices (see Figure 4). In addition the transparency of the objects can be altered in order to make topological relations more understandable (see Figure 4(b)). Another key feature is the ability to move a tracked panel showing original CT data through the scan volume (see Figure 4(c)). It enables a highly intuitive way for visual evaluation of the input segmentation. Clipping the object slightly above the tracked panel, which is also possible, allows for more accurate evaluation at object boundaries (see Figure 4). Besides surface rendering, LSPS also includes hardware-accelerated volume rendering for visualizing context information (see Figure 4(d)).

Interactive Segmentation Refinement

The task of interactive segmentation refinement is closely related to the visual inspection. Instead of just evaluating the segmentation, interaction with tracked input devices is used to manipulate the surface representation of the segmented objects, if needed. The developed liver surgery planning system provides various tools for interactive true 3D editing of the surface representation ranging from generic, mesh based methods to others taking higher level shape information into account (see Figure 5). The results of single deformation steps can be visualized throughout the editing process moving the CT data textured PIP to locations of interest. It is possible to place snapshots of arbitrarily oriented planes cutting through the liver tissue in space (see Figure 8), to keep track of the outcome of surface editing (see Figure 5(d)). The deformed surface representations of e.g. the liver surface may be exported to traditional volume data sets at any time using fast voxelization techniques. The interactive use of these tools enables radiologists to correct imperfect segmentations intuitively, requiring only a little amount of time.

Resection Planning

Once the accuracy of all reconstructed liver structures has been approved by the radiologist, resection planning by the surgeon can be performed using the same tools as for visual inspection. If two physicians are wearing HMDs, the data sets can be explored in a collaborative way. In addition to the common interaction tools, measurement tools are provided to quantify e.g. the total liver volume and the volume of individual liver segments or tumors (see Figure 6). Our system provides two different methods for volume calculation. The first one uses a fast voxelization in order to get the object's volume, whereas the second algorithm applies fast geometric operations on the mesh. Besides volume calculation, distance measurements are realized by other tools within the system (see Figure 7). These tools enable surgeons to decide whether it makes sense for the patient to undergo a resection or not. In case a resection is indicated, a resection plan may be elaborated based on information gained from the visualization. Moreover it is possible to determine the volume of the remaining liver tissue. Once the segment(s) subject to resection are found, the surgical operation can be discussed, in order to find the best approach for the patient based on the available data.

Outlook

After the startup phase of the project in the year 2001, the main focus of research was to develop the core components of the LSPS. This goal has been reached and additional validation steps in cooperation with the medical partners are under way. Validation has shown clear benefits for the surgical planning process as well as a high acceptance by physicians so far. Future work will mainly focus on areas which will extend the core functionality of LSPS. However, feedback from physicians will also be used to improve and enhance the methods already developed.

Acknowledgments

This work was funded by the Austrian Science Foundation (FWF) under grant P14897.