The next tool depicted in FIG. 2 is an airway generation tool 216. The airway generation tool 216 allows the user to determine how many generations of the airways are depicted in the 3D model 202. As will be appreciated, image processing techniques have developed to allow for the identification of the airways throughout the lung tissue. There are up to about 23 generations of airway in the lungs of a human from the trachea to the alveolar sacs. However, while very detailed 3D models can be generated, this detail only adds to the clutter of the 3D model and renders the model less useful to the user as the structures of these multiple generations obscures structures. Thus, the airway generation tool 216 allows the user to limit the depicted generations of the airways to a desired level that provides sufficient detail for the planning of a given procedure. In FIG. 2 the airway generation tool 216 is set to the third generation, and a slider 218 allows for the user to alter the selection as desired.

Both a venous blood vessel generation tool 220 and an arterial blood vessel generation tool 222 are depicted in FIG. 2. As with the airway generation tool 216, the venous blood vessel generation tool 220 and the arterial blood vessel generation tool 222 allow the user to select the level of generations of veins and arteries to depict in the 3D model. Again, by selecting the appropriate level of generation the 3D model 202 may be appropriately decluttered to provide useable information to the user.

While these blood vessel generation tools 220 and 222 and the airway generation tool 216 are described here as being a global number of generations of blood vessels and airways displayed in the 3D model 202, they may also be employed to depict the number of generations distal to a given location or in an identified segment of the 3D model 202. In this manner the clinician can identify a particular branch of an airway or blood vessel and have the 3D model 202 updated to show a certain number of generations beyond an identified point in that airway or blood vessel.

In accordance with the disclosure, a generation algorithm has been developed to further assist in providing useful and clear information to the clinician when viewing 3D models having airways and blood vessels both displayed in the UI 200. Traditionally in a luminal network mapping each bifurcation is treated as creation of a new generation of the luminal network. The result is that a 3D model 202 may have up to 23 generations of, for example, the airways to the alveolar sacs. However, in accordance with one aspect of the disclosure a generation is defined differently by the software application generating the 3D model. The application employs a two-step model. The first step identifies the bifurcation in a luminal network. In a second step, at the bifurcation both subsequent branching lumens are measured and if one of the branching lumens has a diameter that is similar in size to the lumen leading to the bifurcation, that branching lumen segment is considered the same generation as the preceding segment. As an example a branching lumen of “similar size” is one that is at least 50% of the size of the lumen leading to the bifurcation. The result is that a clearer indication of the luminal network from the root lumen is depicted in the 3D model at lower levels of generation. Again, this eliminates much of the clutter in the 3D model providing better actionable data for the clinician.

Additional features of the user interface 200 include a CT slice viewer 226. When selected, as shown in FIG. 2, three CT slice images 228, 230, and 232 are depicted in a side bar of the user interface 200. Each of these CT slice images includes its own slider allowing the user to move alter the image displayed along one of three axes (e.g., axial, coronal, and sagittal) of the patient to view portions of the patient's anatomy. The features identified in the 3D model 202 including airways, venous blood vessels, and arterial blood vessels are also depicted in the CT slice images to provide for greater context in viewing the images. The CT slice images may also be synchronized with the 3D model 202, allowing the user to click on any point in the 3D model 202 and see where that point is located on the CT views. This point will actually be centered in each of the CT slice images 228, 230, and 232. Further, this synchronization allows the user to click on any branch in the 3D model 202 and see where that branch is located on the CT slice images 228, 230, 232. An expand icon 233 in the lower left-hand corner of each CT slice image 228, 230, 232 allows the CT slice image to replace the 3D model 202 in the main display area of the user interface 200.

A hidden tissue feature 234 allows for tissue that is hidden from the viewer in the current view of the 3D model 202 to be displayed in a ghosted or outlined form. Further, toggles 236, and 238 allow for the 3D model 202 to be flipped or rotated.

As described herein there are a variety of tools that are enabled via the UI 200. These tools may be in the form of individual buttons that appear on the UI 200, in a banner associated with the UI 200, or as part of a menu that may appear in the UI 200 when right or left clicking the UI 200 or the 3D model 202. Each of these tools or the buttons associated with this is selectable by a user employing the pointing device to launch features of the application described herein.

Additional features of the user interface 200 include an orientation compass 240. The orientation compass provides for an indication of the orientation of the three primary axes (axial, sagittal, and coronal) with respect to the 3D model. As shown the axes are defined as axial in green, sagittal in red, and coronal in blue. An anchoring tool 241 when selected by the user ties the pointing tool (e.g., mouse or finger on touch screen) to the orientation compass 240. The user then may use a mouse or other pointing tool move the orientation compass 240 to a new location in the 3D model and anchor the 3D model 202 in this location. Upon release of the pointing tool, the new anchor point is established and all future commands to manipulate the 3D model 202 will be centered on this new anchor point. The user may then to drag one of the axes of the orientation compass 240 to alter the display of the 3D models 202 in accordance with the change in orientation of the axis selected.

A related axial tool 242 is can also be used for to change the depicted orientation of the 3D model. As shown axial tool 242 includes 3 axes axial (A), sagittal (S), coronal (C). Though shown with the axes extending just to a common center point the axes extend through to the related dot 244 opposite the dot with the lettering 246. By selecting any of the lettered or unlettered dots the 3D model be rotated automatically to the view along that axis from the orientation of the dot 244 or 246. Alternatively, any of the dots 244, 246 may be selected and dragged and the 3D model 200 will alter its orientation to the corresponding viewpoint of the selected dot. In this way the axial tool 242 can be used in both free rotation and snap modes. FIGS. 16-18 depict the change in position of each of the S, C, A dots 246 are selected by the user, and the related dots 244.

A single axis rotation tool 248 allows for selection of just a single axis of the three axes shown in the orientation compass 240 and by dragging that axis in the single axis rotation tool 248, rotation of the 3D model 202 is achieved about just that single axis. Which is different than the free rotation described above, where rotation of one axis impacts the other two depending on the movements of the pointing device.

A 3D model orientation tool 250 depicts an indication of the orientation of the body of a patient relative to the orientation of the 3D model 202. A reset button 252 enables the user to automatically return the orientation of the 3D model 202 to the expected surgical position with the patient lying on their back.

A zoom indicator 254 indicates the focus of the screen. By default, the inner white rectangle will be the same size as the outer grey rectangle. As the user zooms in on the 3D model 202, the relative size of the white rectangle to the grey indicates the level of zoom. In addition, once zoomed in, the user may select the white rectangle and drag it left or right to pan the view of the 3D model displayed in the user interface 200. The inner white rectangle can also be manipulated to adjust the level of the zoom. The plus and minus tags can also be used to increase or decrease the level of zoom. FIG. 19 depicts the change in the indicator 254 when the view is zoomed in. FIG. 20 depicts the change in the zoom indicator 254 depicting a panning right of the 3D model with the inner white rectangle offset to the right in the grey rectangle.

FIG. 3 depicts a further aspect of control of the user interface 200 and particularly the model 202 displayed therein. As shown in FIG. 3, another way to rotate the 3D model 202 in a specific axis is to move the pointing device to the edges of the screen. When this is accomplished an overlay 302 is depicted showing four rotational cues 304. Selecting one of these rotation cues 304 will cause the 3D model 202 to rotate. Additionally, moving the model (i.e., pan) can also be accomplished in this overlay. Further, the pointing device may be used to identify a new spot on or near the 3D model 202 about which to rotate the 3D model 202.

FIG. 4 depicts further features of the thoracic surgery planning tool. When a user selects the tumor 210, the menu 402 is displayed. As an initial matter, the menu 402 displays the same information as the tumor tool 212. Specifically, the menu 402 may display the dimensions and volume of the tumor. The menu 402 also allows for adjusting the size of the margin around the tumor 210 and elimination of the margin all together.

A crop tool 404 is also provided for in the menu 402. When selected the crop tool defines a region 406 around the tumor 210 as shown in FIG. 5. This region 406 is defined by a series of line segments 408. The user is able to select these line segments 408 to adjust the region 406 around the tumor 210. Once satisfied with the placement of the line segments 408 to define the region 406, the user may select the “crop” button. This removes from the 3D model all tissue that is either not found within the region 406, or is not part of an airway or blood vessel, which passes through the region 406. As with the generation selection tools, described above, the effect of this cropping is that not only are the blood vessels and airways that are within the region 406 displayed so the user can observe them and their relation to the tumor 210, but also displayed are the airways and blood vessels which lead to the airways and blood vessels that are within the region 406.

One of the benefits of this tool is to be able to identify the root branches of the airways and blood vessels leading to the tumor 210. This is made possible by removing all of the clutter caused by the other objects (e.g., airways and blood vessels) of the 3D model that are not related to the cropped region. This allows the user to consider the airways and blood vessels leading to the tumor 210 and determine which segments are implicated by the tumor 210 and which airways and blood vessels might need resection in order to achieve a successful segmentectomy. In this manner the clinician can adjust the size of the margin to identify the relevant blood vessels and airways to minimize the area for resection.

The region 406 may be depicted in the CT image slices 228, 230, 232. Similarly, the tissue that has been cropped from the 3D model may also be cropped in the CT image slices. Further, the tissue that is hidden by the crop selection may not be completely hidden but may be ghosted out to limit the visual interference but leave the clinician able to ascertain where that structure is in the 3D model 202.

FIG. 4 depicts two additional features in the menu 402. One feature is a hide tissue button 408 which when selected hides the tumor 210 and any tissue that is within the margin. Further the anchoring tool 241 is also displayed in the menu allowing selection and placement of the anchor point for the orientation compass 240, as described above.

A second menu 410 may be displayed by the user using the pointing tool to select any location within the 3D model 202. The menu 410 includes a depth slider 412 which is enabled by selecting a button 414 shaped like a palm tree allows the user to change the number of generations related to a tissue at the selected point. This allows for local decluttering around the point selected. Additional features in menu 410 include a clip button 416 which provides an indication of the tissue to be excised in the surgical procedure. By selecting the clip button 416, the user may then use the pointing device to select a location on the 3D model 202. A resection line 418 is drawn on the model at that point and the portions of the 3D model to be resected are presented in a different color. A hide tissue button 420 allows for the selection of tissue using the pointing device and hiding the selected tissue from view to again assist in decluttering the 3D model. A flag button 422 allows for placement of a flag at a location in the 3D model with the pointing device and for the insertion of notes related to that flag.

FIGS. 5 and 6 depict a further aspect of the thoracic surgery planning tool in UI 200. Through any of the manipulations described above where tissue is hidden or cropped or the 3D model 202 is rotated a screenshot may be taken by placing the pointing device on the screen shot icon 424. This may be done many times during the thoracic surgery planning as shown in FIG. 6 with screenshots 426 depicted along the left-hand margin of the UI 200. Each of these screen shots shows some prior manipulation of the 3D model 202. For example, screenshot 1 shows just the airways and the tumor