IV. Use of Machine Learning in Medical Imaging

Various machine learning techniques are being employed towards medical imaging. To date, such techniques have largely been applied towards diagnosis, in particular the detection and identification of a particular disease state, such as a cancerous lesion. However, machine learning techniques, such as supervised learning, may also be applied towards the process of image segmentation.

Supervised learning is useful where an output (such as a segmentation of an image or a label of a segmentation) is available for a training dataset (e.g., a plurality of medical images). For example, a machine learning model may be trained with various collections of three-dimensional image data taken from CT scans and/or MRIs, and in these images, particular collections of voxels may be prelabeled as parenchyma, arteries, veins, fissures, airways, lesions, or lymph nodes. Using this training data, a machine learning model is trained to receive unlabeled and optionally unsegmented image data as an input and provide a segmented, labeled data set as an output. The output may identify collections of voxels that are labeled as parenchyma, arteries, veins, lung surface, fissures, airways, lesions, lymph nodes, or any combination thereof. In another approach, a machine learning model is trained with various collections of three-dimensional image data containing labelled collections of voxels (e.g., parenchyma, arteries, veins, lung surface, fissures, airways, lesions, and/or lymph nodes) and segment boundaries for lung images. Using this training data, a machine learning model is trained to receive labeled image data as inputs and produce fully or substantially fully defined segment boundaries for the lungs.

Examples of machine learning algorithms that are supervised include but are not limited to linear regression, logistic regression, decision trees, support vector machine (SVM), naive Bayes, k-nearest neighbors, and neural networks (multilayer perceptron). Reinforcement learning falls between supervised and unsupervised learning, where some feedback is available for each predictive step or action but there is no precise label. Rather than being presented with correct input/output pairs as in supervised learning, a given input is mapped to a reward function that an agent is trying to maximize. An example of a machine learning algorithm that is reinforcement-based includes a Markov Decision Process. Other types of learning that may fall into the one or more of the categories described above include, for example, deep learning and artificial neural networks (e.g., convolutional neural networks).

Various training tools or frameworks may exist to train the machine learning model. Examples of proprietary training tools include but are not limited to Amazon Machine Learning, Microsoft Azure Machine Learning Studio, DistBelief, Microsoft Cognitive Toolkit. Examples of open source training tools include but are not limited to Apache Singa, Caffe, H2O, PyTorch, MLPACK, Google TensorFlow, Torch, and Accord.Net.

The trained machine learning model may take one of several forms. In some implementations, the trained machine learning model is a classification and regression tree or a random forest tree. In some implementations, the trained machine learning model is an artificial neural network, such as a convolutional neural network. In some implementations, the trained machine learning model is a linear classifier, such as a linear regression, logistic regression, or support vector machine.

V. Image Analysis System

This disclosure relates to systems and methods that generate, from one or more three-dimensional (3D) digital images of a lung, a representation of the boundaries of the segments within that lung. Once defined, the boundaries can also be represented visually or used to calculate volume measurements (e.g. the volume of a segment), distance measurements (e.g. the maximum diameter of the segment, distances between a lesion or other structure and one or more segment boundaries), or other analyses based on the three-dimensional representation of the segment's boundaries.

The types of medical images that may be used to generate the 3D digital image include any relevant medical image, including, but not limited to, a computed tomography scan (commonly referred to as a CT scan), a magnetic resonance imaging scan (commonly referred to as an MRI), a nuclear medicine image including, but not limited to, a positron emission tomography scan (commonly referred to as a PET scan), and/or an ultrasound image, and a three-dimensional reconstruction of any of the foregoing.

VI. Terminology

As used herein, the term “surgery” refers to a procedure for treating injuries, disorders, and other medical conditions by incision and associated manipulation, particularly with surgical instruments. The terms “operation,” “surgical operation,” and “surgery” are used throughout this disclosure. Unless otherwise clear from context, the terms are used interchangeably.

As used herein, an “image” refers to a visible depiction or other representation of an item such as the chest area of a patient. In various embodiments presented herein, images provide representations of morphologies and/or compositions of tissue, bone, or organs in a subject. Such images are sometimes referred to herein as medical images. Medical images of varying modality include without limitation representations of tissue, bone, and/or organs derived from computerized tomography (CT), magnetic resonance imaging (MM), x-ray imaging, positron emission tomography (PET) and other nuclear medicine scans, ultrasound imaging, and two-dimensional and/or three-dimensional reconstructions of any of the foregoing. An image may be composed of and/or transformed to three-dimensional image data formed of voxels.

As used herein, a “lung” refers to a single lung organ as present in any vertebrate animal.

As used herein, a “pulmonary artery” is a blood vessel that carries deoxygenated blood into the lung.

As used herein, a “pulmonary vein” is a blood vessel that carries oxygenated blood out of the lung.

As used herein, “vasculature” or “blood vessel” may be used interchangeably to refer to either a pulmonary artery or a pulmonary vein.

As used herein, a lung “airway” refers to an anatomical structure that carries air into and out of the lung. Parts of the airway include the trachea, the bronchi, and the bronchioles. Airways as used herein refer both to the “lumen”, or inside space of the airway, which is composed of air, and the tissue wall of airway surrounding the lumen.

As used herein, a “lobe” refers to a lobe within a lung.

As used herein, a “segment” refers to a segment within a lobe.

As used herein, the lung “surface” refers to the exterior surface of a lung.

As used herein, a lung “lesion” refers to any abnormal tissue within the lung, which may be malignant, benign, or of uncertain malignancy. Herein the term lesion may be used interchangeably with the term “mass,” which is typically used in the context of larger lung lesions, and “nodule,” which is typically used in the context of a smaller lung lesion.

As used herein, the lung “parenchyma” refers to lung tissue which is not an airway, a pulmonary artery, or a pulmonary vein.

As used herein, the term “segmentation” refers to the process of identifying a structure of interest within a medical image, e.g., an anatomical structure such as an organ, lesion, or blood vessel. When segmentation occurs in a three-dimensional medical image, a three-dimensional representation of a structure of interest is created, which may be referred to as a “3D reconstruction”.

As used herein, the term “labeling” refers to the process of identifying the nature of a structure that has been segmented and assigning a relevant semantic name to such structure.

As used herein, the term “voxel,” or volumetric pixel, refers to a single, discrete point in a regular grid in three-dimensional space. In medical images, voxels are typically rectangular prisms, and each has an intensity value (corresponding to a value on a grey or multi-colored scale).

As used herein, the term “plane” refers to a surface in a Euclidean or other space. The surface has no volume. It need not be “flat” as with a Euclidean plane.

VII. Identification of Anatomical Structures within Medical Images (Excluding Lung Segments)

This section describes various examples of methods to segment and label anatomical structures within medical images of lungs.

Among the relevant structures within medical images of lungs are: lung surfaces, lung parenchyma, pulmonary arteries, pulmonary veins, airways, lung lobes, and lung fissures. The representations of some or all of these structures are inputs to the system and methods described in Section VIII which identify the boundaries between lung segments and which are the subject of the present disclosure.

The airways and pulmonary vasculature of the lung each form a tree-like branching network of tubular structures. One method for segmenting airways and vasculature from three-dimensional medical images is based on the mathematical properties of these tube-like structures. Such method may, in some implementations, merely segment, but not label, tube-like anatomical features. Labeling may be accomplished in a subsequent operation. Methods that analyze images based on these properties include, without limitation, graph-based optimization, Hessian calculations, and flux-based methods. These techniques identify regions of voxels (e.g. a ten by ten by ten cube of voxels within a larger image) where a mathematical operation on that region, such as a Hessian calculation, returns a positive result indicating that the high intensity voxels within that region are in a tubular shape. Examples of such techniques are described by Benmansour 2011, Benmansour 2013, Antiga 2007, Graham 2010, and Helmberger 2014.

Another method to segment airways and pulmonary vasculature is based on a machine learning model, which may also be used to merely segment or segment and label such structures. An appropriate training algorithm, based on any one of several supervised machine learning algorithms, may be used to train a machine learning model using information such as prior segmentations and labelings of airways and vessels. The training algorithm may be used to recognize patterns in the images to accurately identify airways and vessels and/or assign labels to such structures. An example of such technique is described by Ochs 2007.

In some implementations, structures that have been segmented as tubular structures, such as by using graph-based optimization, Hessian calculations, or flux-based methods, may be analyzed and assembled into a tree-like structure that represents an interconnected system of veins, arteries, and/or airways. One approach to such analysis may involve the use of a non-linear integer program to be solved by a non-linear solver. Another approach may utilize a machine learning model. In general, these approaches fit a number of candidate voxels to a model of a tree-like structure. When a non-linear solver is used, such model is represented by non-linear equations; when machine learning is used, the model is the machine learning model which has been trained with examples of tree-like structures. Examples of similar techniques are described by Turetken 2013, Turetken 2016, Payer 2015, and Payer 2016.

Once a tree-like structure has been assembled, a separate technique such as a machine learning model may be used to label the applicable structures comprising such tree-like structure. For example, such technique may receive partially processed image data containing one or more pre-segmented tree-like anatomical structures that could be veins, arteries, and/or airways. The technique then analyzes the tree-like structures and classifies or otherwise labels them as veins, arteries, and/or airways.

Aside from machine learning, another method to label a structures that includes an assembled tree-like structure is to take, as input, a non-linear integer program that represents the segmented artery, vein, and airway structures. That program is analyzed by a non-linear solver to assign labels to each of the segmented structures. An example of such technique is described by Payer 2016.

As an alternative to labeling structures only after they have been segmented, other methods that may be used to segment and label structures during the same process. For example, one technique which is specialized for the segmentation of airways will generate segmented structures that are labeled as airways. Examples of such techniques are described by van Ginneken 2008 and Meng 2018, Pu 2010.

In some embodiments, lung anatomical structures are identified using atlas segmentation, which is a technique that can be used for the segmentation and labeling not only for airways and vessels but also for many other anatomical features such as lung surface. For the remainder of this paragraph, segmentation may refer equally to segmentation only or to segmentation and labelling as part of the same process. In atlas segmentation, an image to be segmented (input) is compared against each image contained in a database of previously segmented images (the atlas). Atlas segmentation will identify the image within the atlas which is most “similar” to the input image. In this instance, “similarity” is based on the degree of transformation required to “register” the input image to the image in the atlas (meaning, align one image to the other). The smaller the degree of transformation required between two images, the more similar those two images are. Multiple similarity metrics are possible, including sum of squared distances, correlation coefficients, and mutual information. Once the most similar atlas image is identified, the nature of the transformation between the atlas image and the input image is determined. Multiple transformation models are possible, including rigid, affine, and deformable transformation models. Once the nature of the transformation is determined, that transformation is applied to the segmentation of the atlas image which results in a segmentation of the input image. An example of such technique is described by Rohlfing 2004.

This paragraph and a few following paragraphs discuss optional techniques for the identification of the lung surface. One method to identify the surface of the lung utilizes atlas segmentation. It compares the input image of the lung to an atlas of previously segmented lung images, determining the most similar atlas lung image, then determining the transformation from the atlas image to the input image, then applying that transformation to the segmentation of the atlas image to generate the segmentation of the input image lung surface.

Another method to identify the surface of the lung is through voxel intensity gradient analysis. This analysis considers voxel intensity gradients within the image, meaning, changes in voxel intensity over a range of contiguous voxels within an image. In CT scans and MM images, lung parenchyma shows up as voxels of low intensity, and the exterior of the lung surface shows up as voxels of higher intensity. A rapidly changing gradient is indicative of a lung surface.

Another method to identify the surface of the lung is to utilize a machine learning model. This machine learning model would be trained with segmentations of the lung surface, along with the images from which those segmentations were generated. It would analyze the voxels within an input image to accurately predict which voxels correspond to the lung surface.

This paragraph and the next describe the identification of lung fissures. One method to identify fissures is through voxel intensity analysis. Fissures can be identified based on gradient analysis, with the analysis identifying voxel intensity changes between parenchyma voxels and fissure voxels, or by identifying voxels that constitute plane-like structures with hessian eigenvalue and eigenvector analysis. An example of a similar technique is described by Lassen 2013.

Another method to identify fissures involves the detection of “planes” (which are three-dimensional surfaces without a volume) through vasculature and/or airway voxel density analysis. Fissures can be represented as planes. One property of fissures is that the density of vasculature and airways is lower around fissures than around parenchyma. One method of finding fissures in lungs is to determine which planes have the lowest density of surrounding voxels which correspond to airways, arteries, and veins. Those planes may be taken to be the representation of the fissures, or may be adjusted in space to align with fissures identified using the gradient-based method described in the preceding paragraph.

This paragraph and the next two discuss the identification of lung parenchyma. One method to identify the lung parenchyma is a subtraction-based method. This method starts by identifying the lung surface and the three-dimensional space occupied within such lung surface. From this three-dimensional space, the three-dimensional space occupied by the vessels, airways and fissures is removed (subtracted). The resulting difference represents a segmentation of the lung parenchyma.

Another method to identify lung parenchyma is via analysis of voxel intensity variation and connectivity, which is the way in which voxels relate to adjacent voxels of similar intensity. This analysis involves identifying voxels that are low in intensity value and which are highly connected to other voxels also low in intensity value. Low intensity voxels associated with airway lumens are then removed.

Another method to identify the lung parenchyma is through Otsu's method of thresholding. Otsu's method determines a single intensity threshold that separates voxels into “foreground” and “background” voxels while minimizing variance between the two groups; in the case of images of lungs, those correspond to low intensity parenchyma voxels within the surface and high intensity voxels exterior to the lung surface. An example of such technique is described by Helen 2011.

The remaining paragraphs within this section discuss the identification of lung lobes. One method to identify lung lobes is to start with (a) the segmented lung parenchyma and/or (b) the identified lung surface, and then subdividing the applicable structure according to identified lung fissures in either or both (a) and (b).

Another method to identify lung lobes employs an algorithm such as a watershed algorithm to pulmonary arteries. The method takes as input a segmentation of the pulmonary arteries. It then identifies each lobar branch of the pulmonary arteries (and the subtree associated with such branch) using the following approach: arterial branches that appear within the segmented pulmonary arteries are matched to common branching variants of lung anatomy contained within predefined abstract representations of arterial anatomy. Alternatively, lobar branches of pulmonary arteries may be identified via a k-means clustering algorithm, where the branches of the tree are partitioned into multiple clusters, and each cluster corresponds to a different lobar branch. Areas identified as artery subtrees are considered “high cost” areas within a “cost image” which is used as an input to a watershed algorithm to determine the planes which are equidistant to the edges of each of the subtrees, or, analogously, which are “lowest cost” by the input cost image. An example of such technique is described by Beucher 1992.

Another method to identify lung lobes is based on the application of a watershed algorithm to pulmonary veins. It takes as input a segmentation of the pulmonary veins. It then identifies each lobar branch of the pulmonary veins (and the subtree associated with such branch) using the following approach: venous branches that appear within the segmented pulmonary veins are matched to common branching variants of lung anatomy contained within predefined abstract representations of venous anatomy. Alternatively, lobar branches of pulmonary veins may be identified via a k-means clustering algorithm, where the branches of the tree are partitioned into multiple clusters, and each cluster corresponds to a different lobar branch. Areas identified as pulmonary veins subtrees are considered “high cost” areas within a “cost image” which is used as an input to a watershed algorithm to determine the planes which are equidistant to the edges of each of the subtrees, or, analogously, which are “lowest cost” by the input cost image.

Another method to identify lung lobes is based on the application of a watershed algorithm to airways. It takes as input a segmentation of the airways. It then identifies each lobar branch of the airways (and the subtree associated with such branch) using the following approach: airway branches that appear within the segmented airways are matched to common branching variants of lung anatomy contained within predefined abstract representations of airway anatomy. Alternatively, lobar branches of airways may be identified via a k-means clustering algorithm, where the branches of the tree are partitioned into multiple clusters, and each cluster corresponds to a different lobar branch. Areas identified as airway subtrees are considered “high cost” areas within a “cost image” which is used as an input to a watershed algorithm to determine the planes which are equidistant to the edges of each of the subtrees, or, analogously, which are “lowest cost” by the input cost image. An example of a technique to determine lobar branches of airways is described by Gu 2012.

Another method to identify lung lobes is based on the application of a watershed algorithm to a combination of at least two of pulmonary arteries, pulmonary veins, and airways. Another method to identify lung lobes is based on any of the methods discussed above in combination with a cost image where areas with fissures are also deemed “high cost”.

Another way to identify the lung lobes is with atlas segmentation. In this case, the atlas contains source medical images (CT, MR, etc.) and/or 3D reconstructions of lungs. The 3D reconstructions may contain the lung surface and may also contain arteries, airways, and/or veins. The input to be compared to the atlas may be a standard medical image (CT, MR, etc.) and/or a 3D reconstruction of such image. The atlas image that is most similar to the input image is determined. Then the transformation from the atlas image to the input image is determined. Then that transformation is applied, in reverse, to the lobe representation of the atlas image to arrive at the lobe representation of the input lung image.

Another method to determine lung lobes is to combine any number of the above methods. Examples of similar techniques are described by Lassen 2013 and Giuliani 2018.

VIII. Identification of Segment Boundaries

This section describes the present disclosure, which is a system and method that enables a computer system to identify and label lung segments from a three-dimensional medical image, and to visually depict such lung segments within a 3D reconstruction.

This paragraph and the next discuss identifying segment boundaries by locating planes that represent the boundaries between adjacent lung segments. One method begins by analyzing a representation of a lobe and its pulmonary artery subtree to identify the segmental branches of such subtree, as in the following approach: arterial branches that appear within the segmented lobar subtree of pulmonary arteries are matched to common branching variants of lung anatomy contained within predefined abstract representations of arterial anatomy. Alternatively, segmental branches of pulmonary arteries may be identified via a k-means clustering algorithm, where the branches of the lobar subtree are partitioned into multiple clusters, and each cluster corresponds to a different segmental branch. Then, after identifying those segmental branches, it determines the lobe's segmental artery subtree. Then, it identifies the planes that are equidistant between each of the arterial subtrees that are contained in adjacent lung segments. Each identified plane represents at least a portion of the boundary between such segments.

Another method begins by analyzing a representation of a lobe and its airway subtree to identify the segmental branches of such subtree as in the following approach: airway branches that appear within the segmented lobar subtree of airways are matched to common branching variants of lung anatomy contained within predefined abstract representations of airway anatomy. Alternatively, segmental branches of airways may be identified via a k-means clustering algorithm, where the branches of the lobar subtree are partitioned into multiple clusters, and each cluster corresponds to a different segmental branch. Then, after identifying those segmental branches, it determines the lobe's segmental airway subtree. Then, it identifies the planes that are equidistant between each of the airway subtrees that are contained in adjacent lung segments. Each identified plane represents at least a portion of the boundary between such segments.

This paragraph and a few subsequent paragraphs discuss identifying segment boundaries by identifying planes based on the location of intersegmental veins. This method analyzes a representation of a lobe and a representation of the lobe's pulmonary vein subtree to identify the intrasegmental and intersegmental veins within that lobe as in the following approach: venous branches that appear within the segmented lobar subtree of pulmonary veins are matched to common branching variants of lung anatomy contained within predefined abstract representations of venous anatomy. Each identified intersegmental vein branch is a line or other path which lies on the plane that defines the boundary between two adjacent segments (the “intersegmental plane”). After identifying this line, the system identifies such plane using, e.g., one of the methods described in the following paragraphs.

The first example method to identify the intersegmental plane is via a watershed algorithm, which assumes that the plane is equidistant to the edges of subtrees of intrasegmental veins, and/or subtrees of pulmonary arteries, and/or subtrees of airways. The plane is then adjusted via surface fitting such that the intersegmental vein will lie on the resulting plane. Examples of surface fitting techniques include least sum of squares fitting, linear and non-linear regression, interpolation, RANSAC, B-Spline surface fitting as described in Liew 2015, and Hough transformations as described by Drost 2015.

The second example method to identify intersegmental planes is via atlas segmentation and adjusted via surface fitting such that the intersegmental veins will lie on the resulting plane.

The third example method to identify intersegmental planes is via by machine learning and adjusting via surface fitting such that the intersegmental veins will lie on the resulting plane.

Another method to generate segment boundaries is via machine learning techniques, directly. In this approach, the machine learning model has been trained with training inputs of three dimensional representations of lungs, which might contain any of: lung surfaces, lung parenchyma, veins, arteries, airways, and fissures, and for which the output is lung segments. This model takes in a three-dimensional representation of a lung without segment boundaries and generates an output that includes segment boundaries.

Another method to generate segment boundaries is with atlas segmentation. Here, the input considered is a three-dimensional representation of a lobe and the atlas constitutes a set of three-dimensional representations of lobes with their corresponding segment boundaries identified. The method compares the input lobe three-dimensional representation to the atlas of lobe three-dimensional representations with segment boundaries identified, determining the most similar lobe three-dimensional representation within the atlas. The transformation from the atlas lobe to the input lobe is determined. The segment boundaries within the atlas lobe have that transformation applied to generate the segment boundaries within the input lobe. In some embodiments, the atlas constitutes a set of three-dimensional representations of a lobe and one or more of airways, arteries, and veins with their corresponding segment boundaries identified.

Another method to generate segment boundaries is by analyzing the density of voxels corresponding to vasculature and airways. Any combination of following lobar subtrees may be considered: lobar subtree of airways, lobar subtree of arteries, lobar subtree of veins. This analysis identifies the planes on the interior of the lung lobe by identifying the airways and/or pulmonary arteries and in particular those areas that have the lowest level of voxel density; a low level of voxel intensity indicates a “channel” through which the plane will run. If considering intersegmental veins, an opposite approach is undertaken; a high level of voxel intensity is indicative of an intersegmental vein, which in turn is indicative of a segment boundary.

If desired, any number of the above-described techniques, such as voxel density analysis, atlas segmentation, machine learning, watershed between subtrees, intersegmental vein location, and surface fitting, may be performed in combination.

IX. Flowcharts of Example Methods of Identifying Lung Features

FIGS. 5A and 5B present a flow chart of one embodiment of a method 500 for determining segment boundaries of a lung. As shown, method 500 initially obtains or receives image data for a lung (operation 502). As indicated, this image data may take the form of voxels or other representation of points in a three-dimensional space. As indicated herein, such data may be obtained from various sources such as x-ray images (e.g., CT scans), nuclear magnetic resonance images, and other medical imaging data such as those obtained by positron emission tomography. In various embodiments, the data includes image information for at least an entire segment of a lung.

As illustrated in an operation 504, the method segments and labels the image data provided in operation 502 to identify those voxels or other points that correspond to the location of pulmonary arteries within a lobe or lung. Also, as illustrated in an operation 506, the method segments and labels the voxel data obtained in operation 502 to identify those voxels that correspond to locations of pulmonary veins. Further, in an operation 508, the method segments and labels the image data obtained in operation 502 to identify voxels corresponding to locations of pulmonary airways or airway tissue within a lobe or lung. Also, in an operation 510, the method segments and labels image data obtained in operation 502 to identify voxels within the three-dimensional space of the lobe or lung where parenchyma is located. Further, in an operation 512, the method segments and labels the image data obtained in operation 502 and identifies those voxels that correspond to locations of fissures in the lung.

To this point, method 500 has analyzed three-dimensional image data of a lobe or lung to identify various anatomical features. In the depicted embodiment, these features are pulmonary arteries, pulmonary veins, pulmonary airways, parenchyma, and lobe or lung fissures. This is not an exhaustive list. The method may analyze the image data to identify one or more other anatomical features such as lymph nodes or lesions. Further, in some implementations, the method does not identify each of the pulmonary arteries, pulmonary veins, pulmonary airways, parenchyma, and lobe or lung fissures. In other words, while the depicted embodiment shows that operations 504 through 512 are performed in the depicted method, this is not required. In some embodiments, any one or more of these anatomical features are segmented and labeled for use in subsequent operations. In certain embodiments, the fissures, the airways, the parenchyma, and the arteries are segmented and labeled, while the veins are not segmented and labeled. In some embodiments, the veins, parenchyma, and airways are segmented and labeled, while the arteries and fissures are not segmented and labeled. In some embodiments, the parenchyma, the fissures, the veins, and the arteries are segmented and labeled, while the airways are not segmented and labeled. In some embodiments, the parenchyma, the fissures, the veins, and the airways are segmented and labeled, while the arteries are not segmented and labeled. In some embodiments, the parenchyma, the airways, the veins, and the arteries are segmented and labeled, while the fissures are not segmented and labeled. In some embodiments, the parenchyma, the arteries, and the airways are segmented and labeled, while the veins and fissures are not segmented and labeled.

In the depicted embodiment, the method next uses one or more of the segmented and labeled anatomical features including the lung fissures, the parenchyma, the pulmonary airways, the pulmonary veins, and/or the pulmonary arteries in order to locate lobe boundaries in the three-dimensional space representing the lung (operation 514). In certain embodiments, the method employs only a subset of the listed operations to identify lobe boundaries. For example, the method may employ only the lung fissures, parenchyma, and arteries. In certain embodiments, and depending on the lung under consideration, operation 514 may be repeated to locate multiple lobe boundaries in the lung under consideration.

Next, in the depicted embodiment, at an operation 516, the method identifies segmental branches and related subtrees of the pulmonary arteries identified in operation 504. Additionally, in the depicted embodiment, at an operation 518, the method identifies segmental branches and related subtrees of the pulmonary arteries identified in operation 508. When the method identifies the branch that leads into a given segment, it can trace the artery network downstream to identify the full tree of an arterial or airway network within a segment. Further, in an operation 520 of the depicted process, the method identifies one or more intersegmental branches of veins within lobes. The intersegmental branches are identified from the veins identified in operation 506.

Next, in an operation 522 of the depicted embodiment, the method uses any one or more of the segmental features identified in operations 516, 518, and 520, optionally along with the parenchyma, to determine at least a portion of the boundaries for a given segment. Note that these operations 516, 518, 520, and 522 are performed within a given lobe, which contains multiple segments. Operation 522 identifies a boundary between two adjacent segments in a lobe. It does not necessarily identify the full boundary of a segment. In various embodiments and depending on how many segments reside in a given lobe, the method repeats operation 522 to generate intersegmental boundaries between multiple segments (sometimes all segments) in a lobe under consideration.

Next, in an operation 524 of the depicted embodiment, the method uses some or all of the intersegmental boundaries identified in operation 522 along with some or all of the lobe boundaries identified in operation 514 to identify the all or substantially all of the lung segment boundaries for at least one segment within a lobe.

Finally, in an operation 526, the method generates a representation of the lung or lobe showing some or all the segment boundaries identified in operation 524, optionally along with lobe boundaries identified in operation 514.

Note that the segment boundaries include interior portions within a lobe interior and surface portions on the lobe surface. In some cases, the method uses the same technique to identify both the interior and surface portions of the segment boundaries. The relevant technique is employed in operation 522 of the depicted embodiment.

FIGS. 6A and 6B present a flow chart of another embodiment of a method 500 for identifying and optionally representing boundaries of one or more lung segments. Some of the operations are similar to those shown in FIGS. 5A and 5B. For example, operation 510 (segmenting and labeling parenchyma) may be implemented using Otsu's thresholding. As another example, the operations 504 and 506 that involve segmenting and labeling voxels corresponding to arteries and veins may be implemented by performing tubularity analysis to identify voxels likely to be vasculature, followed by assembling such voxels (tubular collections of voxels) into subtrees using a non-linear program, and, ultimately, labelling the subtrees as veins and arteries using a non-linear program. As a further example, operation 508, which involves segmenting and labelling voxels corresponding to airways may be implemented using a machine learning model with all image voxels as inputs.

In method 600, lobe boundaries are identified by fitting planes to (1) areas of low airway, vein, and/or artery voxel density, and (2) high fissure voxel density. The method also identifies branches of arteries and arterial subtrees within particular segments of a lobe. The method also identifies branches of airways and airway subtrees within particular segments of a lobe. Still further, the method also identifies intersegmental veins within particular segments of a lobe. In the depicted embodiment, the method uses cost images and a watershed algorithm to identify planes between segmental airways and arterial sub-trees. The voxels of airways and arteries are rendered as high cost voxels.

In the depicted embodiment, the method then fits the planes to intersegmental veins between adjacent segments. This allows the method to identify boundaries between adjacent segments within the same lobe. The method then uses these boundaries and the lobe boundaries to define full lung segment boundaries. Ultimately, the method may generate a representation of the lung or lobe with substantially complete segment boundaries.