>Processing system 112 may reconstruct acoustic data received from receiver unit 115 into a 2D acoustic image corresponding to an acoustic image plane which intercepts area of interest 10, and subsequently causes display device 116 to display this 2D acoustic image. The reconstructed 2D acoustic image may for example be an ultrasound Brightness-mode “B-mode” image, otherwise known as a “2D mode” image, a “C-mode” image or a Doppler mode image, or indeed any acoustic image.

In various embodiments, processing system 112 may include a processor (e.g., processor 200) which may execute software in one or more modules for performing one or more algorithms or methods as described below with respect to FIGS. 8 and 9.

Of course it is understood that acoustic imaging instrument 110 may include a number of other elements not shown in FIG. 1, for example a power system for receiving power from AC Mains, an input/output port for communications between processor 112 and acoustic probe 300, a communication subsystem for communicating with other eternal devices and systems (e.g., via a wireless, Ethernet and/or Internet connection), etc.

In some embodiments, acoustic imaging instrument 110 also receives an inertial measurement signal from an inertial measurement unit (IMU) included in or associated with acoustic probe 300 as described below with respect to FIG. 3. The inertial measurement signal may indicate an orientation or pose of acoustic probe 300. The inertial measurement unit may include a hardware circuit, a hardware sensor or Microelectromechanical systems (MEMS) device. The inertial measurement circuit may include a processor, such as processing system 200, running software in conjunction with a hardware sensor or MEMS device.

In other embodiments, acoustic imaging instrument 100 does not receive any inertial measurement signal, but may determine a relative orientation or pose of acoustic probe 300 as described in greater detail below, for example with respect to FIG. 9.

FIG. 2 is a block diagram illustrating an example processing system 20 according to embodiments of the disclosure. Processing system 20 may be used to implement one or more processing systems described herein, for example, processing system 112 shown in FIG. 1. FIG. 2 illustrates an example embodiment of a processing system 20 which may be included in an acoustic imaging system (e.g., acoustic imaging system 100) and/or an apparatus (e.g., acoustic imaging instrument 110) for registering and fusing acoustic images of a region of interest (ROI) of a subject obtained in the absence of any tracking devices, with images of the ROI in the subject which were obtained via other imaging modalities such as magnetic resonance imaging (MRI).

Processing system 20 includes a processor 200 connected to external memory 22, including one or more external memory devices, by an external bus 216.

Processor 200 may be any suitable processor type including, but not limited to, a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable array (FPGA) where the FPGA has been programmed to form a processor, a graphical processing unit (GPU), an application specific circuit (ASIC) where the ASIC has been designed to form a processor, or a combination thereof.

Processor 200 may include one or more cores 202. The core 202 may include one or more arithmetic logic units (ALU) 204. In some embodiments, the core 202 may include a floating point logic unit (FPLU) 206 and/or a digital signal processing unit (DSPU) 208 in addition to or instead of the ALU 204.

Processor 200 may include one or more registers 212 communicatively coupled to the core 202. The registers 212 may be implemented using dedicated logic gate circuits (e.g., flip-flops) and/or any memory technology. In some embodiments the registers 212 may be implemented using static memory. The register may provide data, instructions and addresses to the core 202.

In some embodiments, processor 200 may include one or more levels of cache memory 210 communicatively coupled to the core 202. The cache memory 210 may provide computer-readable instructions to the core 202 for execution. The cache memory 210 may provide data for processing by the core 202. In some embodiments, the computer-readable instructions may have been provided to the cache memory 210 by a local memory, for example, local memory attached to the external bus 216. The cache memory 210 may be implemented with any suitable cache memory type, for example, metal-oxide semiconductor (MOS) memory such as static random access memory (SRAM), dynamic random access memory (DRAM), and/or any other suitable memory technology.

Processor 200 may include a controller 214, which may control input to the processor 200 from other processors and/or components included in a system (e.g., user interface 114 shown in FIG. 1) and/or outputs from the processor 200 to other processors and/or components included in the system (e.g., communication interface 118 shown in FIG. 1). Controller 214 may control the data paths in the ALU 204, FPLU 206 and/or DSPU 208. Controller 214 may be implemented as one or more state machines, data paths and/or dedicated control logic. The gates of controller 214 may be implemented as standalone gates, FPGA, ASIC or any other suitable technology.

Registers 212 and the cache 210 may communicate with controller 214 and core 202 via internal connections 220A, 220B, 220C and 220D. Internal connections may implemented as a bus, multiplexor, crossbar switch, and/or any other suitable connection technology.

Inputs and outputs for processor 200 may be provided via external bus 216, which may include one or more conductive lines. External bus 216 may be communicatively coupled to one or more components of processor 200, for example the controller 214, cache 210, and/or register 212.

External bus 216 may be coupled to external memory which includes one or more external memory devices. The external memory devices may include Read Only Memory (ROM) 232. ROM 232 may be a masked ROM, Electronically Programmable Read Only Memory (EPROM) or any other suitable technology. External memory 22 may include Random Access Memory (RAM) 233. RAM 233 may be a static RAM, battery backed up static RAM, Dynamic RAM (DRAM) or any other suitable technology. External memory 22 may include Electrically Erasable Programmable Read Only Memory (EEPROM) 235. External memory 22 may include Flash memory 234. External memory 22 may include a magnetic storage device such as disc 236. In some embodiments, external memory 22 may be included in a system, such as acoustic imaging system 100 shown in FIG. 1.

FIG. 3 illustrates an example embodiment of acoustic probe 300. Acoustic probe 300 may include an array of acoustic transducer elements 322, a beamformer 324, a signal processor 326, and a communication interface 328. In various embodiments, acoustic transducer elements 322 may comprise, for example a two dimensional (2D) array or a linear or one dimensional (1D) array. For example, in some embodiments, transducer elements 322 may comprise piezoelectric elements. In operation, at least some of acoustic transducer elements 322 receive electrical transmit signals from transmit unit 113 of acoustic imaging instrument 110 and convert the electrical transmit signals to acoustic beams to cause the array of acoustic transducer elements 322 to transmit an acoustic probe signal 15 to an area of interest 10. Acoustic probe 300 may insonify an acoustic image plane in area of interest 10 and a relatively small region on either side of the acoustic image plane (i.e., it expands to a shallow field of view).

Also, at least some of acoustic transducer elements 322 of acoustic probe 300 receive acoustic echoes from area of interest 10 in response to acoustic probe signal 15 and convert the received acoustic echoes to one or more electrical signals representing an acoustic image of area of interest 10, in particular a two dimensional (2D) acoustic image. These electrical signals may be processed further by acoustic probe 300 and communicated by a communication interface 328 of acoustic probe 300 to receive unit 115 as one or more acoustic image signals.

300322322

In some embodiments, particularly in the case of an embodiment of acoustic probe 300 and acoustic imaging system 100 which is used in a training phase of a process or method as described in greater detail below for example with respect to FIG. 8, acoustic probe 300 may include or be associated with an inertial measurement unit 321, as described above, or another tracking device for obtaining relative orientation and position information for acoustic probe 300, and the 2D acoustic images obtained by acoustic imaging system 100 via acoustic probe 300 include or have associated therewith pose or tracking information for acoustic probe 300 while the 2D acoustic images are being acquired. In some embodiments, inertial measurement unit 321 may be a separate component not included within acoustic probe 300, but instead connected to or otherwise associated therewith, such as being affixed to or mounted on acoustic probe 300. Inertial measurement units per se are known to those of skill in the art. Inertial measurement unit 321 is configured to provide an inertial measurement signal to acoustic imaging instrument 110 which indicates a current orientation or pose of acoustic probe 300 so that a 3D volume and 3D acoustic image may be constructed from a plurality of 2D acoustic images obtained with different poses of acoustic probe 300.

In other embodiments, particularly in the case of an embodiment of acoustic probe 300 and acoustic imaging system 100 which is used in an application phase of a process or method as described in greater detail below for example with respect to FIG. 9, acoustic probe 300 does not include or utilize any inertial measurement unit 321, and the 2D acoustic images obtained by acoustic imaging system 100 via acoustic probe 300 do not include or have associated therewith any pose or tracking information for acoustic probe 300 while the 2D acoustic images are being acquired.

In some embodiments, one or more inertial measurement signals output by inertial measurement unit 421 may be supplied to communication interface 428 and thence to acoustic imaging instrument 110 where any desired processing may occur. In other embodiments, the one or more inertial measurement signals output by inertial measurement unit 321 may be supplied to signal processor 326 (instead of directly to communications interface 328) which may process the inertial measurement signal(s) as desired and provide processed inertial measurement signal(s) to communication interface 328, and thence to acoustic imaging instrument 110.

Disclosed in greater detail below are arrangements based on acoustic imaging systems such as acoustic imaging system 100 which may be employed in a method of fusing acoustic images, obtained in the absence of any tracking devices or systems, with images obtained via other imaging modalities, such as magnetic resonance imaging, MRI, computed tomography (CT), cone beam computed tomography (CBCT), etc.

In some embodiments, these arrangements include what is referred to herein as a “training framework” and what is referred to herein as an “application framework.”

The training framework may execute a training process, as described in greater detail below, for example with respect to FIG. 8, using an embodiment of acoustic imaging system 100 which includes and utilizes IMU 321 and/or another tracking device or system which allows acoustic imaging system 100 to capture or acquire sets of spatially tracked two dimensional (2D) acoustic images.

The application framework may execute an application process, as described in greater detail below, for example with respect to FIG. 9, using an embodiment of acoustic imaging system 100 which does not include or utilize IMU 321 or another tracking device or system.

In some embodiments, the training framework may be established in a factory or laboratory setting, and training data obtained thereby may be stored on a data storage device, such as any of the external memories discussed above with respect to FIG. 2. Optimized parameters for a convolutional neural network of acoustic imaging system 100, as discussed in greater detail below, may also be stored on a data storage device, such as any of the external memories discussed above with respect to FIG. 2.

In some embodiments, the application framework may be defined in a clinical setting where an embodiment of acoustic imaging system 100 which does not include or utilize IMU 321 or another tracking device or system is used by a physician or clinician to obtain acoustic images of a subject or patient. In various embodiments, the data storage device which stores the optimized parameters for the convolutional neural network may be included in or connected, either directly or via a computer network, including in some embodiments the Internet, to an embodiment of acoustic imaging system 100 which executes the application framework. In some embodiments, optimized parameters for the convolutional neural network may be “hardwired” into the convolutional neural network of acoustic imaging system 100.

Summaries of embodiments of the training framework and the application framework will now be provided, followed by more detailed descriptions thereof.

In some embodiments, the following operations may be performed within the training framework.

(1) Acquisition of spatially tracked acoustic “sweeps” across a region of interest (i.e. sets of spatially tracked two dimensional (2D) acoustic images in a region of interest (ROI) or organ of interest in a subject population. In some embodiments, the number of images (N_image)≥20 and the number of subjects (N_subject)≥20.

(2) Acquisition of a 3D reference image of the same ROI or organ in the same or a similar subject population using another imaging modality, such as CT, MRI, CBCT, etc., which provides an absolute three dimensional reference frame for the 3D reference image.

(3) Reconstruction of a volumetric three dimensional (3D) acoustic image from each sweep of the tracked 2D acoustic images. This reconstruction also yields the pose of each constituent 2D acoustic image, S_i, within the 3D acoustic reconstruction coordinate system, i.e., the transformations T_{2DUS_to_3DUS}for each 2D acoustic frame, i=1 . . . . N_image, which are computed based on the individual tracked poses of each 2D acoustic image S_iin tracking coordinates, and the known pose of the 3D acoustic reconstruction in tracking coordinates.

(4) Segmenting the region of interest (ROI) or organ of interest in the three dimensional (3D) reference images.

(5) Defining a standardized 3D coordinate system based on the segmentations in the 3D reference images, e.g. defined by an origin of the coordinate system at the centroid of the segmentation, and rotation of the coordinate system to align the XY, XZ, and YZ planes with the axial, sagittal and coronal planes of the organ, or alternatively axial, sagittal, and coronal planes defined by the 3D reference imaging modality (i.e. patient orientation).

Although in the example above, it should be understood that the standardized 3D coordinate system could also have an origin at a vessel bifurcation and an axis oriented along one or two vessels; it could also have an origin at a distinguishable anatomical landmark, such as bony structure, etc. Every feature that one can manually or automatically define in the 3D reference image and relate to the acoustic image can be employed to define the standardized 3D coordinate system.

FIG. 4 illustrates an example of a definition of standardized 3D coordinate system 400. Here, a 3D image of an organ 412 is obtained in several (e.g., three) patients, using an 3D imaging modality such as CT or MRI. Organ 412 is then segmented for each patient, providing the segmentations (S₁to S_n, three shown) in the individual reference 3D coordinate systems (EM₁to EM_n) for the 3D reference images. A standard anatomical plane, such as the mid-sagittal plane through the organ segmentation, is identified (described by the dashed line). Note that the center of the segmentation (C_n) and the orientation of the standard plane may differ from one acquisition to another. Likewise the angle of the mid-sagittal plane may differ in each 3D coordinate system.

For each acquisition, an organ-centered reference 3D coordinate system 410, 420, 430 is defined with origin 414 at the center of the segmentation, and the anatomical plane aligned with 2 of the coordinate axes of the standardized 3D coordinate system 400 (here: Y_Stand Z_St). The standard anatomical plane for each acquisition is not necessarily aligned with its organ-centered reference 3D coordinate system 410, 420, 430, although it may be.

(6) Registration of the 3D acoustic reconstruction with the 3D reference image, i.e., determining the transformation from acoustic 3D coordinates to reference 3D coordinates (T_{3DUS_to_MRI}), using methods known in the art.

(7) Training a convolutional neural network (CNN) to predict the 2D acoustic frame positions in the standardized 3D coordinates, by providing to the network input/output pairs (each 2D acoustic image S_ipaired with its actual standardized pose T_iin standardized 3D coordinates) and performing an optimization of the parameters/weights of the CNN until the standardized predicted poses output by the CNN are optimally predicted compared to the actual standardized poses based on the 2D acoustic image input.

FIG. 5 depicts graphically an example of a deep convolutional neural network 500 which may be employed in the training phase and/or application phase of operation as described herein, and in greater detail with respect to FIGS. 8 and 9 below, for an acoustic imaging system such as acoustic imaging system 100. Convolutional neural network 500 processes input data sets 502. Convolutional neural network 600 includes a plurality of intermediate layers 510A, 510B, 510C and 510D. Convolutional neural network 500 may have more than four intermediate layers or fewer. Convolutional neural network 500 may have one or more final fully connected or global average pooling layer 520, followed by a plurality of regression layers 530A and 530B for regression of translational and rotational components of a rigid coordinate system transformation. Convolutional neural network 500 may have one regression layer, two layers or more than two layers. In one exemplary embodiment, the intermediate layers are convolutional layers, including convolution operations, non-linear regularization, batch normalization, and spatial pooling. This rigid transformation might have a vectorial or non-vectorial parametric representation and therefore the number and dimensions of last regression layers may vary. For instance, rotation can be represented as Euler angles, quaternions, matrix, exponential map, and angle-axis, whereas translation can be separated into direction and magnitude.

Convolutional neural network 500 may be trained using a batch-wise approach on the task to regress the rigid transformation given an input 2D ultrasound image.

During training, an input to convolutional neural network 500 is a 2D ultrasound image and the output is a standardized predicted pose of that 2D acoustic image with respect to a standardized 3D coordinate system. The input to the training framework is pairs of (2D acoustic image, actual poses). The optimizer in the training framework modifies the CNN's parameters so that the prediction for each image approximates the corresponding known actual pose in an optimal way (e.g. minimizing the sum of absolute differences of the pose parameters between prediction and actual pose) In operation after training, convolutional neural network 500 takes a currently produced 2D acoustic image of a subject and predicts the rigid transformation to yield a predicted pose for the 2D acoustic image in the standardized 3D coordinate system.

Accordingly, the training framework automatically generates a training dataset of 2D acoustic images of a region or organ of interest, and corresponding actual poses of those 2D acoustic images in the standardized 3D coordinate system using registration transformations between 3D acoustic coordinates and reference 3D coordinates for the 3D reference image provided by image registration methods that are known in the art, such as mutual-information based registration, or landmark-based registration. The training framework then uses the training dataset to train a neural network (e.g., convolutional neural network 500) using the training dataset to optimize the neural network's ability to predict poses for other 2D acoustic images of the region or organ of interest which are obtained without any tracking information.

In some embodiments, the following operations may be performed within the application framework.

(A) Acquisition of a 3D reference image in a subject, using a 3D imaging modality which allows segmentation of the ROI or reference structure (e.g., organ) within the ROI. This may be the same imaging modality as was used to acquire the 3D reference images during training, or another 3D imaging modality (e.g., MRI or CT) which allows segmentation of the ROI or reference structure (e.g., organ) within the ROI.

(B) Segmentation of an ROI or a reference structure (e.g., an organ) in an ROI in the 3D reference image.

(C) Defining a standardized 3D coordinate system in the same way as was done during the training phase as described above with respect to FIG. 4.

(D) Acquisition of at least one 2D acoustic image in the same subject without spatial tracking of the acoustic probe, in or near the same ROI or organ, i.e., within the area covered by at least one of the acoustic sweeps which were made during training.

(E) For each acquired 2D acoustic image, using the trained convolutional neural network, obtained during the training phase, to generate a standardized predicted pose for the 2D acoustic frame in the standardized 3D coordinate system.

(F) Converting the standardized 3D coordinates of each 2D acoustic image to the reference 3D coordinate system specific to the 3D reference image acquired for this subject. This converts each standardized predicted pose, in the standardized 3D coordinate system, to a reference predicted pose in the reference 3D coordinate system.

(G) Using the resulting reference predicted pose of the 2D acoustic image in reference 3D coordinates to co-display the 2D acoustic image(s) and the corresponding multi-planar reconstruction (MPR) from the 3D reference image, or otherwise visualize the spatial pose of the 2D acoustic image relative to the reference image in a clinically beneficial way.

As an example, FIG. 6 illustrates a display of an example freehand two-dimensional acoustic image 610 obtained for an organ, and a corresponding multi-planar reconstruction (MPR) 620 from an magnetic resonance imaging (MRI) scan in the same subject, based on the predicted pose of the 3D acoustic image relative to the MRI image using the training framework and application framework described above. Here, as is well known in the art, freehand imaging refers to acoustic imaging where the operator moves the probe freely without respect to any particular orientation while capturing acoustic images of an area of interest. The contour 612 shows the intersection of the current view plane with the 3D segmentation of the organ of interest that was obtained from the MRI scan.

Various components of systems implementing the training framework and the application framework will now be described in greater detail.

Some embodiments of the training framework utilize a training dataset, a dataset processing controller (DPC), and a neural network training controller. In some embodiments, the DPC and/or the neural network training controller may comprise a processing system such as processing system 20 described above.

The training dataset consists of a collection of spatially tracked 2D acoustic image sweeps over a specific part of the anatomy (e.g., an organ) in a subject population (for example, a population of at least 20 subjects). In some embodiments, the subject population exhibits variations in age, size of the anatomy, pathology, etc. 3D acoustic volumes are reconstructed from 2D acoustic images using methods which are known in the art (e.g., Qing-Hua Huang, et al., “Volume reconstruction of freehand three-dimensional ultrasound using median filters,” ULTRASONICS 48, pp. 182-192, 2008). The acoustic probe (e.g., acoustic probe 300) which is used with an acoustic imaging system (e.g., acoustic imaging system 100) to obtain the spatially tracked 2D acoustic image sweeps can be tracked using one of the position measurement systems known in the art, such as optical tracking devices or systems, electromagnetic (EM) tracking devices or systems, IMU-based tracking, etc., Based on the spatial tracking of the acoustic probe while acquiring the 2D acoustic images, the transformation describing the pose of each 2D acoustic image S_irelative to the reconstructed 3D acoustic volume, T_{2DUS_to_3DUS}, is known.

In addition to the spatially tracked and reconstructed acoustic imaging sweeps, a 3D reference image obtained via a different imaging modality, such as CT, MRI, CBCT, is also acquired in the same subject, and referred to herein as the 3D reference image. Each 3D acoustic volume is spatially registered to its corresponding 3D reference image by methods known in the art, e.g. manually, or semi-automatically as, e.g., implemented on the PHILIPS® URONAV® Fusion Biopsy System, yielding the registration transformation T_{3DUS_to_MRI}.

The dataset processing controller (DPC) is configured to: load a single case from the training dataset, segment the area of interest or organ of interest from the 3D reference images (e.g., CT, MRI, etc.); based on the segmented mask create a mesh using, e.g., a marching cubes algorithm that is known in the art; and based on the mesh define a standardized 3D coordinate system (see FIG. 4, discussed above) by setting the origin of the standardized 3D coordinate system at the centroid of the segmentation (the centroid is an arithmetic mean of all vertices (pϵR³) of the mesh), and setting the orientation of the 3D coordinate axes defined by the axial, coronal and sagittal planes of the organ or region of interest (e.g., identified via principal component analysis of the mesh).

The pose of the standardized 3D coordinate system relative to the reference 3D coordinate system for each reference image is expressed by the transformation T_{MRI_to_MRIreference}(which in some cases may be the identity transform).

Optionally the DPC may preprocess one or more 2D acoustic images, for example by cropping the 2D acoustic image to a relevant rectangular region of interest.

The DPC may also compute the actual pose T_iof each (potentially pre-processed) 2D acoustic image relative to the standardized 3D coordinate system using the equation:

T
i

=

T

2
⁢
DUS
⁢
_
⁢
to
⁢
_
⁢
MRIreference

=

T

MRI
⁢
_
⁢
to
⁢
_
⁢
MRIreference

*

T

3
⁢
DUS
⁢
_
⁢
to
⁢
_
⁢
MRI

*

T

2
⁢
DUS
⁢
_
⁢
to
⁢
_
⁢
3
⁢