450a and 450h and several in-between frames 450b-g. In accordance with one embodiment, a user inputs control points over one or more anatomical structures shown in the key frame 450a and an imaging system (e.g., the ultrasound system of FIG. 1) draws a boundary 452a. The user repeats the sequence for the key frame 450h and the imaging system draws a boundary 452h. The system then generates a set of boundaries 452b-g in the in-between frames 450b-g by interpolating between the boundaries 452a and 452h.

In one embodiment, the user is not manipulating the underlying ultrasound data shown in each of the frames. Rather, the user is inputting a number of data points that define a shape that is separate from the underlying ultrasound data. The ultrasound system 131 determines intermediate shapes between those that are input by the user and uses the input and determined shapes to calculate volumes, surface areas etc.

Referring to FIG. 4B, the user can modify the control points placed over an image frame 450e by inputting additional control points or moving the control points along a boundary 452e′. The ultrasound system therefore computes a new boundary from the modified control points. The ultrasound system then performs a re-interpolation of the boundaries 452b-d for the in-between frame using the boundary 452a and the modified boundary 452e′ as well as the boundaries 452f and 452g. Using the modified boundary 452e′ and the user input boundary 452h, the accuracy of the boundaries in the in-between frames 452b-d, 452f and 452g is increased without user input in the in-between frames.

FIG. 5A is a diagram illustrating the generation of boundaries of anatomical structures in accordance with an embodiment of the disclosed technology. FIG. 5A shows an image set 560 comprising key frames 550a and 550c over which a user has placed control points to define boundaries 552a and 552c that trace a heart wall. As described above in reference to FIGS. 3A-4B, the user inputs one or more control points 554 (numbered in the frames 550a-c as control points 554a-c) over the key frames 550a and 550c. A plurality of segments 556 (e.g., cubic splines) connect adjacent control points 554 in the individual key frames 550a and 550c. The system (e.g., the system 131 of FIG. 1) can automatically generate control points 554b for the in-between frame 550b by interpolation and/or morphing along lines 559 to define the control points for the in-between frames.

As in traditional key frame animation, a bounding pair of frames define the key frames. The interior, or ‘in-between’ frames may only include only a slight modification of the outer frames. For example, a heart wall boundary on the in-between frames may be sufficiently approximated using the information present in the key frames. The in-between frame traces can be morphed based on their proximity to the traces defined by the user. In some embodiments, all the walls of the heart may be successfully traced from control points placed over only a few ‘key’ frames drawn by the user. The traces on the other frames can be morphed or interpolated representations of the boundaries in these key frames. As the user adjusts the control points on the key frames, the control points for some or all of the in-between frames may be automatically adjusted based on the updated information. As discussed above, the user can adjust control points in the in-between frames that he or she deems not lying on or near a boundary of the heart wall. This additional information is then applied to the entire data set to improve the quality of the remaining in-between frames.

FIG. 5B is a diagram illustrating the generation of boundaries of anatomical structures in accordance with another embodiment of the disclosed technology. An image set 561 includes key frames 550d and 550f having corresponding control points placed over the image frame that define boundaries 552d and 552f that represent a heart wall. In the illustrated embodiment, the boundaries 552d and 552f have substantially the same shape but different sizes. Rather than the user inputting a completely new set of control points over an in-between frame, the system can automatically generate control points 554e by interpolating between control points 554d in the key frame 550d and control points 554f in the key frame 550f. In some instances, for example, it is faster to resize or move the entire boundary. The system can copy the control points and connecting splines for the boundary shape from either of the key frames and allow the user to simply enlarge or contract the boundary size for the in-between frame without changing its shape. In some embodiments, the boundary can also be rotated and/or shifted to a new position without changing its size or shape.

FIGS. 6A and 6B illustrate another interpolation method in accordance with an embodiment of the disclosed technology. Referring to FIGS. 6A and 6B together, the system generates a number (e.g., 128, 256, 512) of points 658 along the splines 556 between adjacent control points 554. User interaction is simplified by only presenting and allowing user modification of control points. For example, the original boundary points as entered by the user are control points. The user may enter for example, 3, 4, 5 or any number of control points. The system then generates internal points 658 which are not presented to the user but used for internal calculations. In some embodiments, the indexing of the points 658 can start at the same rotational position along each of the splines 556. For example, the system can begin ordering with the point 658 at a 12 o'clock position (i.e., a 0° and/or vertical position as shown in the diagram) and continue to index additional points 658 in a clockwise and/or counterclockwise direction. The system can then select a matching pair of points from the left key frame 550d and the right key frame 550f (FIG. 6B). For example, from each point set 658 and 658′ select the n'th point (i.e. the 16'th point). As shown in FIG. 6B, the point 658 in frame 550d is matched with a point 658′ in frame 550f Given a coordinate location of each point (horizontal and vertical, x, y, defined in mm), a parametric linear equation can be calculated between these matched points 658 and 658′. Using these equations and the position of the image frame 550e, the system selects a coordinate location for new point 658″ on the in-between frame 550e. In some embodiments, position might be time, or frames. This is repeated for all of the calculated internal point pairs (for example, 128, 256, 512 point pairs). The result is a boundary representation 658″ on frame 550e.

In some embodiments, a linear equation, e.g. y=mx+b, is used to calculate the points on the in-between frame 550e where the dependent variable x is frame position or time. For example, for each of the two spatial parameters of a point (horizontal and vertical position) a linear equation defined as y=mx+b can be used to determine corresponding points for the in-between frame 550e using the values from the key frames 550d and 550f In this equation, y is one of the spatial parameters (e.g., position) and x is the frame position, which can be measured, e.g., in units of frames (or time or position). To derive the physical position of the point for the in-between frame 550e, the linear equation defined can be used by inserting the variable x for the correct frame position. For example, referring to FIG. 6B, if control points are interpolated within a time point from control points 658 to 658′, these points might be described with coordinates x, y, z where x is the horizontal position within the 2D image 550d, y is the vertical position within the 2D image 550d, and z describes the position of the image as acquired by the 3D motor. For example, control point 658 might be x, y, z (5.5 mm, 3.3 mm, 6 mm). Control point 658′ might be (4.0 mm, 5.2 mm, and 8 mm). To interpolate the location of the control point 658″ on frame 550e where the z position of this frame is 7 mm, the linear equations are solved for the two parameters as follows y=mz+b and x=mz+b. Thus the equation for x values is x=−0.75z+10.0 and the equation for y values is y=0.95 z−2.4, both as a function of z. Thus the point 658″ is (4.75 mm, 4.25 mm, 7 mm). When interpolating across time points, then the z axis becomes time and the two points would be described x, y, t where t is in units of time. For example, the two points might be 658 (5.5 mm, 3.3 mm, 6 ms) and 658′ (4.0 mm, 5.2 mm, and 8 ms). The linear equations are solved for the two parameters as follows y=m t+b and x=m t+b. Thus the equation for x values is x−−0.75 t+10.0 and the equation for y values is y=0.95 t−2.4, both as a function of t. Thus, the interpolated control point 658″ has values (4.75 mm, 4.25 mm, 7 ms). The process above repeats for all the point pairs from the key frames 550d and 550f. The result is that the points 658″ for the in-between frame 550e will have a shape defined by the new points as shown in FIG. 6B.

In some embodiments, the internal points along the splines between the user defined control points can be interpolated for the in-between frames instead of using the user defined control points. Each frame may include several hundred points (e.g. 128, 256, 512 etc.) calculated along the splines. In one embodiment, each of the these points is numbered starting from a common position such as the 12 o'clock position. A mathematical line can be determined between all, or fewer than all, of these points on one key frame and the same numbered point on a second key frame to calculate a corresponding point in the in-between frame.

In order to reduce the number of interpolated points that are shown to the user for the in-between frame, the system determines which of the interpolated points corresponds most closely to the user determined control points in the key frames. For example, a user defined control point in one key frame might be closest to point 63 in one key frame while the corresponding user defined control point in the other key frame might be closest to point 75. If the number of control points in key frame 550d does not match the number in key frame 550f, additional control points can be calculated and inserted along at least one of the splines 658 or 658′ such that they each contain the same number of control points. This new control point only exists for internal calculations and not be shown to the user. The same or similar linear interpolation described above can be used to calculate positions for a new control point index for the in-between frame 550e. For example, the linear equation for the system {63, 6 mm} and {75, 8 mm} is solved for in-between frame 550e, which exists at 7 mm. The equation of this system is Index=6 x+27 where x is the position of the frame. Calculating for frame 550e at 7 mm, the index is thus 69. Therefore, interpolated point 69 is selected as a control point to show to the user for the in-between frame. This process is repeated for each pair of control points on key frames 550d and 550f The result is a set of interpolated control points on frame 550e which are presented to the user and can be selected and modified.

In some embodiments, instead of using linear interpolation of points and control point indexes between two frames, cubic interpolation or even quadratic interpolation could be used. In this case, instead of using two bounding frames to solve the system three frames would be used in the case of cubic interpolation and four in the case of quadratic interpolation. Cubic spline interpolation could also be used which would use data from all the frames to generate data for the in-between frames.

Once the interpolated points are computed, they can be plotted on the in-between frame and splines calculated that connect the interpolated control points. The user can move the position of the calculated control points. If the position a control point is moved to better coincide with an anatomical feature, the in-between frame can then be designated as a key frame and the positions of the control points in the frame can be used as a basis for determining the positions of control points in other in-between frames. In some instances, for example, it is faster to resize or move the entire boundary, rather than modify individual control points. The user can enlarge or contract the boundary size for the in-between frame without changing its shape. In some embodiments, the boundary can also be rotated and/or shifted to a new position without changing its size or shape.

FIG. 7 is a flowchart illustrating a process 700 of generating a boundary of one or more anatomical structures (e.g., heart, liver, kidney and/or one or more portions thereof) in one or more 2D ultrasound image frames. In some embodiments, instructions for causing a processor to implement the process 700 can be stored on a memory (e.g., the memory 121 of FIG. 1) and executed by the processor (e.g., the processor 134 of FIG. 1) of an ultrasound imaging system (e.g., the system 131 of FIG. 1). At block 710, the process 700 generates and transmits ultrasound energy (e.g., ultrasound energy having a center frequency greater than about 15 MHz) from an ultrasound transducer probe (e.g., the probe 112 of FIGS. 1 and 2) toward a region of interest (e.g., a heart, liver, kidney) in a subject (e.g., a human or an animal, such as a rat or a mouse). At block 720, the process 700 acquires ultrasound data corresponding to ultrasound echoes received from the subject and uses the acquired ultrasound data to form one or more ultrasound image frames (e.g., the image frames 350 of FIG. 3A). In some embodiments, the process 700 can acquire the image data at discrete positions relative to the region of interest. As described above, for example, in reference to FIG. 2, the process 700 can control a motor (e.g., the motor 180) and move the probe a predetermined incremental distances relative to the region of interest to acquire ultrasound data at a plurality of positions. In other embodiments, however, the process 700 acquires the ultrasound image data from the region of interest in a single data acquisition.

A block 730, the process 700 optionally constructs one or more 3D or 4D images using the image frames acquired at block 720. The process 700 can form a 3D image using a plurality of 2D image frames acquired at predetermined positions relative to the region of interest. The process 700 can also form one or more 4D images using, for example, several 3D images acquired at different portions of the subject cardiac cycle. In one embodiment, the process 700 constructs a 4D image using one or more methods disclosed in the applicant's co-pending application Ser. No. 14/072,755, published as U.S. Patent Publication No. 2014/0128738, and incorporated by reference above. In other embodiments, the process proceeds directly to block 740 without constructing 3D and/or 4D images.

At block 740, the process 700 presents the 2D image frames acquired at block 720 to an operator (e.g., the image set 360 of FIG. 3A). The operator selects one or more 2D image frames as key frames and places one or more points or markers as control points near and/or along a boundary of an anatomical structure in the individual image frames.

At block 750, the process 700 generates a boundary in one or more acquired image frames based on the user input received at block 740. As discussed above in reference to FIGS. 4A and 4B, for example, the user can select two key frames and trace or plot points around an anatomical boundary in the key frames. Splines or other mathematical curves/lines can be computed to connect the control points to trace the perimeter of the anatomical features. The process 700 can interpolate between individual control points in a first key frame and corresponding control points in a second key frame to automatically generate boundaries in one or more in-between image frames that are between the key frames.

At block 760, the process can present the 2D image frames with drawn boundaries to the user via a display for additional editing. As discussed above in reference in FIG. 4B, for example, the process 700 can receive user input for any of the generated boundaries to further increase accuracy of the boundaries. For example, the user can adjust a boundary generated in one of the in-between frames in which the generated boundary does not match up with a boundary of the anatomical structure in the image. Each in-between image that the user manually adjusts becomes a new key frame and therefore can improve the accuracy of the remaining in-between frames.

At block 770, the process 700 can determine a measurement (e.g., a surface area, circumference, volume) of a region defined by the manually-input and generated boundaries in the 2D image frames.

At block 780, the process 700 outputs the determined measurement(s) to the operator.

FIGS. 8A-8E are schematic diagrams illustrating the generation of boundaries of anatomical structures in a 4D image set 860 in accordance with some embodiments of the disclosed technology. In the figures, anatomy boundaries shown in solid lines are input by a user, while those shown in dashed lines are computed. Referring to FIGS. 8A-8E together, a 4D image set includes a plurality of 2D image frames of a region of interest arranged in a grid of rows and columns. The rows A-M represent 2D image slices taken at different positions relative to the region of interest as discussed above, for example, with reference to FIG. 2. The columns TP1-TP9 represent 2D images taken at different time points and/or periods of time. Taken together, the 2D image frames in each column can comprise a 3D image of the region of interest acquired at a particular time TP1-TP9. In some embodiments, the times TP1-TP9 correspond to different portions of the subject's cardiac cycle. TP1 can represent, for example, a first time point in the subject's heart cycle (0 ms time). TP5 can represent, for example, a time point approximately ½ way through the heart cycle. Depending on the heart rate, TP5 might be 50 ms (if the period of the heart cycle is 100 ms, for example). In some aspects, TP9 and TP1 can generally represent the same or similar time point in a cycle as the heart completes one cycle and begins a subsequent cycle. In some aspects, the data obtained at time points at the ¼ point (TP3) and the % point (TP7) are similar.

In the illustrated embodiment, the 4D image set includes 2D image frames acquired at 13 positions and 9 time points. In some embodiments, however, the 4D image set can include 2D image frames acquired at fewer or more positions and/or time points. Moreover, the individual 2D image frames are shown for illustrated purposes as two-dimensional grid. In other embodiments, however, the 2D frames can be presented in any suitable format. For example, in some embodiments, the ultrasound system may present one image or a portion of the images at a time.

Referring now to FIG. 8A, an operator inputs points along a boundary of an anatomical structure in an image frame 850a and 850b. The ultrasound system (e.g., the system 131 of FIG. 1) draws corresponding boundaries 852a and 852b in the image on TP1 on frames 850a (TP1-A) and 850b (TP1-M), respectively. The ultrasound system then generates control points for the in-between frames TP1-B through TP1-L. The operator then modifies the position of any of the control points any in-between frames in accordance with the methods described above such that all the boundaries for this time point (TP1) suitably match the anatomical structure. For example, a user may modify the location of the control points in frame TP1-G to better coincide with the boundary of the anatomy captured in the ultrasound image. In some embodiments, the boundary may not start on TP1-A and finish on TP1-M but start and stop on other positions (e.g., the operator might begin placing control points on the image frame TP1-C and finish on TP1-K depending on the extents of the anatomical structure being traced).

Referring now to FIG. 8B, the system then directs the user to draw completed boundaries for at least two image frames obtained at two additional time points; the ¼ position time point TP3 and the ½ position time point TP5. In other embodiments the ¼ and ½ time points will be different depending on how many total time points were used in performing a scan. The operator completes similar steps as described above to complete the trace for those time points. In some embodiments, the system automatically readjusts the display such the operator is presented only with image frames obtained at the ¼ time point for tracing. Once the image frames at the ¼ time points are complete, the system then moves to the ½ time point and instructs the user to complete the boundaries on at least two image frames obtained at this time point. After this has been completed, the user will have drawn complete boundaries for three time points TP1, TP3 (e.g. the ¼ position) and TP5 (e.g. the ½ position). The system will generate control point locations for the in-between frames based on those time points. For example, referring to