EP1090376A1

EP1090376A1 - A method and apparatus for interactively constructing relational geometric objects

Info

Publication number: EP1090376A1
Application number: EP00918857A
Authority: EP
Inventors: Bert L. A. Verdonck; Rutger Nijlunsing
Original assignee: Koninklijke Philips Electronics NV
Current assignee: Koninklijke Philips NV
Priority date: 1999-04-20
Filing date: 2000-04-06
Publication date: 2001-04-11
Also published as: WO2000063844A1; JP2002541950A

Abstract

For in a medical environment interactively constructing relational geometric objects that are each defined in an object category within a finite set of categories, each object gets assigned a number of parameters that collectively fix the object in question. In particular, each object gets assigned one or more directional linkings to other such objects in a predefined linking structure. Furthermore, various such objects are geometrically associated with image positions in a medical image that has a first dimension and a second dimension, whilst adjusting parameter values of various such objects to their associated image positions, and updating parameters of other objects linked therewith according to the structure, for after terminating the updating accommodating geometrical information with respect to the medical image.

Description

A method and apparatus for interactively constructing relational geometric objects.

BACKGROUND OF THE INVENTION

The invention relates to a system as recited in the preamble of Claim 1. Modern medicine is ever more frequently depending on the expert handling of medical images that may be produced with a variety of technologies, such as single-shot and multiple shot X-Ray, computer tomography, magnetic resonance, ultrasound, and others. Subsequent medical procedures that are based on those images need precise and detailed descriptions of the various objects represented in the image or images, for deciding, administering, and sizing a particular treatment. The inventors of the present invention have recognized that interrelating such objects in a systematically organized structure would allow better ease-of-handling and improved reliability. This structure would to some extent mirror the relationships of the imaged elements in vivo.

SUMMARY TO THE INVENTION

In consequence, amongst other things, it is an object of the present invention to provide for a structured handling of various geometric objects in the image or images, in such manner that a certain geometrical consistency is maintained, and which handling may be effected by a user person, or by a protocol or wizard. Now therefore, according to one of its aspects the invention is characterized according to the characterizing part of Claim 1.

The invention also relates to a device arranged for practicing the method. Further advantageous aspects of the invention are recited in dependent Claims.

BRIEF DESCRIPTION OF THE DRAWING

These and further aspects and advantages of the invention will be discussed more in detail hereinafter with reference to the disclosure of preferred embodiments, and in particular with reference to the appended Figures that show:

Figure 1 , a device for practicing the invention;

Figure 2, hip prosthesis planning;

Figure 3, ditto after selecting various objects;

Figure 4, indication of vertebral landmarks; Figure 5, length and angle measurements from calculated vertebral corner points;

Figure 6, an exemplary object structure;

Figures 7a-7c, the interrelating of image elements in a temporal structure.

BACKGROUND DISCUSSION OF HANDLING MEDICAL IMAGES

In an image, geometrical objects will often relate to other geometrical objects. Conventionally, this relationship is not enforced by the graphical representation itself, and a change made to one object would therefore often necessitate many changes in other objects, often to the extent to having to redraw or regenerate the complete graphical construction. The present invention proposes a method to force the relations between the graphical object representations in an electronic and conceptually straightforward manner.

The objects may have two, three, or even more dimensions, where also time could represent a variable. The complete set of objects represents a toolkit, including functions for measurement, analysis, and construction operations. The relations between objects may be purely geometrical, such as that they would be parallel, perpendicular, or concentric with regard to each other. Alternatively, such relations may follow a more complex formalism, such as one that gives rise to fixing or optimizing a distance, as depending on a set of other objects. Still further, the relations between objects may be determined by information in the underlying image data or in statistical models, such as the local image intensity center of gravity, image gradient edges, corners, or other features, statistical shape models, etcetera. The designing of a structure of objects may follow from easy-to-learn user interactions. Alternatively, the protocol may be preprogrammed, such as for executing complex and/or repetitive tasks.

The toolkit contains various tool types, that may be elementary or compound in nature. In the latter case they may be derived from a set of various objects provided with primitive types and/or other derivative types. Each object type has a geometrical representation that may depend on the image type on which the object is superimposed, of on a user's viewpoint, such as the projection direction of a 3D object's view.

The intuitive and interactive user interface allows the creating of new objects, the correcting of object positions, the creating of derivative objects in relation to other objects, and the amending of remaining undetermined object parameters. Depending on the number and types of selected objects, the user sees what new object may be created. Such object then will have a predetermined relation with the selected objects. The remaining degrees of freedom pertaining to this object may then be interactively updated. Implementation may be straightforwardly done in an object oriented design or programming environment. A tool translates into a class, objects translate into instances from that class. The relations between objects are maintained in a graph structure.

By way of example, two fields of application are mentioned. First, the planning of implants on X-ray images, such as for hip and knee prostheses. Next to a number of elementary geometrical objects such as points, lines, and ellipses, a model of a prosthesis template is displayed and coupled to the other objects. Note that the prosthesis generally comes in a range of different sizes. Second, the planning of the saw planes in a knee osteotomy may be done as follows. First, the support structure for the saw is positioned through several adjusting screws. Next, the remaining degree of freedom for the sawing direction is determined. This gives the first sawing plane. Starting from this first sawing plane, the sawing is continued through using various sawing guides.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS Figure 1 shows a device for practicing the invention. Herein, item 20 is a medical object, such as a part of the human body. Item 22 is a medical imaging system, such as an X-ray device that provides a pattern of relative transmittivity. For other imaging technologies such as Magnetic Resonance, the physical background that would produce the image intensity is of course different. Item 24 is an image processing device, that may execute various types of image enhancement other processing. Next, the image is stored in data processing device 26, and displayed on user display 28. Through an appropriate user interface, such as exemplified through keyboard 32, and generally enhanced by mouse-type or other features not shown for simplicity, a user person may select various image positions for associating thereto a geometrical object. This selecting will be retrocoupled to device 26 for subsequent display on item 28. Furthermore, the device 26 may execute various updates of the relations if necessary. The overall geometrical structure may be stored in memory 30 that accommodates the storage of such data and programs as necessary. If applicable, a structure may be stored in memory for repetitive later use.

By way of example, the following objects and relations may apply: Point

Line

Ellipse (special case: circle) Rectangle (special case: square) GeoPath (that is an open/closed sequence of lines or PolyLine, or a sequence of Bezier curves or PolyBezier)

Text (length measurement, angle measurement, area measurement, etc.)

Examples of Relations : Point(s) one point defines a line, with three parameters: direction, start, stop one point defines an ellipse, with three parameters: two radii, and an orientation two points define a line, with two parameters: start, stop three points define a circle more than two points: line with least squares fitting more than three points: circle/ellipse fitting

Ellipse(s) an ellipse has a center point an ellipse has a point on its contour (parameter: orientation) an ellipse has a tangent line on a point of its contour (parameter: orientation) an ellipse defines another ellipse with the same midpoint (three parameters: orientation and radii); in a special case an ellipse defines two circles with the same midpoint and with short/long axis as respective radii two ellipses define a line through both midpoints two ellipses define two lines tangent to both ellipses an ellipse has a text object that represents its shortest radius an ellipse has a text object that represents its longest radius

Line a line defines a line parallel to it (parameter: distance between both lines) a line defines a midpoint, or a rational point, i.e. a point at a fixed ratio with respect to the two generating points of the line two lines define an intersection

Points-Ellipse the line through the point and through the midpoint of the ellipse One point and one ellipse: the intersection point of the line through the point and the midpoint of the ellipse with the ellipse one point and one ellipse: one or two lines through the point, tangent to the ellipse

Point-Line a line parallel to the Line through the Point the projected point of Point on the Line a line orthogonal to the Line through the Point a circle with the Point as midpoint, tangent to the Line

GeoPath a joint on the geoPath a tangent line in a point of the geoPath a normal line to a point of the geoPath a geoPath branching off in a point of this geoPath

In particular, the invention may be practiced on three respective different levels of interaction. First, a user person may produce a structure completely by hand, such as for developing a solution for a new or uncommon situation. Second, the structure can be protocolized on various levels, so that the system will guide the user how to build the structure. Finally, the construction may be fixed and recovered from memory as programmed, or after storage of an initial construction. Then it has only to be adapted to the actual instance.

As a few examples, the following is recited. If the object in question is the midpoint of a line drawn between two other points = objects, the shifting of these points will immediately cause an appropriate shifting of the midpoint. Inversely, the shifting of the midpoint may cause the shifting of the line, such as by keeping its length and direction while changing its location. Note that the structure of the model will govern and quantify the update. For example, the line may be finite or infinite, and an intersection with the infinite line beyond the finite line endpoints may be considered valid or not. In the latter case, the action that moved the intersection beyond the finite line endpoints may be judged invalid, and therefore ineffective, so that the action may be undone.

Figure 2 shows hip prothesis planning, in particular, a part of the various geometrical constructions that are needed for planning hip protheses. Various primitives have been plotted: two base points of the pelvis, the ellipse approximating the right femur head, and four points on the left femur bone contour. From these primitives several depending objects are constructed: a line through the two pelvis base points, the center point of the right femur head ellipse, a line parallel to the pelvis line through the right femur head center, the left femur head ellipse with its center on the parallel line, the midpoints of both femur contour point pairs, and the template of a prosthesis on this line through those midpoints.

Figure 3 shows the same after selecting various geometrical objects. In practical usage, this image is colored, and it illustrates that when an object is selected this will be highlighted, and by way of example thereby gets a yellow color, the objects that directly influence this object, the so-called parent objects, also change their color, such as to red. Alternatively or additionally, also objects that directly depend on the selected object may have their color changed as well, such as to green. Other highlighting techniques would be self- apparent to a skilled art person.

Figure 4 shows indicating of vertebral landmarks, and in particular, a construction for obtaining the four corner points of a vertebral body. Two primitive points are indicated on each of the four vertebral body sides: two endplates and two sides. The point pairs are interconnected through lines and the four crossings of successive lines are calculated. The crossing point on the lower right is selected (large "yellow" diamond): the lines from which this is the crossing point (the corresponding parents) are therefore highlighted as well. Figure 5, shows further to Figure 4 length and angle measurements from calculated vertebral corner points: the length of the upper vertebral end-plate and the angle between the upper and lower endplate.

Figure 6 gives an exemplary object structure. This example can be best read from bottom to top. First, four 2D points are created in blocks 62, 66, 68, 72, and overlayed in the color green, not shown here, on an image that is called tkclmage, symbolized by blocks 50, 54, 56, 60. The coloring itself is symbolized by blocks 64, 70. Further, the midpoints of the lines connecting point pairs are calculated in blocks 52, 58, and give rise to a line that is represented by block 40. Furthermore, two other points are calculated on the line connecting the midpoints with "between -0.5", represented by block 46 and with "between 1.5", represented by block 48. The relevant values are relative coordinates with respect to the two midpoints. In their turn, the latter two points constitute a further line represented by block 44, that is displayed on the image as represented by block 42. In practice, such a graph will continue with further relations and/or objects not shown for clarity.

In Figure 6, all relations have been shown as being one-directional. As recited supra, all or part of them may be defined as bidirectional. In updating the overall set of objects, often a single run may suffice, in particular, when all relations are one-directional only, such as the bottom up run when in Figure 6 one of the four basic points 62, 66, 68, 72 is changed. If bidirectional relationships are present as well, it may be that a bottom-up run is being followed by a top-down run, and then a second bottom-up run. Other patterns with more, or with fewer runs, may be suitable for the particular situation or configuration. Of course, no more will be executed than really necessary.

Regarding to the 2D hip prosthesis planning, the following calibration procedure is described. The planning in fact depends on an accurate determination of the scale factor of the projected objects in the displayed scene from pixels to millimeters. Usually, such a calibration procedure consists of indicating an object with a known size: a line is drawn on this object to present to the system the real length of this line. This will fix the calibration factor for subsequent use for all length measurements. This calibration procedure may be incorporated into the connected graph, so that all length measurements are linked to this calibration factor. The same goes for the size of the implant template which is defined in absolute measures such as millimeters. Changing the calibration at any time will immediately show the effect on all length measurements and on the size of the template. This allows to assess the sensitivity of the outcome on the precision of the calibration.

The above described procedure for hip implant planning is being based on 2D projection images. An extension thereof to 3D images, such as are produced by Computed Tomography is straightforward. Therein, the femur bone may be modeled as a cylinder, the femur head as a sphere or as an ellipsoid, the pelvic base as a plane, etcetera. The axis of the femur may be approximated by a line through the midpoints of two circles, instead of by a line through the midpoints of two lines. Other 2D relations may be generalized directly, such as by a line that runs parallel to a plane and through the midpoint of a sphere or of an ellipsoid. An additional strong advantage of the graph of relational geometric objects is that it allows to study the effect of a change in one of the parameters of one of it's objects directly. Moving one of the objects on the display screen, or changing the setting of a user interface slider that is coupled to an object parameter, will show the effect on all other involved objects in a visual manner. Also quantitative analysis of error propagation can be easily implemented, by adding an error range to each parameter and calculating the effects thereof on all related objects.

Regarding Figures 4 and 5, the previous construction illustrated the indicating of landmarks on a single vertebral body. In principle, this procedure must be repeated on all relevant vertebral bodies, up to 12 thoracal and 5 lumbar vertebrae. Further, the entire set of landmarks from all vertebral bodies may also be created and maintained with the toolkit concept, recited supra. This allows the calculating of additional quantitative parameters over the entire set of objects, such as the overall length of the spinal column, the maximum curvature, or the largest offset of a particular vertebral midpoint with respect to a vertical axis. Since this procedure is time-consuming, some automation is desirable, such as has been disclosed in Patent Application (PHN 17133) assigned to the present assignee in another context. When a limited number of vertebral landmarks have been indicated manually, the remaining landmarks can be efficiently interpolated between the already known data. According to the present invention, all interpolated landmarks will be updated immediately each time an operator person will move one of the landmarks to a new or improved position, until such operator will judge that an optimum will have been attained. This procedure will considerably limit the number of points that need to be indicated manually. This continual update procedure can be easily implemented with the geometrical relational graph concept. The interpolation procedure may be further improved by making use of a statistical model of heights, widths, and relative positions of all vertebral bodies.

Figures 7a-7c show the interrelating of image elements in a temporal structure. For simplicity, a temporal structuring of a one-dimensional quantity has been shown, so that a 2D structure results. In it's most advanced realization, the temporal change of a 3D object will produce a 4D structure, such as produced by a sequence of cardiac MR measurements or ultra- fast CT measurements on the moving heart. Segmentation of the heart wall is accomplished by tracing the inner boundary or endocardium and also the outer boundary or epicardium. Now, manual contour tracing is extremely cumbersome and error-prone, inasmuch as a single session will produce hundreds of contours for a single person measured.

In line with the foregoing, a certain degree of automation may be introduced by following the teachings of the invention presented supra with reference to the interpolation of vertebral body landmarks. One approach is to use each separate contour as a starting point for interpolating and/or extrapolating the contours in all other timeslices and or heartbeat phases.

A concrete implementation as shown in Figures 7a-7c starts from a set of contour points on a regular star-shaped pattern that represents three short-axis slices from different positions or time instants of the heart. The centers of the star pattern and all radii propagate to all other slices and phases in a model-based manner. The entire set of contours is updated continuously according to the principle of the present invention, until a user would judge that the interpolated outline is sufficiently precise. Primary objects have been shown as dark circles, interpolated objects as open circles.

Claims

CLAIMS:

1. A method for in a medical environment interactively constructing relational geometric objects that are each defined in an object category within a finite set of categories, by assigning to each object a number of parameters that collectively fix the object in question, characterized by assigning to each object one or more directional Unkings to other such objects in a predefined Unking structure, geometrically associating various such objects with image positions in a medical image having at least a first dimension and a second dimension, and adjusting parameter values of various such objects to their associated image positions, whilst updating parameter values of other objects linked therewith according to said structure, for after terminating said updating accommodating geometrical information with respect to said medical image.

2. A method as claimed in Claim 1 , for use with a plurality of medical images that collectively define at least a third dimension as well, through assigning various such Unkings across sequenced images in said plurality.

3. A method as claimed in Claim 1, wherein selected Unkings are one-directional and others are bidirectional.

4. A method as claimed in Claim 3, wherein said Unkings divide into a first group that connect all objects in a directed graph, and a second group that run counter to respective Unkings in said first group.

5. A method as claimed in Claim 4, wherein said updating is effected in a limited number of runs with respect to at least one of said first group and said second group.

6. A method as claimed in Claim 1, wherein said adjusting is conditional on a selecting operation with respect to a predetermined object, and said selecting will highlight the predetermined object in a first manner, and other objects connected therewith through selective types of linking in a second, different manner.

7. A method as claimed in Claim 1 , whilst introducing a calibration object into said linking structure.

8. A method as claimed in Claim 1 , wherein a particular sequence of similar objects features corresponding sub-objects, and said updating is performed through interpolating either with respect to corresponding sub-objects across said sequence, and/or with respect to various sub-objects pertaining to a single object.

9. A method as claimed in Claim 1 , wherein at least one of said objects has a predetermined amount of uncertainty introduced in one of it's parameter values, and said uncertainty amount is translated into an uncertainty parameter value of other objects.

10. A method for in a medical environment interactively handling relational geometric objects that are each defined in an object category within a finite set of categories and each have assigned a number of parameters that collectively fix the object in question, and furthermore one or more directional Unkings to other such objects in a predefined linking structure, characterized by geometrically associating various such objects with image positions in a medical image having at least a first dimension and a second dimension, and adjusting parameter values of various such objects to their associated image positions, whilst updating parameter values of other objects linked therewith according to said structure, for after terminating said updating accommodating geometrical information with respect to said medical image.

1 1. A device being arranged for practicing a method as claimed in Claims 1 or 10.