JP6643416B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program for performing alignment between images.

  2. Description of the Related Art In image diagnosis using medical images (three-dimensional tomographic images representing information inside a subject), a doctor performs diagnosis while comparing images captured by a plurality of imaging devices (modalities), different body positions, times, imaging parameters, and the like. I do. However, since the posture and shape of the subject are different between the images, it is difficult to identify or compare a lesion. Therefore, it has been attempted to perform registration between a plurality of images. As a result, it is possible to generate an image in which one of the images is subjected to the transformation or deformation of the posture and the same as the other image.

  However, the result of general alignment includes an error and is not always accurate. For this reason, there is a problem that the doctor cannot judge how much the result of the alignment is reliable.

  On the other hand, Patent Literature 1 discloses a method for estimating an alignment error based on the instability (ambiguity of a solution) of a deformation parameter estimated as a result of the deformation alignment. In the method of Patent Literature 1, an error of the position is estimated based on a fluctuation range of an estimated position of a corresponding point of a target point when an unstable parameter is intentionally changed.

JP 2013-198722 A

  However, the error estimation method described in Patent Literature 1 has a problem that factors other than instability of the estimation parameter cannot be considered.

  The present invention has been made in view of the above problems, and has as its object to provide a method for acquiring an estimation error and reliability in alignment between a plurality of images (particularly, alignment caused by interpolation). .

An image processing apparatus according to one embodiment of the present invention that achieves the above object has the following configuration. That is,
Information acquisition means for acquiring image deformation information,
Image generation means for generating a deformed image by performing coordinate transformation on the image based on the deformation information,
A resulting unit preparative you get the volume data representing the estimated error or the reliability of deformation in the deformation image,
A display control unit that superimposes a cross-sectional image of the deformed image and a map indicating a cross-section in the volume data corresponding to a cross-section of the cross-sectional image of the deformed image on a display unit;
It is characterized by having.

  ADVANTAGE OF THE INVENTION According to this invention, the estimation error in the position alignment between several images and reliability can be acquired.

FIG. 1 is a diagram illustrating a functional configuration of an image processing system and an image processing apparatus according to a first embodiment. 5 is a flowchart illustrating a processing procedure of the image processing apparatus according to the first embodiment. FIG. 4 is a schematic diagram illustrating an example of an estimation error display according to the first embodiment. FIG. 10 is a schematic diagram illustrating an example of an estimation error display according to the second embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the scope of the present invention is not limited to the illustrated example.

(1st Embodiment)
The image processing apparatus according to the present embodiment is an apparatus that performs deformation alignment between a plurality of three-dimensional tomographic images, and deforms one three-dimensional tomographic image so that the position and shape match the other three-dimensional tomographic image. Generated transformed image. At this time, the first feature is that an estimation error and reliability of the alignment at each position of the generated deformed image are obtained, and the distribution is displayed in association with the deformed image. Hereinafter, the image processing apparatus according to the present embodiment will be described.

<1. Configuration of Image Processing System 1>
FIG. 1 shows a configuration of an image processing system 1 according to the present embodiment. As shown in FIG. 1, the image processing system 1 according to the present embodiment includes an image processing device 10, a data server 20, and a display 30. The image processing apparatus 10 includes an image acquisition unit 110, a correspondence information acquisition unit 120, a setting unit 125, a positioning unit 130, an image generation unit 140, an estimated error acquisition unit 150, and a display control unit 160.

  The image acquisition unit 110 acquires, from the data server 20, a plurality of three-dimensional tomographic images (that is, a first image and a second image) of the subject to be aligned.

  The correspondence information acquisition unit 120 acquires correspondence information in the respective image coordinate systems of the first image and the second image.

  The setting unit 125 sets a condition (first condition) for executing the alignment process (deformation estimation process). Details of the condition setting will be described later. The setting unit 125 sets a second condition different from the first condition based on the first condition.

  The alignment unit 130 performs an alignment process (a deformation estimation process) between the first image and the second image under the conditions set by the setting unit 125, based on the correspondence information acquired by the correspondence information acquisition unit 120. Execute

  The image generation unit 140 generates a new image (deformed image) obtained by performing coordinate transformation on the first image so as to match the second image based on the alignment result obtained by the alignment unit 130. .

  The estimation error acquisition unit 150 controls the image generation unit 140 based on the alignment result obtained under the first condition set by the setting unit 125 and the alignment result obtained under the second condition. The estimation error and the reliability of the alignment at each point on the obtained deformed image are calculated, and an estimation error image showing the distribution is generated.

  The display control unit 160 performs control to display the cross-sectional image of the second image and the corresponding cross-sectional image of the deformed image generated by the image generation unit 140 on the display 30.

  Here, the data server 20 holds a plurality of three-dimensional tomographic images of the subject to be aligned. Each of the three-dimensional tomographic images includes, as incidental information, image size, resolution, modality type, imaging information (imaging part, body position, etc.), case information (patient information, organ area information, attention area information, etc.), etc. Contains. These supplementary information is transmitted to the image processing apparatus 10 together with the image as needed.

<2. Processing Performed by Image Processing Apparatus 10>
Next, a procedure of processing performed by the image processing apparatus 10 according to the present embodiment will be described with reference to a flowchart of FIG. In the following description, the first image and the second image obtained by imaging the same subject in different body positions are read from the data server 20 as a plurality of three-dimensional tomographic images, and the first image is subjected to a deformation process. A case where a deformed image deformed so as to match the position and shape with the second image will be described as an example.

(Step S200; acquisition of input image)
In step S200, the image acquisition unit 110 acquires from the data server 20 a plurality of three-dimensional tomographic images (that is, a first image and a second image) of the subject to be aligned. Then, the acquired image is transmitted to image generation section 140 and display control section 160.

(Step S210: Acquisition of correspondence information)
In step S210, the correspondence information acquisition unit 120 acquires correspondence information in the respective image coordinate systems of the first image and the second image. Then, the acquired correspondence information is transmitted to the positioning unit 130. Here, the correspondence information is information on points, lines, and planes corresponding between two images. The acquisition of the correspondence information is executed, for example, by inputting the corresponding points between the images visually identified by the user to the image processing apparatus 10. More specifically, while comparing the cross-sectional images of the respective three-dimensional tomographic images displayed on the display 30, the positions considered anatomically the same on the respective images are clicked by a mouse (not shown) or the like. Acquired by inputting as corresponding points.

(Step S220; parameter setting)
In step S220, the setting unit 125 sets a condition (first condition) for executing the positioning process (the deformation estimation process) (acquires the user's selection) and transmits the condition to the positioning unit 130. . The first condition includes the type of the deformation description model used when estimating the deformation and the detailed setting of the deformation description model. For example, a user's selection regarding the following conditions is acquired.

Selection of a deformation description model (for example, selection between using FFD (Free-Form Deformation) or using a radial basis function)
Selection of grid size of control points when using FFD (for example, selection from 5 mm, 10 mm, 15 mm, and 20 mm)
Selection of the shape of the radial basis function when using the radial basis function (for example, selection from TPS (Thin Plate Spline) function, Gauss function, Wendland function, and cubic function)
-Presence or absence of various regularization terms (terms for evaluating preservation of volume, maintenance of smoothness, etc.) in the cost function used for optimization calculation of deformation and setting of weights Note that the user does not necessarily need to set all conditions It is not necessary that only some of the conditions can be set by the user, and other conditions may be set to predetermined values. Of course, the above conditions are only examples, and it is desirable that conditions other than the above can be set according to the deformation description model and the alignment method to be adopted.

(Step S230; alignment)
In step S230, based on the correspondence information acquired in step S210, the positioning unit 130 executes a positioning process (deformation estimation process) between the first image and the second image under the conditions set in step S220. I do. That is, when the first image is deformed under the set conditions, the deformation information such that the residual of the corresponding point position between the first image and the second image is minimized (or a cost function including the residual) is minimized. (First deformation information) is estimated (the deformation parameter under the condition). Then, the obtained estimation result (hereinafter, referred to as a first alignment result) is transmitted to the image generation unit 140 and the estimation error acquisition unit 150. In the following description, the deformation information estimated in this step is expressed as a mapping function φ (p) of the coordinates p of the first image onto the second image.

(Step S240: Alignment under condition fluctuation)
In step S240, based on the first condition set in step S220, the setting unit 125 sets one or more (N kinds of) conditions (second conditions) different from the first condition. Then, the alignment unit 130 executes the alignment processing of the first image and the second image based on the correspondence information acquired in step S210 under each newly set condition. The processing after setting the respective conditions is the same as in step S230. Then, deformation information (second deformation information) (hereinafter, referred to as a second alignment result), which is a plurality of obtained estimation results, is transmitted to estimation error acquisition section 150. In the following description, the i-th deformation information estimated in this step is referred to as a mapping function φi (p) of the coordinates p of the first image onto the second image.

  Here, the setting of the second condition is performed by fixing other than some of the first conditions set in step S220 and changing the values of some of the conditions (variable conditions) that are not fixed. . That is, the second condition is set by changing at least a part of the first condition. At this time, the fluctuation condition can be determined according to the first condition. For example, a variation condition is determined according to the deformation description model set as the first condition. For example, according to the modification description model of the first condition, a part of the detailed settings of each modification description model is set as a variation condition.

  More specifically, when FFD is set as the deformation description model under the first condition, for example, the grid size of the control point is set as the change condition. At this time, for example, if the grid size of the control points is set to 5 mm under the first condition, the grid size of the FFD is changed to a value other than 5 mm, that is, 10 mm, 15 mm, and 20 mm, respectively. Is the second condition.

  When the radial basis function is set as the deformation description model under the first condition, the shape of the radial basis function is set as the variation condition. At this time, for example, when the TPS function is set as the radiation basis function under the first condition, the radiation basis function is changed to a function other than the TPS function, that is, a Gauss function, a Wendland function, and a cubic function. Are the second conditions.

  Alternatively, the variation condition may be determined according to the setting of the validity / invalidity of each regularization process under the first condition. For example, regarding the regularization processing that is set to be valid under the first condition, each case where the weight of the corresponding regularization term is changed to various values other than the set value is set as the second condition. be able to. Note that the variation conditions may be configured to be specified by the user via the setting unit 125.

(Step S250; generation of deformed image)
In step S250, the image generation unit 140 generates a new image (deformed image) obtained by performing coordinate transformation on the first image so as to match the second image based on the alignment result obtained in step S230. I do.

(Step S260: Acquisition of estimation error)
In step S260, the estimation error acquisition unit 150 determines the position on the deformed image obtained in step S250 based on the alignment result (difference between the first deformation information and the second deformation information) obtained in steps S230 and S240. The estimation error and the reliability of the alignment at each point are calculated, and an estimation error image indicating the distribution is generated. Then, the generated estimated error image is transmitted to display control section 160.

  The estimation error image in the present embodiment is a volume image in which each voxel value indicates the estimation error of the alignment at that position. Hereinafter, a method of acquiring an estimation error in a voxel of interest (referred to as a coordinate q) of the estimation error image will be described. First, based on the mapping φ (first deformation information) obtained in step S230, the coordinates p of the first image mapped to the coordinates q by φ are derived (that is, p where q = φ (p) is obtained). Ask). Next, the coordinates p are mapped with each of the mappings φi (second deformation information) obtained in step S240, and coordinates qi = φi (p) of the deformed image are derived. Here, for the coordinate p of the first image, the coordinate q is a displacement destination based on the alignment result in step S230, while the coordinate qi is a displacement destination based on each alignment result in step S240. Therefore, the estimation error of the coordinate q as the displacement destination of the coordinate p is calculated based on the relationship between the coordinate q and the coordinate qi. More specifically, it is calculated based on the variation of the coordinates qi with respect to the coordinates q. For example, one of the average value, median value, and maximum value of the three-dimensional distance between q and each qi is defined as an estimation error. Alternatively, the distance between q and each qi on the x, y, and z axes may be obtained, and the estimation error for each axis may be obtained in the same manner. Alternatively, the variance or standard deviation of the point cloud obtained by combining q and qi may be obtained, and this may be used as the estimation error. Further, a value converted into 0 to 1 by performing a predetermined normalization process on the value obtained in this way may be held as the reliability. The reliability is obtained based on a difference between the first deformation information and the second deformation information or an estimation error, and is an index indicating how reliable each point on the deformed image is. For example, if the difference between the first deformation information and the second deformation information or the estimation error is small, the reliability is high. If the difference between the first deformation information and the second deformation information or the estimation error is large, the reliability is low.

  The above processing is performed on a predetermined voxel on the deformed image to generate an error estimation image. Note that the region for calculating the estimation error may be the entire deformed image (all voxels), or may be within a region of interest such as an organ region of interest or a lesion. In the latter case, the information of the organ region and the region of interest acquired from the data server 20 in step S200 is referred to. Alternatively, an organ region or a lesion region is acquired by using an image thresholding method or an existing region extraction method. Alternatively, an area on the image specified by the user is acquired as the attention area. Then, a point in the region is set as a point of interest, and an estimation error is obtained for each point of interest. This makes it possible to omit calculations unnecessary for the subsequent processing. In addition, the point of interest may be all voxels in the region or at regular intervals (for example, every five voxels). By doing so, the calculation time of the estimation error can be reduced.

(Step S270: Display of cross-sectional image and estimation error)
In step S270, the display control unit 160 performs control to display the cross-sectional image of the second image and the corresponding cross-sectional image of the deformed image generated in step S250 on the display 30 according to a user operation. In addition, the display control unit 160 performs control to cut out a cross section corresponding to the cross-sectional image of the deformed image from the estimated error image acquired in step S260 and display the cut-out image on the display 30 as an estimated error map.

  FIG. 3A shows an example of a cross-sectional image 310 of a deformed image displayed on the display 30. FIG. 3B shows an example of a display of the estimated error map displayed on the display 30. In this example, the estimation error map 320 is displayed so as to be superimposed on the cross-sectional image 310 of the deformed image. The estimated error map 320 displayed here is a cross-sectional image obtained by cutting the estimated error image (volume data) acquired in step S260 by a cross section corresponding to the cross-sectional image 310 of the deformed image. The luminance value of the image is a value obtained by converting the voxel value of the estimation error map. For example, by converting a predetermined estimation error (for example, an estimation error of 10 mm) to a luminance value of 255 and converting an estimation error of 0 mm to a luminance value of 0, a grayscale estimation error map is created, and a pseudo color is assigned to the map. Display as a map.

  The estimated error map may be displayed side by side on the cross-sectional image or may be displayed in a superimposed manner. On / off of the superimposed display can be controlled, for example, by the user turning on / off a superimposed display button on a GUI (not shown) displayed on the display 30. When this button is off, the estimated error map is not displayed, and only the cross-sectional image 310 is displayed. When the button is on, the estimated error map 320 and the cross-sectional image 310 are displayed in a superimposed manner. When the estimation error in each axis direction is obtained in step S260, the respective estimation error maps may be displayed side by side, or it is possible to select which axis direction estimation error map to display. You may do so.

  Note that the estimated error map displayed in this step may be a map obtained by performing a predetermined process on the estimated error image. For example, red and the like may be translucently superimposed and displayed only on voxels having an estimation error equal to or larger than a threshold value, so that a portion where the estimation error is insufficient may be more clearly confirmed. Of course, it is desirable that the user can select which estimation error map is to be superimposed.

  As described above, the image processing device (image processing device 10) according to the present embodiment estimates the deformation between the first image and the second image under the first condition, and obtains the first deformation information. And a second estimating unit (positioning unit 130, S230) for estimating the deformation between the first image and the second image under a second condition different from the first condition. A second estimating unit (positioning unit 130, S240) that acquires the deformation information of the image, and an estimation error or reliability of the deformation at a point on the image based on the difference between the first deformation information and the second deformation information. And an acquisition unit (estimation error acquisition unit 150, S260) for acquiring a degree.

  According to the present embodiment, it is possible to acquire the estimation error or the reliability of the deformation. Further, since the estimated error map is superimposed and displayed on the cross-sectional image of the deformed image, the user can determine the reliability (how much reliability is required) that each point of the displayed cross-sectional image is at that position. Easy to grasp.

(Modification 1)
The generation of the estimated error image as the volume image in step S260 is not always essential. Instead, an estimation error may be obtained only for each voxel on the cross-sectional image of the deformed image determined to be displayed in the process of step S270, and a cross-sectional image of the estimated error to be displayed may be directly generated.

(Modification 2)
The acquisition of the correspondence information performed by the correspondence information acquisition unit 120 in step S210 may be automatically performed by an image analysis process. For example, a characteristic point or line of an image pattern may be detected from each image and automatically acquired based on the similarity of the image pattern. Alternatively, a corresponding point automatically acquired by the image analysis processing may be set as a candidate, and a point manually corrected by the user may be set as a final corresponding point position. Note that acquisition of the correspondence information may be performed by reading information held by the data server 20.

(Modification 3)
In the processing performed by the positioning unit 130 in step S240, the method of changing the estimation method (estimation condition) is not limited to the above. For example, different deformation estimation results may be obtained by using a plurality of different deformation description models. For example, when the deformation estimation using the FFD is performed in step S230, the deformation estimation using the radial basis function may be performed in this step.

<Second embodiment>
In the first embodiment, the example has been described in which the estimation error map in which the distribution of the estimation error is visualized is superimposed on the cross-sectional image of the deformed image. On the other hand, the image processing apparatus according to the second embodiment is characterized in that the estimation error is displayed in a form other than the map. Hereinafter, in the image processing apparatus according to the present embodiment, only the parts different from the first embodiment will be described.

  The configuration of the image processing system 1, the operation of each unit of the image processing apparatus 10, and the processing procedure in this embodiment are substantially the same as those in the first embodiment. However, only the processing performed by the display control unit 160 in step S270 is different from the first embodiment.

(Step S270: Display of cross-sectional image and estimation error)
In step S270, the display control unit 160 performs control to display the cross-sectional image of the second image and the corresponding cross-sectional image of the deformed image generated in step S250 on the display 30 according to a user operation. In addition, the display control unit 160 performs control to acquire an estimated error of a point designated by the user on the cross-sectional image of the deformed image from the estimated error image acquired in step S260 and display the acquired error on the display 30.

  FIG. 4 shows an example of the display of the estimation error in the present embodiment. When the display control unit 160 acquires an instruction by a mouse or the like of the coordinates on the cross-sectional image 310 of the deformed image displayed on the display 30, the display control unit 160 acquires an estimation error at the coordinates from the estimation error image. Then, for example, as shown in FIG. 4A, character information 420 indicating an estimated error at the coordinates indicated by the cursor 410 is superimposed and displayed near the cursor 410. Alternatively, as shown in FIG. 4B, an ellipse indicating the distribution (error estimation range) 440 on the cross-sectional image of the estimation error at the coordinates indicated by the cursor 430 is displayed. Thus, the user can know the estimated value of the error in the coordinates only by moving the cursor on the displayed cross-sectional image 310.

  As described above, in the first embodiment, the estimated error map in which the distribution of the estimated error is visualized is superimposed and displayed on the cross-sectional image of the deformed image. In the second embodiment, a character representing the estimated error is displayed at the coordinates indicated by the cursor. Information and an error estimation range are displayed on the cross-sectional image of the deformed image. However, the method of obtaining the deformation information and the method of obtaining the estimation error or the reliability at this time are not limited to the methods described in the first embodiment, and various methods may be used. By displaying the deformed image and the estimation error or the reliability in association with each other, the user can grasp the reliability (how much reliable) that each point of the displayed cross-sectional image is at that position. If possible.

  As described above, the image processing apparatuses (image processing apparatuses 10) according to the first and second embodiments include an information acquisition unit (corresponding information acquisition unit 120, the positioning unit 130, and the like) that acquires deformation information of an image. An image generation unit (image generation unit 140) that generates a deformed image obtained by performing coordinate transformation on an image based on deformation information, and an obtaining unit (estimated error obtaining unit) that obtains an estimated error or reliability of deformation on the deformed image 150) and information indicating the estimation error or the reliability (the estimation error map 320 in the first embodiment, the character information 420 and the ellipse indicating the error estimation range 440 in the second embodiment, etc.) are associated with the deformed image (eg, 150). For example, a display control unit (display control unit 160) for displaying (superimposed).

  According to the present embodiment, the estimation error or the reliability of the deformation can be presented by a display method other than the map, and the disadvantage that the cross-sectional image is difficult to see due to the superposition of the error estimation map or the like can be avoided.

(Other embodiments)
The present invention is also realized by executing the following processing. That is, software (program) for realizing the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program and reads the program. This is the process to be performed.

  10: image processing device, 20: data server, 30: display, 110: image acquisition unit, 120: correspondence information acquisition unit, 125: setting unit, 130: alignment unit, 140: image generation unit, 150: estimation error acquisition Section, 160: display control section

Claims (10)

Information acquisition means for acquiring image deformation information,
Image generation means for generating a deformed image by performing coordinate transformation on the image based on the deformation information,
Acquisition means for acquiring volume data representing an estimation error or reliability of the deformation in the deformed image,
A display control unit that superimposes a cross-sectional image of the deformed image and a map indicating a cross-section in the volume data corresponding to a cross-section of the cross-sectional image of the deformed image on a display unit;
An image processing apparatus comprising: The obtaining means generates an estimated error image representing a distribution of the estimated error,
The image processing apparatus according to claim 1, wherein the display control unit causes the display unit to display a cross section of the estimated error image corresponding to the cross section image of the deformed image as the map.   The image processing apparatus according to claim 1, wherein the display control unit switches whether to display the cross-sectional image of the deformed image and the map in a superimposed manner in accordance with an instruction from a user. apparatus.   The image processing apparatus according to claim 1, wherein the obtaining unit obtains the volume data representing an estimation error or reliability of deformation at each point on the deformed image.   4. The image processing apparatus according to claim 1, wherein the acquiring unit acquires the volume data representing an estimation error or reliability of a deformation in a region of interest on the deformed image. 5. .   The image processing apparatus according to claim 1, wherein the map is a color map converted based on the estimation error or the reliability.   4. The display control unit according to claim 1, wherein the cross-sectional image of the deformed image is superimposed on a point on the map at which the estimation error or the reliability is equal to or more than a predetermined threshold. 7. The image processing device according to any one of 6.   The image processing apparatus according to claim 1, wherein the reliability is a value obtained by performing a normalization process on the estimation error. An information acquisition step of acquiring deformation information of the image,
An image generation step of generating a deformed image by performing coordinate transformation on the image based on the deformation information,
An acquisition step of acquiring volume data representing an estimation error or reliability of the deformation in the deformed image,
A display control step of superimposing and displaying a cross-sectional image of the deformed image and a map indicating a cross-section in the volume data corresponding to the cross-section of the cross-sectional image of the deformed image on a display unit;
An image processing method comprising: A program characterized by causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 8.