JP3438937B2 - Image processing device - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus for estimating a three-dimensional shape of an object by using an image obtained by picking up an image of the object and estimating its error.

[0002]

2. Description of the Related Art When the relative arrangement of image pickup means arranged so as to overlap the field of view is known, the problem of estimating the shape of an object from a picked-up image is solved by a so-called stereo image shape. Various methods have been proposed as estimation problems.

Furthermore, in recent years, several proposals have been made for extracting three-dimensional structures from motion. This is a method that also tries to estimate the relative movement of the image pickup means from a plurality of images.

Each of these methods uses, as the original information, that the positions of the same point on the object in each image are associated with each other between the plurality of images.

Various methods have been proposed for detecting the position of the same point on an object on each image. In the case of artificial objects, since the corners and contour components are often clear, the method of structure extraction such as line segment extraction is effective, but it is difficult to apply to general natural images. The density gradient method, which is often used for time-series images, gives very good results when the apparent movement of the object between images is very small and the image quality is good, but the constraints of the imaging conditions are large. .

Therefore, in general, a point of interest on the image and its surrounding area are extracted as a reference area, and a correlation operation is performed between the area extracted from the image of the object to be searched and a position giving the maximum correlation value. A method (block matching) with the corresponding points is often used. The block matching method is
Clear textures on the image give relatively stable results. However, on the other hand, when the texture is unclear, an incorrect corresponding point may be detected.

Further, as an essential drawback, when the object is a solid and the block includes a boundary with the background, for example, the detection result is unreliable. Further, it is also reliable when the deformation of the image between the images of the plurality of image pickup devices is large or the difference in the image size is large, such as when the object surface is tilted with respect to the image pickup device or there is a difference in distance. Poor sex.

[0008] As a problem related to the estimation of the three-dimensional shape, there is a problem regarding occlusion and matching of the estimation result. In particular, in stereo estimation, there is a part that is hidden behind the object and is not imaged, or a part that is imaged by only one of the image pickup means, and handling of these regions poses a problem.

A large number of image pickup means inevitably results in a large area to be picked up and a small hidden area. However, especially when the position of the image pickup means is not known or the estimation is inaccurate, these areas are hidden. Matching between images is not easy.

Most of the conventionally proposed methods target an image of an artificial object. Or, for these problems that naturally occur when an object is a natural image, it is considered under the condition that the influence is excluded or mitigated by making any assumption, and it is considered that the problem is practically sufficient. I can't say.

For example, an image obtained from a biomedical endoscope is one of the types of images that are difficult to apply the method proposed hitherto and have great practical value when the function is realized.

By inserting an elongated insertion portion into a body cavity, an endoscope can be used for observing an affected area in the body cavity without the need for incision and for medical treatment using a treatment tool if necessary. The tip size must be functionally minimized. For this reason, it is not possible to install components other than those necessary for the doctor to perform observation or treatment.

Several proposals have already been made for grasping the shape of a target object endoscopically. For example, a spot light system for projecting pattern light or the like onto an object to be observed (Japanese Patent Laid-Open No. 64-64525) and a compound eye system for obtaining an image with parallax by a compound eye placed at the tip (Japanese Patent Laid-Open No. 6-64525).
3-244011). Japanese Unexamined Patent Publication No. 62-168791 by the same applicant discloses a method for estimating the shape of an object from a plurality of images obtained by moving the endoscope tip by hand operation, and measuring the amount of movement of the tip with the operation. A measuring mechanism that does this is disclosed. According to this method, an absolute object shape can be estimated without impairing the function of the current endoscope. These will be referred to as absolute shape methods.

On the other hand, JP-A-6-728 by the same applicant
According to Japanese Patent Publication No. 9, the same object is imaged from a plurality of positions, the shape of the object is estimated from the plurality of imaged images, and the shape is determined using the relationship between the estimated shape and the position of the imaging means. There is disclosed a method for estimating a three-dimensional shape by the weighting function described above, that is, a relative shape estimation (relative shape) method.

[0015]

When the same point is detected and the shape is estimated by the above-mentioned relative shape method, compound eye method which is an absolute shape method, and spot light method, an error occurs due to the following factors. .

(1) In any of the three methods, an error occurs due to noise in the image.

(2) Since the same target object is imaged at different positions, an error occurs because the texture is deformed. It is considered that they are the same when viewed in a local area, but they are actually different. The relative shape method and the compound eye method are applicable.

(3) Since the same target object is imaged at different positions, unevenness in illumination occurs, resulting in an error. This also applies to the relative shape method and the compound eye method.

(4) In the spot light system, an error occurs because the image of the spot light created as the template image is strictly different from the image of the spot light applied to the target object (see FIG. 37 ( See a) and (b)).

In the invention described in Japanese Patent Application Laid-Open No. 6-7289, the same point is detected when the three-dimensional shape of the object is estimated from a plurality of images. Even if there is an error in, the estimated three-dimensional shape will be displayed with the shape error included. Therefore, it is not possible to know how much error the estimated three-dimensional shape includes, and it is not possible to grasp the degree of its reliability.

The same applies to the systems disclosed in Japanese Patent Laid-Open Nos. 64-64525 and 63-244011.

The present invention has been made in view of the above circumstances. It is possible to estimate the shape of an object, grasp the degree of error in the estimated shape, and evaluate the reliability of the estimated shape. An object is to provide an image processing device.

[0023]

Image by Means and operation for solving the object of the present invention
The image processing device , based on the information obtained from the image of the object imaged by the imaging means and the information about the imaging means,
A three-dimensional shape estimating means for estimating a three-dimensional shape of the target object, and a three-dimensional shape estimating means for estimating the three-dimensional shape of the target object.
A three-dimensional shape error estimating means for estimating the error of the shape generated in that process, estimated by the three-dimensional shape error estimating means
And an output means for outputting the error to the display device, and the image pickup means relates to the same object.
Each image for multiple images captured from multiple positions.
Position detecting means for detecting the position of the same point on the image ,
Place on each of the images detected by the position detecting means.
From the same position information
And a position estimation means for determining the position of the image pickup means.
Information is information on the position estimated by the position estimating means.
It is a feature. Further, the image processing according to the present invention
The processing device has a projection means for projecting one or more pattern lights.
A pair in which the pattern light is projected by the imaging means having
From the image of the elephant, the position information of the pattern light can be obtained.
The position information detecting means for detecting and the position information detecting means
It includes the detected position information and each irradiation direction of the pattern light.
The absolute value information about the image pickup means
The distance to the pattern light irradiated on the
Absolute solid shape that estimates the absolute solid shape of the object from
And Jo estimating means, distance determined by the absolute three-dimensional shape estimation unit
The detection error included in the separation is calculated, and based on this error, the absolute
Shape Error Estimator for Estimating Error Range of 3D Shape
And the error estimated by the three-dimensional shape error estimation means.
And a means for outputting the difference to the display.
To do.

[0024]

Embodiments of the present invention will be described below with reference to the drawings. 1 to 20 relate to a first embodiment of the present invention,
1 is an overall configuration diagram of an electronic endoscope system including the first embodiment, FIG. 2 is a block diagram showing a block configuration of an electronic endoscope device, FIG. 3 is a block diagram showing a configuration of an image processing device,
4 is a block diagram of image processing configured by a CPU, FIG. 5 is a block diagram of an image processing device configured by hardware, FIG. 6 is a block diagram showing the configuration of a recording device, and FIG. 7 is three-dimensional image data of two-dimensional image. FIG. 8 is a flow chart showing the processing contents of converting into data and estimating an error, FIG. 8 is an explanatory diagram regarding estimation of a three-dimensional shape, FIG.
FIG. 10 is an explanatory diagram showing how the shift amount between the template image and the reference image is obtained, and FIG. 11 is an explanatory diagram showing how the shift map is calculated from the two-dimensional image data sequence.
Is a flow chart showing a concrete process of iterative processing of a motion vector by calculating a shift map, FIG. 13 is an explanatory view showing a coordinate system of central projection, FIG. 14 is an explanatory view showing a rotation axis and a rotation vector, FIG. 15 is an explanatory diagram showing that three vectors are on the same plane, FIG. 16 is an explanatory diagram showing movement of the tip of the endoscope, FIG. 17 is a flowchart diagram showing a state of iterative estimation, and FIG. 18 is an epipolar line. FIG. 19 is an explanatory diagram of the error estimation at the position of deviation from the epipolar line, and FIG. 20 is an explanatory diagram of the display of the estimated three-dimensional shape and the error range.

First, an outline of the present embodiment will be described, and then a specific configuration, operation, etc. will be described.

The image processing apparatus according to the present embodiment has a schematic configuration in which the same object is imaged from a plurality of positions by the image pickup means, the position of the same point on each screen is detected, and the image pickup means is used. Has a relative shape estimating means for estimating the relative shape of the object by estimating the relative movement of the object, and further estimated by the relative shape estimating means from the detection error generated by the position detecting means. Error estimation means for estimating the error in the shape of the target object.

An endoscopic image processing apparatus having this image processing apparatus has a structure in which the estimated three-dimensional shape of the object and the estimated error are combined and displayed, and the same object is imaged by the imaging means. Images are taken from multiple positions, and the relative shape of the object is estimated by using the correlation between the images. At the same time, the shape error range due to the detection error in the detection of the same point for estimating the shape is estimated and the object is estimated. The relative shape and the error range are displayed.

As shown in FIG. 1, an endoscope system 1 estimates an electronic endoscope apparatus 2 equipped with an image pickup means, a three-dimensional shape of a diseased part or the like based on a picked-up image, and estimates a shape error. The image processing device 3 of the first embodiment for performing image processing for
A recording device 4 for recording an image and a monitor 5 for displaying the image-processed image.

The electronic endoscope apparatus 2 includes an electronic endoscope 6, a light source section 7A (see FIG. 2) for supplying illumination light to the electronic endoscope 6, and a signal processing section 7 for performing signal processing for the image pickup means.
The observation device 7 has a built-in B, and the observation monitor 8 displays the image signal output from the observation device 7.

The electronic endoscope 6 has an elongated insertion portion 11 to be inserted into a living body 9, an operation portion 12 formed at a rear end of the insertion portion 11, and a universal portion extended from the operation portion 12. The observation device 7 includes a connector 14 that is composed of a cable 13 and is provided at the end of the universal cable 13.
Can be connected to. The insertion portion 11 includes a hard tip portion 16a and a bendable bending portion 16 in order from the tip side.
It has b and the flexible tube part 16c.

A light guide 15 is inserted into the insertion portion 11 and the connector 14 is connected to the observation device 7, whereby illumination light is supplied from the light source portion 7A to the incident end face as shown in FIG. The light is transmitted by the light guide 15 and is emitted forward from the end face on the side of the tip portion 16a, so that the living body 9
Illuminate the target part inside. The illuminated target portion is imaged by the objective lens 17 provided on the tip portion 16a on the CCD 18 arranged at the image forming position, and photoelectrically converted. The objective lens 17 and the CCD 18 form an image pickup section 19 as an image pickup means. The image pickup means is not limited to the CCD or the like, but may be another element, or an electronic camera or the like provided in the eyepiece of the optical fiber endoscope.

The image signal photoelectrically converted by the CCD 18 is subjected to signal processing by the signal processing section 7B in the observation device 7 to generate an image signal.
And is output to the image processing device 3.

The structures of the light source section 7A and the signal processing section 7B in the observation device 7 are shown in FIG. The light source unit 7A includes a lamp 21 that emits light in a wide band from ultraviolet light to infrared light. As the lamp 21, a general xenon lamp, a strobe lamp, or the like can be used. The xenon lamp and strobe lamp emit a large amount of not only visible light but also ultraviolet light and infrared light.

Power is supplied to the lamp 21 from a power source 22. A rotary filter 50, which is driven to rotate by a motor 23, is arranged in front of the lamp 21. Filters that transmit light in the respective wavelength regions of red (R), green (G), and blue (B) for normal observation are arranged in the rotary filter 50 along the circumferential direction.

The rotation of the motor 23 is controlled by a motor driver 25 so that the motor 23 is driven. The light that has passed through the rotary filter 50 and is time-sequentially separated into lights of respective wavelength regions of R, G, and B is further light guide 1.
The light is incident on the incident end of No. 5, is guided to the emitting end face on the side of the tip portion 16a through the light guide 15, and is emitted forward from this emitting end face to illuminate the observation site and the like.

The return light from the subject (subject) such as the observation site due to this illumination light is transmitted by the objective lens 17 to the CC
An image is formed on D18 and photoelectrically converted. A drive pulse from the driver 31 in the signal processing unit 7B is applied to the CCD 18 via the signal line 26, and only the electric signal (video signal) corresponding to the image of the subject photoelectrically converted by the drive pulse is applied. Read-out is performed.

The electric signal read from the CCD 18 is input to the preamplifier 32 provided in the electronic endoscope 6 or the observation device 7 via the signal line 27. The video signal amplified by the preamplifier 32 is input to the process circuit 33, subjected to signal processing such as γ correction and white balance, and then A / D converter 34.
It is designed to be converted into a digital signal.

This digital video signal is selected by the select circuit 35, for example, three memories (1) 36a, memory (2) 36b, and memories corresponding to each color of red (R), green (G), and blue (B). (3) It is adapted to be selectively stored in 36c. R stored in the memory (1) 36a, the memory (2) 36b, the memory (3) 36c,
The G and B color signals are simultaneously read out, converted into analog signals by the D / A converter 37, and output to the color monitor 8 as R, G and B color signals via the input / output interface (I / O) 38. The color monitor 8 is used to color-display the observed region.

Further, a timing generator 42 for making the timing of the entire system is provided in the observation device 7, and the timing generator 42 synchronizes the circuits such as the motor driver 25, the driver 31, and the selection circuit 35. Has been.

In this embodiment, the output terminals of the memory (1) 36a, the memory (2) 36b, the memory (3) 36c and the synchronization signal output terminal of the timing generator 42 are connected to the image processing apparatus 3. Further, the image processing device 3 is connected to the color monitor 5, and the calculation result by the image processing device 3 is displayed on the color monitor 5.

FIG. 3 shows the structure of the image processing apparatus 3.
A CPU 40, an information input device 41, a main storage device 42 including a RAM, an image input interface 43, a display interface 44, a ROM 45, and a recording device interface 46 are provided, and these are connected to each other by a bus.

As shown in FIG. 4, the CPU 40 has a ROM 4
5 has an image processing means 47 which operates according to the program stored therein. This image processing means 47 is a distortion aberration correcting means 49 for correcting the influence of the distortion aberration by the optical system so as to eliminate it from the image, and, for example, for a plurality of images taken by the imaging means from a plurality of positions with respect to the same object, The position detecting means 50 for detecting the same position on each image, the position estimating means 51 for estimating the position of each imaging means, that is, the motion from the position of the same point on each image, and the position detecting means 50. A shape estimating means 52 for estimating the three-dimensional shape of the object from the detected position of the same point and the position (movement) of each imaging means estimated by the position estimating means, and a plurality of images of the same object. Averaging processing means 53 for averaging each three-dimensional shape obtained by dividing the three-dimensional shape into one shape, and estimation caused by, for example, a detection error of the position detecting means 50 or the like. An error estimating means 54 for estimating the error of the three-dimensional shape of the target object, consisting of the image synthesizing means 55 or the like and for combining the three-dimensional shape and the error of the estimated object.

The information input means 41 is composed of a keyboard or the like, and can input data such as the type of the electronic endoscope 6. The image input interface 43 includes a memory (1) 36a, a memory (2) 36b, and a memory (3) 3.
6c is connected to receive these image data. Further, the display interface 44 is adapted to send the image data displayed on the monitor 5. Further, FIG. 5 shows a block configuration example when the arithmetic processing of the image processing apparatus 3 is configured by hardware.

Memories (1) 36a, (2) 36b,
(3) The time-series image data recorded in 36c is temporarily stored in the memory 60 including a plurality of frame memories via the image input interface 59. The stored image data is corrected by the distortion aberration correcting means 61, and the corrected image data is corrected by the position detecting means 62, the position estimating means 63, the shape estimating means 64, the averaging processing means 65 and the error estimating means 66. An error between the three-dimensional shape of the object and the three-dimensional shape is calculated. The calculated three-dimensional image data or two-dimensional image data and the stereoscopic error data are combined by the image combining means 67 and stored in the memory 68.

The error between the three-dimensional image data or the two-dimensional image data stored in the memory 68 and the three-dimensional shape is read out,
It is converted into an analog signal by the D / A converter 69, and the three-dimensional or two-dimensional shape and shape error of the target site are displayed on the monitor 5 via the display interface 70.

Further, in the image processing device 3, the distortion aberration correcting means 61, the position detecting means 62, the position estimating means 63, the shape estimating means 64, the averaging processing means 65, and the error estimating means 66.
Further, an arithmetic processing controller 71 for controlling the arithmetic operation of the image synthesizing means 67 and a memory controller 72 for controlling the reading and writing of data of the memories 60 and 68 are provided. The selector 73 is the memory controller 72
One of the memory 60 or the memory 68 and the recording device 4 are selectively connected to each other via the recording device interface 74 by the control signal of 1., and data can be transferred between the recording device 4 and the memory 60 or the memory 68. It has become. The memory controller 72 is an information input device 75.
Are connected.

In this embodiment, the image of the object portion obtained by the electronic endoscope 6 is processed by the image processing device 3 and the processed result is output to the monitor 5.

FIG. 6 is connected to the image processing apparatus 3 and records image data which is image-processed by the image processing apparatus 3 or records image data which is processed by the image processing apparatus 3 and displayed on the monitor 5. A more specific configuration example of the recording device 4 will be shown.

The image recording device 80 is an analog type image recording device such as a VTR or an analog magneto-optical disk device, and its output is A / D converted by an A / D converter 81.
After the conversion, the digital image is input to the image processing apparatus 3. Further, the processing result image by the image processing device 3 can be recorded in the image recording device 80 via the D / A converter 84.

The moving picture file device 82 is a semiconductor memory,
It is a digital moving image file device including a hard disk, a digital magneto-optical disk device, etc., and its output is input to the image processing device 3 as a digital image. The still image filing device 83 is a digital still image filing device including a semiconductor memory, a hard disk, a digital magneto-optical disc device, etc. like the moving image filing device 82, and its output is a digital image as an image processing device. Input to 3.

Further, the processing result image by the image processing device 3 can be recorded in the moving image file device 82 and the still image file device 83. In the present embodiment, the so-called relative position shape is estimated by moving the image capturing means and using a plurality of images captured at a plurality of positions to estimate the three-dimensional shape. Therefore, the image pickup means including the CCD 18 is moved to capture a plurality of images. The capturing of a plurality of images by moving the endoscope can be performed by the following method, for example.

For the same target object, the endoscope bending section 1
The endoscope distal end portion 16a is moved in a specific direction by the bending operation of 6b.
To move. At this time, the bending operation is gently performed so that the image shift does not occur. With the bending operation, a plurality of time-series continuous images are obtained for the same target object. From this continuous image, the three-dimensional shape and the error of the target object are estimated as described later.

As described above, in addition to the method of continuously moving the image pickup means at a predetermined speed or less in order to suppress the occurrence of error as much as possible, a plurality of completely stationary motions from different positions with respect to the same target object. It is also possible to capture an image and estimate the three-dimensional shape of the target object from the still image.

In FIG. 2, the image signal photoelectrically converted by the CCD 18 becomes an image signal signal-processed by the signal processing unit 7B in the observation device 7, and is output to the observation monitor 8 and the image processing device 3. To be done.

The two-dimensional image data input to the image processing device 3 is once sent to the recording device 4 and the image signal is recorded. Also, the recorded image signal is sent again to the image processing device 3, where the process of estimating the three-dimensional shape of the subject and the error estimation is performed, and is sent to the recording device 4 and the monitor 5, and the estimated three-dimensional image is obtained. The shape and error are recorded and displayed.

First, in this embodiment, 2
Processing for obtaining three-dimensional data from a three-dimensional image is performed. The endoscopic image obtained by the electronic endoscope 6 is sequentially sent out to the image processing device 3 as an image signal by the observation device 7.
The image processing device 3 sends the image signals of the endoscopic images sequentially sent in an arbitrary period to the recording device 4 for recording.

The image data of a plurality of frames of the endoscopic image data (two-dimensional image data) recorded in the recording device 4 is sent to the image processing device 3, and the image data of a plurality of frames sent to the same point After detecting the position, a shift map representing the movement of the detected position is calculated.

The motion vector of the image pickup means (motion vector of the tip of the electronic endoscope 6) is estimated from the obtained shift map. Further, the error is estimated from the position detection result for the same point and the motion vector.

Further, each shape is estimated by using the estimated motion vector of the image pickup means and the shift map between each image. One solid shape is obtained by performing weighted addition or the like on each estimated shape according to the reliability.

With respect to the obtained three-dimensional data, image data to be displayed as a two-dimensional image or a three-dimensional image when viewed from the direction of an arbitrary image pickup means (wire frame etc.) is created. The shape estimation error is combined with the three-dimensional image data or the two-dimensional image data created by the image processing device 3 and displayed on the monitor 5 and simultaneously recorded on the recording device 4.

FIG. 7 and FIG. 8 show a flow chart until the two-dimensional image data is converted into three-dimensional data by the image processing device 3. The process until the three-dimensional image data is calculated will be described below.

Step S1 shows that an image taken while the image pickup means is moved is input to the image processing apparatus 3. That is, while moving the image pickup means (the tip of the electronic endoscope 6) with respect to the same object, the position is slightly different.
Input dimensional image data.

Next, in step S2, distortion aberration correction processing is applied to the plurality of images obtained in step S1,
Correct image distortion. About this distortion correction processing,
Give a supplementary explanation.

Since the endoscope image data sent from the recording device 4 is distorted by the wide-angle lens, distortion correction processing is applied to each image to correct the image distortion. When, for example, the square cells shown in FIG. 9A are imaged using the electronic endoscope 6, the obtained image is as shown in FIG.
It becomes as shown in (b). It is possible to realize distortion aberration correction processing by predetermining the correction value for each image so that this distorted image becomes a square grid image and correcting the actual captured image. Become. More specific processing method is USP 4,89
It is disclosed in Japanese Patent No. 5,431.

Next, the corresponding points are tracked, that is, the object is imaged using the plurality of corrected images (image sequences).
One image is selected, and a plurality of points on the image are selected to track how each point moves on another image (reference image). The tracking of the corresponding points will be described below.

In the corresponding point tracking, as shown in FIG. 10A, first, as shown in FIG. 10A, a rectangular area (template image) centering on the point selected earlier in the image in which the object to be detected is taken is t. (X). As shown in FIG. 10B, a search area S of a certain size is set in the reference image, and block matching between the corresponding area f (x) in the search area S and the rectangular area t (x) of the template image is performed. A process, for example, a cross-correlation calculation is performed to calculate the area having the maximum correlation value, and the moving direction and the moving amount of the reference image with respect to the template image in that case are calculated.

For example, the following normalized cross-correlation D (u, v); To find the correlation value, find the area where it is maximum,
In that case, the moving direction and the moving amount are obtained. Here, the double integration symbol represents the integration within the search area S,
<F> and <t> are S of f (x + u, y + v) and t (x, y), respectively.
Indicates the average within.

The block matching process is not limited to the cross-correlation calculation, and the US
The color matching method disclosed in P4,962,540 may be applied. (Reference: Introduction to computer image processing, Research Institute Publishing Co., Ltd. Hideyuki Tamura, supervised: p.1
48-p.150) As described above, the shift map describing the values of the moving direction and the moving amount of the point selected on the template image is obtained.

Using the shift map obtained in step S2, in step S3, the motion vector of the image pickup means is obtained by iterative processing such as the steepest descent method, and the relative positional relationship between the object and the position of the image pickup means is obtained. Ask.

In step S4, an error is estimated from the value of the relative position between the image pickup means and each point of the object and the motion vector of the image pickup means.

In step S5, the positional relationship between the subject and the image pickup means obtained by each shift map is converted into the same coordinate space, and the positions of the subject and the image pickup means at the same points are averaged to obtain one subject. Find the shape. Thereafter, the three-dimensional image data estimated in step S6 is obtained, and in step S7, the image data having the estimated shape and the relative position error obtained in step S4, that is, the data of the shape error are predetermined. It is synthesized in the format of and is output to the display device side.

Next, each process for calculating the three-dimensional image data will be sequentially described. (1) Creation of Shift Map from Input Image (Detection of Position) First, the endoscopic device 2 performs imaging from a plurality of positions on a target site desired to be diagnosed inside the living body 9 or the like, A plurality of images of this target part are recorded in the recording device 4. Regarding this point, as described above, the electronic endoscope 6
The distal end side of the image pickup device is imaged with respect to the same target site by gradually bending, and the obtained plurality of images are recorded in the recording device 4.

As a result, the electronic endoscope 6 shown on the left side of FIG.
The endoscopic images of the image rows f0 to fNi (shown on the right side of FIG. 11) taken with the movement of (the image pickup means built in) the front end portion 16a are obtained.

The image processing apparatus 3 applies the distortion correction processing to the endoscopic image sent from the recording apparatus 4,
Ni + 1 frame of the corrected endoscopic image (image sequence f0
.About.fNi) is divided into N sets, and corresponding points corresponding to the movement loci of the image pickup means indicated by P0 to PN having the divided i + 1 frame images are traced to calculate respective shift maps.

As shown in FIG. 11, the shift map 1 uses the image of frame i + 1 of the image sequence f0 to fi to sequentially calculate the amount of movement between each frame while tracking the following two corresponding points. .

(1) Corresponding points are traced in the images f0 and f1, then in the order of images f1 and f2, and then in the order of images f2 and f3, the obtained movement amount data are all added. Find the correspondence between the images f0 and fi.

(2) The images f0 and f1 are associated with each other,
Based on the movement amount obtained by this association, the image data f
Corresponding points are traced in the order of images f0 and f3 based on 0 and f2 and their values, and finally the correspondence between images f0 and fi is obtained.

The relationship between (1) and (2) is (1), (2)
The corresponding points are tracked by the method of (2) and the images f0 and fk and the images f0 and f of the method (2) at a certain point
The increase of the parallax amount of k + 1 and the images fk and fk obtained by the method (1).
If the difference exceeds a certain amount (3 pixels experimentally), it is determined that a mismatch has occurred at that point, and the The amount of movement at that point is re-estimated from the data using an interpolation function. Note that k represents an arbitrary image in the N-divided image example (for example, the image sequence f0 to fi that is the target of the present description). The contents of this process are as shown in the flowchart of FIG.

The image of the data Ni + 1 is taken in at step S11, and the distortion aberration correction process is performed at step S12. In the next step S13, the image of Ni + 1 is N
Divide into sets. After the next step S13A, the corresponding points are traced by the methods (1) and (2) with respect to each image sequence divided into N sets, and it is determined whether or not the variation amount of the parallax amount exceeds the threshold value. to decide.

That is, m = 1 in step S13A, and n = 0 in step S14. Then, in the next steps S15a and S15b, the corresponding points between the image f0 and the image f1 are traced by the methods (1) and (2). Each is performed, and the amount of change in the amount of parallax between the processes (1) and (2) is calculated in step S16.

In the next step S17, it is determined whether or not each change amount exceeds a threshold value (the above-mentioned fixed amount), and if it exceeds this threshold value, interpolation processing is performed in step S18.
If the threshold is not exceeded, n = i-1 in step S19.
Determine whether or not. If not n = i-1, step S
At 20, n = n + 1 is set and the process returns to steps S15a and S15b. In steps S15a and S15b,
Tracking the corresponding points of the image f1 and the image f2 in (1) is (2)
The corresponding points of image f0 and image f2 in
Each parallax amount between the image f0 and the image f2 is obtained, and the change amount thereof is calculated. In step S17, the change amount and the threshold value are compared, and the same processing as described above is performed. In this way, similar processing is performed for images up to n = i-1.

The processing of FIG. 12 is performed, and the shift amount between each point and each image is sequentially obtained, and the position P (image f0 and fi) is calculated.
The shift map 1 is calculated by obtaining the shift amount between 0 and the position P1.

When the shift map 1 is calculated, n = i-1 holds in step S19, and the process proceeds to step SC. In step S13C, it is determined whether m = N and m = N.
If not, m = m + 1 is set in step S13B, the process returns to step S14, and the shift map for the next image sequence is calculated. In this way, the image sequences fi to f2i, ..., F (n-1) i
The above calculation is also performed for .about.fNi, and shift map 2 to shift map N (shift amount between P1 and P2 or PN-1 and PN) are obtained. By calculating the N shift maps, the movement amounts of the corresponding points on the images picked up by the image pickup means at the positions indicated by P0 to PN are calculated in association with each other, and the position up to a certain image is calculated. Once determined, the position of the same point on each of the other images will be calculated.

(2) Position estimation by motion estimation The motion of the image pickup means (motion of the endoscope tip 16) is estimated from the shift map thus obtained using an 8-point algorithm. However, if an error is included in the correspondence between the images, the correct solution cannot be obtained by the 8-point algorithm. Therefore, the least squares method is introduced into the 8-point algorithm, and the steepest descent method is applied to obtain the solution, and the motion vector of each shift map is obtained.

First, the 8-point algorithm will be described (reference: image understanding, Kenichi Kanaya, Morikita Publishing Co., Ltd.).

FIG. 13 shows the coordinate system of the central projection. An image plane is taken perpendicular to the Z axis at a distance f from the origin O. At this time, it is assumed that the point P (X, Y, Z) in the space is projected at the intersection of the straight line image plane connecting the point P (X, Y, Z) and the origin O. The image coordinates of that point are (x,
From the geometrical relationship, y and y are as follows: x = Xf / Z y = Yf / Z (1)

The point X in the space X =
When (X, Y, Z) is moved to X '= (X', Y ', Z'), the motion of the image pickup means is then the orthogonal matrix R representing the rotation around the visual axis passing through the origin and the translation. Vector h = (hx,
hy, hz) T, and is expressed by the following equation. X = RX ′ + h (2) Here, as shown in FIG. 14, when n = (nx, ny, nz) is the rotation axis direction and θ is its rotation angle, R is expressed as follows. .

[0088] The points X and X'are the distance from the origin and the unit direction vector m (N
When expressed using vectors) and m ′, they are as follows.

[0089] Further, when the equation (4) is used, the equation (2) becomes rm = r′Rm ′ + h (6)

As shown in FIG. 15, three vectors m,
Since Rm ′ and h exist on the same plane in space, their scalar triple product becomes 0. That is, | mhRm ′ | = (m, h × Rm ′) = 0 (7) If the outer product of h and R in equation (7) is G, then (m, Gm ′) = 0 (8).

However, G = (h × r1, h × r2, h × r3) (9), and r1, r2, r3 are the first, second, and third column vectors of R, respectively. When the vector product of both sides of Expression (6) and h is taken, it becomes rh × m = r′h × Rm ′ (= r′Gm ′) (10).

Taking the inner product of both sides and h × m, Becomes From this, | hmGm ′ | ≧ 0 (12) is obtained.

Here, the element of G is determined from R and h. Equation (8) is a simultaneous equation regarding the nine elements of G. Therefore, the ratio of the elements of G can be determined from the eight linear equations obtained from the eight corresponding pairs.

Next, Let G ^ be a constant multiple such that This corresponds to normalizing the translation vector h into a unit vector.

According to equations (8) and (9), the translation vector h is orthogonal to all the column vectors of G. Therefore, assuming that the column vector of G ^ is g ^ 1, g ^ 2, g ^ 3, the unit vector h ^ that (g ^ i, h) = 0, i = 1,2,3 (14) is calculated. . The sign is Equation (12)
Therefore, | h ^ mαG ^ m′α | ≧ 0, α = 1, ..., 8 (15)

Next, based on the equation (9), a unit vector (R) which is orthogonal to each other and forms a right-handed system such that h ^ × r ^ = g ^ i, i = 1, 2, 3 (16) is obtained. r ^ 1, r ^ 2, r ^ 3) is obtained, and a matrix having columns in this order is R = (r ^ 1, r ^ 2, r ^ 3).

Taking the inner product of both sides of equation (6) with m and Rm ', respectively, r ^ α = r' ^ α (mα, R ^ m'α) + (h ^, m '^ α) r ^ Α (mα, R ^ m'α) = r '^ α + (h ^, m' ^ α) (17) If this is solved, the distance to each point can be obtained as follows.

[0098] As described above, the 8-point algorithm cannot obtain a correct solution when the correspondence between images includes an error. Therefore, it is necessary to perform optimization so that the necessary conditions are strictly met and the other conditions are met on average.

Then, it is assumed that N sets (N ≧ 8) of corresponding points are observed, and the least squares method is used. Then, a matrix G is obtained, and this is approximately decomposed into a unit vector h and an orthogonal matrix R (least squares algorithm).

Therefore, a certain constraint condition is applied to the motion of the image pickup means, and the motion parameters h0 and R0 of the image pickup means are applied.
Is estimated, and h and R that minimize Equation (19) are estimated using this as an initial value. This makes it possible to limit the existence space of the solution in advance and minimize the estimation error.

Initial values h0 of motion parameters of the image pickup means,
Find R0. As shown in FIG. 16A, the movement of the tip of the electronic endoscope 6 (which incorporates an image pickup means that moves by an external operation) can be regarded as movement in a certain direction.

As shown in FIG. 16B, the movement of the image pickup means in the visual axis direction (Z direction) due to the rotation of the swing is small, and the translation vector h representing the movement is in a plane perpendicular to the Z axis. The initial value can be obtained by assuming that the rotation axis U is perpendicular to h and the rotation angle θ is + in the direction of h.

The moving directions of a plurality of points are obtained from the shift map and averaged to determine the translation vector h0. When the rotation axis U0 is determined perpendicularly to this h0, three equations including three unknowns r, r ', and θ0 are obtained from the equation (8), and θ0 is obtained by solving them.

Θ0 is obtained for a plurality of points, an average value is obtained from each θ0, and an initial value is determined. With the value thus obtained as an initial value, the optimum solution is determined by the steepest descent method from the equation (19).

As shown in FIG. 17, the solution is obtained by the steepest descent method. Each element of the translation vector h and the rotation matrix R of the j-th shift map M (j) among the plurality of shift maps is represented as follows. [Hx (i), hy (i), hz (i), nx (i), ny (i), nz (i), θ (i)] ji = 0,1,2, ..., n FIG. 1) by the method described in a) and 16 (b).
As in step S21 of 7, the translation vector h0 and the rotation matrix R0 that are the initial values of the motion vector are obtained. At this time, each value is expressed as follows. [Hx (0), hy (0), hz (0), nx (0), ny (0), nz (0), θ (0)] j One of each vector obtained as in step S22 Select a parameter and set Δd (≧
0) is given as follows, a total of 14 parameter value pairs can be obtained.

[Hx (0) + Δd, hy (0), hz (0), nx (0), ny (0), nz (0), θ (0)] j [hx (0) -Δd, hy (0), hz (0), nx (0), ny (0), nz (0), θ (0)] j [hx (0), hy (0) + Δd, hz (0), nx (0), ny (0), nz (0), θ (0)] j [hx (0), hy (0) -Δd, hz (0), nx (0), ny (0), nz ( 0), θ (0)] j ・ ・ ・ [hx (0), hy (0), hz (0), nx (0), ny (0), nz (0), θ (0) + Δd] j [hx (0), hy (0), hz (0), nx (0), ny (0), nz (0), θ (0) -Δd] j For each set of parameter values , The value of the next evaluation function O (i, j) is obtained (step S23).

[0107] As described in the previous 8-point algorithm, if G is a correct value, (m, Gm ') shows a value close to 0. (M, m '
Therefore, it does not become 0 because it contains an error.) Therefore, the smallest value among the obtained 14 evaluation functions O (i, j),
That is, the parameter set having the minimum value is selected (step S24). If the minimum value does not exceed the threshold value in step S25, the process returns to step S22 and the set of parameters is set as a new parameter. For example, when the first set of parameters minimizes the evaluation function, new parameter values can be obtained as follows.

Hx (1) = hx (0) + Δd, hy (1) = hy (0), hz (1) = hz (0), nx (1) = nx (0), ny (1) = ny (0), nz (1) = nz (0), θ (1) = θ (0) Similarly, Δd (≧ 0) is given to the parameter with the minimum evaluation function as follows.

[Hx (1) + Δd, hy (1), hz (1), nx (1), ny (1), nz (1), θ (1)] j [hx (1) -Δd, hy (1), hz (1), nx (1), ny (1), nz (1), θ (1)] j [hx (1), hy (1) + Δd, hz (1), nx (1), ny (1), nz (1), θ (1)] j [hx (1), hy (1) -Δd, hz (1), nx (1), ny (1), nz ( 1), θ (1)] j ・ ・ ・ [hx (1), hy (1), hz (1), nx (1), ny (1), nz (1), θ (1) + Δd] j [hx (1), hy (1), hz (1), nx (1), ny (1), nz (1), θ (1) -Δd] j Obtain the value of the evaluation function for each parameter , Select the parameter set that minimizes the evaluation function. By repeating the above procedure, each element of the translation vector h and the rotation matrix R that minimizes the evaluation function O (i, j) is obtained.

That is, 1 for one shift map
Two motion vectors (translation vector h and rotation matrix R) can be estimated.

Using the equation (18) from the parameters of each shift map and the corresponding motion vector, the relative position of the subject can be calculated from the viewpoint of each point selected in each shift map.

(3) Shape estimation Using the plurality of motion vectors of the image pickup means obtained from the above and the relative positional relationship from the subject of each point selected in each shift map to the position of the image pickup means, each shift is performed. The relative positional relationship from the subject to the position of the image pickup means on the map is converted into the same coordinate space with the Z axis as the line-of-sight direction with the position of one predetermined image pickup means as the origin.

Since each shift map or motion vector contains an error, the relative positional relationship from each image pickup means to the subject is not the same.

Therefore, the relative positional relationship from the image pickup means to the object at each same point converted into the same space is simply averaged to obtain one value, which is used as the estimated shape.

Since the error of the shape obtained by the (relative) shape estimating means is caused by the error of the shift map of the position detecting means, the method of obtaining the shape error by the error of the shift map of the position detecting means will be described. This will be described below.

(4) Error estimation When the shift map is accurately obtained by the position detecting means, for example, the same point on the images picked up at two different positions is the same as the epipolar line (refer to the epipolar line shown in FIG. 18). Reference: Image Analysis Handbook Supervised by The University of Tokyo Press Mikio Takagi, Yosuke Shimoda: p.597 ~ p.598). However, when the shift map includes an error, when the epipolar line is determined based on a certain point, the other point exists at a position apart from the epipolar line as shown in FIG.

Therefore, the epipolar line 1 (L) is ai x + bi y + ci = 0 (21), and the position of one point on the epipolar line is (xi, yi),
The true position on the epipolar line of another point is (xi0 ', yi
0 '), the position of the point deviated from the epipolar line is (x
i ′, yi ′), and the vector from the true position on the epipolar line to the point And

At this time, if the length of the perpendicular line from the position of the point deviated from the epipolar line to the epipolar line is di, Becomes

The vector in the direction of the epipolar line l is Then, the equations (23) to (22) are as follows.

[0120] From the above, the size from the shift map error to the position deviated from the epipolar line can be calculated.

Now, there is basically no correlation between ‖δi‖ and sin θi. That is, for the straight line l, (δxi, δ
There is no correlation between the direction in which yi) appears and the relationship between the size of ‖δ‖ and the angle. Therefore, Then, Here, the error (δxi,
δyi) cannot be obtained. However, Σdi2 (d
Since i2 is the square of di), it can be calculated using equation (2
From 5), the variance σ 0 of di can be obtained.

When di is assumed to be a two-dimensional isotropic normal distribution, σ 1 is set as the variance, and an appropriate value of k 0 is set, The distribution of di is estimated as

As shown in FIG. 19, the position of the point obtained by the position estimating means exists at a position displaced from the epipolar line due to an error in the shift map. Further, assuming that the error of the shift map is a two-dimensional isotropic normal distribution and estimating its variance from the equation (26), an appropriate k1 is set and k1 · σ1 is set as the error distribution of the shift map. And

At this time, a circle having a radius s is drawn from k1 · σ1 and it is assumed that the true position exists in the circle, and the error range of the shape of the object is calculated.

Since the shape obtained by the relative shape estimating means is a relative shape, the corresponding vector of the point on the object is And Further, when r is derived from the equation (6) (the equation (18) is modified). Becomes Further, an equation in which the error of the shift map described above is added to m'of the equation (27) is as follows. However, m0 'is a true vector with no error added,
m ″ is a vector containing only error components.

[0126] At this time, the equation (28) is obtained as follows.

[0127] Differentiating both sides of the equation (30) by θ and setting dr / dθ = 0 and obtaining θ at that time, the maximum and minimum values of the equation (30) can be obtained.

Therefore, the maximum value and the minimum value of the relative position r from the obtained θ to an arbitrary point on the object from the viewpoint are obtained, which is the value of the arbitrary point on the object. It indicates the error range of the shape. Similarly, maximum and minimum values can be obtained for all points on the object.

Further, the error range of the shape obtained as described above is displayed on the display device by the display method as shown in FIG. The left side of FIG. 20 shows the shape of the object, and the right side shows the cross section and the error range of the object.

In the present embodiment, the same point of a plurality of images obtained by moving the image pickup means side with respect to the object is detected, and the relative positional relationship of the image pickup means with respect to the object is detected from the information of the detected same point. The shape of the subject is estimated from the estimated position.

That is, since the same point is detected (determined) on the picked-up image and the relative positional relationship of the image pickup means with respect to the subject is estimated based on the information, the movement of the image pickup means , Even when the subject moves, it is hardly affected by the movement of the subject,
This has the advantage that the relative positional relationship of the imaging means with respect to the subject can be determined.

Further, since the shape of the subject is estimated using the imaged image information, an additional measuring mechanism such as a means for projecting a pattern is not required as the electronic endoscope forming the image pickup means. A normal electronic endoscope can be used (without making the distal end thick).

That is, an ordinary electronic endoscope can be used instead of an electronic endoscope having a thickened tip, and the image processing device 3 is attached to the observation device 7 when measurement is required. By adding it, it becomes possible to display the shape of the subject, and it is possible to construct a system that is highly expandable.

When this image processing apparatus is applied to a medical endoscopy, a portion of interest such as an affected area can be three-dimensionally displayed, so that the degree of swelling of the affected area can be recognized more accurately than before and a diagnosis can be made. It will be effective if.

Furthermore, in the present embodiment, a plurality of images of the same object are taken from different positions, and at the same time the shape of the object is estimated from the obtained images, at the same time, the error occurring in the process of shape estimation is obtained and the error is calculated. Can be reflected in the estimated shape and displayed. Since the shape error due to erroneous calculation that occurs in the shape estimation process is considered rather than displaying only the shape estimated by the conventional method, it is possible to evaluate the reliability of the estimated shape and It is possible to provide certain three-dimensional data.

21 to 24 relate to the second embodiment of the present invention. FIG. 21 is an overall configuration diagram of an electronic endoscope system, FIG. 22 is an explanatory diagram of length measurement by a compound eye endoscope, and FIG. CP
FIG. 24 is a block diagram of image processing configured by U, and FIG. 24 is a block diagram of an image processing apparatus configured by hardware.

The image processing apparatus of the second embodiment uses the 8-point algorithm and the least-squares method to calculate the three-dimensional shape from the relative values calculated by the apparatus of the first embodiment using the image pickup screens obtained from a plurality of positions. While the shape was estimated, the shape is estimated by the absolute value. In order to estimate the shape based on this absolute value, in this embodiment, a compound eye (stereo) is used as an imaging method, and an image obtained by this compound eye and an absolute value of an image pickup (related to) optical system are used, It is designed to estimate the three-dimensional shape. In the present embodiment, the error is estimated by the error from the deviation from the epipolar line as in the first embodiment.

In the image processing apparatus of this embodiment, the absolute shape of the object is estimated by utilizing the parallax, and at the same time, the error range of the shape due to the detection error in the detection of the same point for estimating the shape is estimated, The absolute shape and error range of an object can be displayed simultaneously.

First, an endoscope system including the image processing apparatus according to the second embodiment will be described with reference to the drawings.
An endoscope system 301 shown in FIG. 21 includes an electronic endoscope device 302 that captures an image of a body cavity, an image processing device 303 according to this embodiment, a recording device 304, and a monitor 305.

The electronic endoscope apparatus 302 performs signal processing on the compound eye electronic endoscope 306, the light source apparatus 307 for supplying illumination light to the electronic endoscope 306, and the left image obtained from the electronic endoscope 306. The left image signal processing device 308A, the right image signal processing device 308B that processes the right image obtained from the electronic endoscope 306, and the image signals output from the respective signal processing devices 308A and 308B are displayed. It is composed of observation monitors 309A and 309B.

The electronic endoscope 306 has an elongated and flexible insertion portion 311 so that it can be inserted into the living body 318.
And an operation section 312 provided at the rear end of the insertion section 311.
And a light guide cable 313 and a universal cable 314 extended from the operation unit 312.

The connector 315 at the end of the light guide cable 313 is connected to the light source device 307. The universal cable 314 is branched at the end side, and the connectors 316A and 316B at each end are connected to the left image signal processing device 308A and the right image signal processing device 308B, respectively.

The distal end portion 319 of the insertion portion 311 is provided with a plurality of, for example, two observation windows and an illumination window. Inside the respective observation windows, the right-eye objective lens system 320 and the left-eye objective lens system 3 are provided at positions having parallax with each other.
21 is provided. Each objective lens system 320, 321
At the image forming position of the solid-state image pickup device 32 constituting the image pickup means.
2, 323 are arranged respectively.

A light distribution lens 324 is provided inside the illumination window, and a rear end of the light distribution lens 324 is provided at the rear end of the light distribution lens 324.
A light guide 325 made of a fiber band is arranged. The light guide 325 is provided in the insertion portion 311.
Illumination light is supplied from a light source device 307 to the entrance end face thereof.

Illumination light is transmitted by the light guide 325, and is radiated to the subject through the tip surface and the light distribution lens 324. The reflected light from the subject is formed on the solid-state image pickup devices 322 and 323 as the right image and the left image, respectively, by the objective lens systems 320 and 321.

The image processing apparatus 303 has the same configuration as that of FIG. 3, and therefore, the illustration and description thereof will be omitted, and only different points will be described. The CPU 40 installed in the image processing apparatus 303 of this embodiment has an image processing unit 330 shown in FIG. 23 which operates according to a program stored in the ROM 45. The program loaded in the ROM 45 is different from the program of the first embodiment.

The image processing means 330 is provided for the two left and right images picked up by the image pickup means (stereo).
Distortion aberration correcting unit 331 that corrects so as to eliminate the influence of distortion aberration caused by the optical system, and position detection for detecting the same position between left and right images taken by the left and right image pickup units of the same object, for example. Means 332
And the object obtained from the data obtained by previously measuring the position and parallax of the optical system of the endoscope, and the distance and deviation from the center of the left and right imaging systems as the position obtained by the position detecting unit 332. Shape estimation means 333 for estimating the three-dimensional shape of the object, error estimation means 334 for estimating an error in the three-dimensional shape of the object caused by, for example, the above-mentioned data or a detection error by the position detection means 332, and the shape estimation means 333. It comprises a combining means 335 and the like for combining the estimated three-dimensional shape of the object and the shape error of the error estimating means 334 to create an image for output to a display device or the like.

The left image use signal processing device 308A, 30
The 8B can have, for example, a configuration similar to that of the signal processing unit 7B shown in FIG.

Next, FIG. 24 shows a block configuration example in the case where the arithmetic processing of the image processing apparatus 303 is constituted by hardware.

The image processing apparatus 303 shown in FIG.
RL, GL, B of the left images stored in a plurality of memories (not shown) of the left image use signal processing devices 308A, 308B.
L color signal, right image RR, GR, BR color signal, and synchronization signal S
It is designed to input YNC. These time-series image data (color signals) are input to the image input interface 339.
Is temporarily stored in the memory 340 including a plurality of frame memories. The stored image data is corrected by the distortion aberration correction means 341, and the corrected image data is corrected by the same-point position detection means 342 between the left and right images, the shape estimation means 343, and the error estimation means 344 to obtain the three-dimensional shape of the object. And the error of the three-dimensional shape is calculated. The calculated three-dimensional image data or two-dimensional image data and the stereoscopic error data are combined by the shape / error combining unit 345 and stored in the memory 346.

The error of the three-dimensional image data or the two-dimensional image data stored in the memory 346 and the three-dimensional shape is read out, converted into an analog signal by the D / A converter 347, and the monitor 3 via the input / output interface 348.
A three-dimensional or two-dimensional image of the target region is displayed on 05.

Further, the image processing device 303 includes a distortion aberration correcting means 341, a position detecting means 342, and a shape estimating means 3.
43, error estimating means 344 and shape / error combining means 34
Controller 35 for controlling the calculation of 5
0, and a memory controller 351 for controlling reading and writing of data in the memories 340 and 346 are provided. The selector 352 connects one of the memory 340 or the memory 346 and the recording device 304 to the recording device interface 35 according to a control signal from the memory controller 351.
3, and the recording device 304 and the memory 3 are selectively connected.
40 or the memory 346. The memory controller 351 is connected to the information input device 354. Further, the predetermined data storage means 349 is adapted to supply the predetermined information designated by the information input device 354 to the error estimation means 344.

The operation of the image processing apparatus 303 will be described. The color signals of the left and right images obtained by the imaging means are
Memory 3 through each image interface 339
Stored in 40. The left and right images stored in the memory 340 are corrected by the distortion correction unit 341, and the position detection unit 34 for detecting the position of the same point between the left and right images.
Entered in 2.

The information input device 354 inputs data such as the position and parallax of the optical system of the endoscope, and sends the input data to the storage means 349. In addition, a control signal such as an instruction for starting the shape estimation or the error estimation is transmitted to the information input device 3.
It is emitted from 54 and sent to the memory controller 351.

The detection result of the position detecting means 342 and the predetermined data stored in the storage means 349 are input to the shape estimating means 343 and the error estimating means 344, respectively. The shape estimating unit 343 uses the position data of the same point in the left and right images from the position detecting unit 342 and the data of the position and parallax of the optical system of the endoscope from the storing unit 344 to measure from the viewpoint of the endoscope to the object. Then, the distance is calculated to estimate the shape of the target object.

The error estimating means 344 uses the position detecting means 34.
Position data of the same point of the left and right images from 2 and the storage means 34
The detection error (shift map error) in the same point detection is calculated from the data of the position of the optical system of the endoscope and the parallax from 9 and the distance from the viewpoint of the endoscope to the object is calculated from the detection error. The error in the shape of the target object is estimated by calculating the error.

The three-dimensional shape estimated by the shape estimating means 343 and the shape error estimated by the error estimating means 344 are input to the synthesizing means 345, and data for displaying the three-dimensional shape and its error are created and stored in the memory. 346.

The data stored in the memory 346 is D /
It is sent to the external display device 305 through the A converter 347 and the display interface 348.

Next, the principle of length measurement for calculating the distance from the distal end surface of the endoscope to the designated point in the compound-eye electronic endoscope 306 for shape estimation will be described below.

In FIG. 22, a point D is a point of interest of the target object 328 such as inside the living body 318, and points P and Q are objectives corresponding to the points imaged on the left image capturing means 322 and the right image capturing means 323. It is a point on the focus image of the lenses 320 and 321. Further, a point O is an intersection of the center line of the optical axes of the objective lenses 320 and 321 and the endoscope front end surface 326.

The distance between the point D and the front end surface 326 of the endoscope is d, and the lateral shift of the objective lenses 320 and 321 with respect to the center line of the optical axis is x.

The endoscope front end surface 326 and the objective lens 32.
The distance between the focal planes of 0 and 321 is f, and the objective lens 32
The distance between the optical axes of 0 and 321 is 2s. Further, the distance between the optical axis of the objective lens 320 and the point P is a, and the objective lens 32 is
The distance between the optical axis of 1 and the point Q is b.

From the above, from the similarity of the triangles in FIG. 22, f (s + x) = ad f (s−x) = bd holds. Solving x and d from the above two equations yields x = s (ab) / (a + d) d = 2fs / (a + b). Therefore, since f and s are known quantities, unknown x and d can be obtained. Further, since a and b are calculated as deviations from the target point and the center of the image in each of the left and right images, a and b can be calculated if the same point can be detected in the left and right images. Then, in this embodiment, the same point between the left and right images is detected by the position detecting means as described in the first embodiment, and the specific area in the left image is used as the template image, and the correlation value in the right image is calculated. The position of the same point is detected by obtaining the area where is maximum.

From the detected movement amount of the same point and the parallax of the image pickup means, the distance between the position on the object and the image pickup means corresponding to the same point obtained by the position detection means can be obtained.

Since the same point between images is detected by the position detecting means described in the first embodiment, the absolute shape error obtained in this embodiment is the error estimating means described in the first embodiment. It can be obtained by using a method similar to.

Now, when the positions of the optical systems of the image pickup means for the left and right images can be accurately obtained (assuming that the lenses are not tilted or the depths of focus are different between the left and right lenses), the respective positions are determined. The positional relationship corresponds to the motion parameter of the first embodiment, and the absolute motion parameter can be obtained.

Therefore, the error range of the shift map is estimated as an absolute value using the method of estimating the error of the shift map obtained in the first embodiment. Further, the obtained absolute shape and the error range thereof are displayed on the display device by the display method as shown in FIG. 20 of the first embodiment.

As described above, in the present embodiment, by capturing a plurality of images of the same object by the imaging means having compound eyes, the error range of the shape is estimated in addition to the absolute shape of the object. Since the shape error due to erroneous calculation that occurs in the shape estimation process is taken into consideration, it is possible to obtain more reliable shape data than to display only the absolute shape estimated by the conventional method. Can be provided.

FIGS. 25 to 31 relate to the third embodiment of the present invention, FIG. 25 is an overall configuration diagram of a two-dimensional light spot type electronic endoscope system, and FIG. 26 is a configuration diagram of the tip of an endoscope. 27 is a block diagram of the projection means of the two-dimensional spot light, FIG. 28 is an explanatory view of the length measurement principle in the two-dimensional light spot method,
9 is a block diagram of an image processing apparatus including a CPU, FIG.
Reference numeral 0 is a block diagram of an image processing apparatus configured by hardware, and FIG. 31 is an explanatory diagram relating to detection of spot light projected on an object.

In contrast to the image processing apparatus of the third embodiment, the apparatus of the second embodiment estimates the three-dimensional shape by using the absolute value of the optical system and the like together with the image obtained by the compound eye (stereo). Then, for example, a plurality of light spots arranged two-dimensionally is imaged by an image pickup means having a projection means for projecting as pattern light, and the mutual positions of the plurality of light spots appearing on one image picked up are determined. The detected position is detected and the three-dimensional shape is estimated using the absolute value of the optical system or the like. Incidentally, in the present embodiment, the error is estimated by adopting the method of estimating the error due to the deviation from the epipolar line as in the first and second embodiments, and the shape is estimated from the detection error when detecting the spot light. The error range is estimated.

First, an endoscope system including the image processing apparatus according to the second embodiment will be described with reference to the drawings.
FIG. 25 shows an endoscope system 400, which is the electronic endoscope 6 and the image processing apparatus 3 in the system of the first embodiment shown in FIG.
Instead, it has a measuring electronic endoscope 6A and an image processing apparatus 3A according to the present embodiment. The endoscope system 400 includes the laser light source 4 in addition to the system of the first embodiment.
Has 01. Other configurations and operations similar to those of the first embodiment are designated by the same reference numerals and description thereof will be omitted.

FIG. 26 shows the structure of the distal end portion 16a of the measuring electronic endoscope 6A, in which 402 is an observation window for obtaining an image of the object, 403 is an illumination window for illuminating the object,
Reference numeral 404 is an emission window for irradiating spot light, and 405 is a channel opening.

A light guide fiber (not shown) inserted into the insertion portion 11 of the electronic endoscope 6A transmits the laser light emitted from the laser light source 410 to the tip portion 16a side. On the rear end side of the exit window 404, a transmission type fiber diffraction grating 4 as a projection means shown in FIG.
11 are arranged. The laser light emitted from the emission end of the light guide fiber enters the diffraction grating 411. Further, in the observation window 402,
It has an imaging section 19 in which the objective lens 17 and the CCD 18, which is an imaging element, are arranged in order from the front end side.

The transmission type fiber diffraction grating 411 constitutes an image pickup means together with the image pickup section 19, and a plurality of fibers are arranged so as to be orthogonal to each other vertically and horizontally. The laser light transmitted through the diffraction grating 411 is separated into a plurality of spot lights, and when projected onto a flat screen orthogonal to the optical axis of the laser light, the laser light is regularly arranged in a lattice shape at predetermined intervals. It has become. That is, it becomes a regular two-dimensional spot light. The array of the two-dimensional spot light changes according to the unevenness of the target object. The projection means may irradiate a single light spot as pattern light, or may project the band-shaped light so as to be arranged in a grid pattern.

The CPU 40 mounted in the image processing apparatus 3A of the present embodiment operates according to the program stored in the ROM 45, and the image processing means 4 shown in FIG.
Has 30. The program loaded in the ROM 45 is different from the program of the first embodiment.

The image processing means 430 operated by the CPU 40 according to the program stored in the ROM 45,
It has the following means. That is, the image processing means 43
As shown in FIG. 29A, 0 is a distortion aberration correction unit 431 for correcting the distortion aberration of the image of the two-dimensional spot light irradiated on the target object, the target object and the spot after the distortion aberration correction. The light image is a spot light detecting means 43 in which only the spot light is obtained as an image by threshold processing.
2 and. Since the spot light is generally larger than the brightness value of the target object, as shown in FIG. 29B, an appropriate brightness value is set and only a value larger than the brightness value is extracted from the image, that is, the threshold value. By processing, an image of spot light can be obtained.

In the case of projecting the pattern composed of the grid-like band-shaped light, if the brightness at the cross points of the grid is high, the above threshold processing can be used.

The image processing means 430 also detects the position of the spot light from the image of only the spot light obtained by the spot light detecting means 432.
And data obtained by previously measuring the position of the optical system of the scope, the irradiation direction of the spot light, and the like, and the position detecting means 43.
Shape (distance) estimating means 434 for estimating the distance from the viewpoint to the position of the spot light radiated on the target object by using the result of 3 and the data and position detecting means. Error estimation means 43 for estimating a shape error obtained from the detection error of the spot light position by 433.
5 and the shape and error estimation means 43 in the shape estimation means 434.
The shape error of No. 5 is combined with the combining unit 436 for creating an image to be output to the display device.

Next, FIG. 30 shows an example of a block configuration when the arithmetic processing of the image processing apparatus 3A is configured by hardware. As shown in FIG. 30, the spot light irradiated on the target object and the image of the target object are the image input interface 4
A memory 45 composed of a plurality of frame memories through 50
Stored in 1. The image stored in the memory 451 is corrected by the distortion correction unit 452, and obtained by the spot light detection unit 453 as an image of only spot light.
The image of the spot light obtained by the spot light detection means 453 is input to the position detection means 454 for detecting the position of each spot light.

The information input device 75 inputs data such as the position of the optical system of the endoscope and the irradiation direction of spot light, and sends the input data to the storage means 461. When a control signal such as an instruction for starting the estimation of the shape and its error is generated by the information input device 75, this control signal is sent to the memory controller 463.

The detection result from the position detecting means 454 and the data in the storing means 461 are the shape estimating means 45, respectively.
5, input to the error estimation means 456.

In the shape estimating means 455, the position data of the spot light from the position detecting means 454 and the storage means 461 are stored.
The shape of the target object is calculated by calculating the distance from the viewpoint of the endoscope to the position of the spot light irradiated onto the target object from the data of the position of the optical system of the endoscope and the irradiation direction of the spot light from To estimate.

In the error estimation means 456, the position data of the spot light from the position detection means 454 and the storage means 461 are stored.
Calculate the detection error in the position detection of the spot light from the data such as the position of the optical system of the endoscope and the irradiation direction of the spot light, and irradiate the target object from the viewpoint of the endoscope from the detection error. The error in the shape of the target object is estimated by calculating the error in the distance to the position of the spot light.

The three-dimensional shape estimated by the shape estimating means 455 and the shape error estimated by the error estimating means 456 are respectively input to the synthesizing means 457, and the data for displaying the three-dimensional shape and the error are input. It is created and stored in the memory 458.

The data stored in the memory 458 is D / A.
It is sent to the external display device 5 via the converter 459 and the display interface 460.

The arithmetic processing controller 462 has the distortion correction means 452, the spot light detection means 453, the position detection means 454, the shape estimation means 455, and the error estimation means 456.
Also, the calculation of the synthesizing means 457 is controlled. The memory controller 463 controls the reading and writing of data with respect to the memory 451 and the memory 458.

The selector 464 is the memory controller 46.
One of the memory 451 and the memory 458 is selected so as to be connectable to the recording device 4 through the recording device interface 465 according to the control signal of No. 3, so that data can be transferred between the memory device 4 and the memory 451 or the memory 457. It has become.

Next, the principle of length measurement for calculating the distance from the distal end surface of the endoscope in the measuring electronic endoscope 3A to the specified point for shape estimation will be described below. Figure 2
As shown in FIG. 7, the laser light emitted from the laser light source 410 is incident on the transmission fiber diffraction grating 411, the two-dimensional spot light arranged in a matrix is irradiated to the object, and the two-dimensional spot light is emitted. Pattern is imaged by the imaging means. In the image of the pattern of the two-dimensional spot light, since the intervals of the spots arranged in a matrix change according to the object, the spot light is irradiated by estimating the position of the spot light in this image. The distance to the target object can be obtained.

Therefore, as shown in FIG. 28, φi is the direction of the i-th order spot, θi is the direction in which the i-th order spot is observed, O is the center position of the lens, c is the center position of the image sensor P, and f is Is the focal length of the objective lens 17, A is a point on the object, ZA is the length of a perpendicular line drawn from the point A on the object to a straight line connecting the center O of the lens and the emission position of the spot light, l (L) is the length connecting the center of the lens and the emission position of the spot light, a is the position where the laser light of the i-th order light reaches the image sensor P, and the distance from the center c of the image sensor P to the position a. Let be e.

ZA can be obtained as follows.

[0191] Also, XA is Can be asked as

Here, the direction of the i-th order spot is Becomes However, λ is the wavelength of the laser beam and D is the pitch of the diffraction grating.

Therefore, as shown in FIG. 31, the spot light on the image is obtained by means of threshold processing or the like. In the obtained spot light image, if the shape of the object is not extremely deformed, the spot light is arranged in a predetermined order in a state where the intervals are changed on the matrix. Therefore, it can be known from the order of arrangement of the spot lights that each spot light on the image is a spot of the primary light.

Further, the position of the spot light on the image has the maximum correlation value with the image obtained by the threshold processing by using the template image of the spot light in the position detecting means as described in the first embodiment. The position of the spot light is detected by obtaining the area that becomes. Since the position of the spot light on the image and the spot light of the spot light at that position are obtained, the distance between the spot light projected on the object and the imaging means can be obtained.

Since the position of the spot light on the image is detected by the position detecting means described in the first embodiment, the absolute shape error obtained in this embodiment has been described in the first embodiment. It can be obtained by using the error estimation means.

Now, assuming that the arrangement of the optical system of the image pickup means (focal length, position of the image pickup means, etc.) is accurately obtained, the respective positional relationships correspond to the motion parameters of the first embodiment, and the absolute movement is performed. Parameters can be calculated.

Therefore, the error range of the shift map is estimated as an absolute value using the estimated direction of the shift map error obtained in the first embodiment. Further, the obtained absolute shape and its error range are displayed on the display device by the display method as shown in FIG. 20 of the first embodiment.

As described above, in the present embodiment, the absolute shape of the target object is determined from one or more images obtained by capturing the target object onto which the two-dimensional spot light from the irradiation means is projected by the imaging means. It is possible to estimate the error range of the shape, and it is possible to consider the shape error due to miscalculation that occurs in the shape estimation process, rather than displaying only the absolute shape estimated by the conventional method. More reliable shape data can be provided.

FIG. 32 is an explanatory diagram showing a display example of the estimated shape and the estimation error according to the fourth embodiment of the present invention.

The endoscope system according to the fourth embodiment may be the system according to any of the first, second and third embodiments, and is different only in the method of displaying the estimated shape error. In the first embodiment and the like, the upper and lower limits of the error at the target point are displayed. In contrast, the fourth
In the embodiment, a specific position on the object is set as a reference point, and the error range at positions other than the reference point is displayed together with the shape.

Other than that, since the configuration and the operation similar to those of the first to third embodiments are the same, the drawings and the description will be omitted, and only the different points will be described.

This embodiment is the same as any of the embodiments until the shape of the object is displayed. When displaying the error range of the shape, as shown in FIG. 32, the specific position of the object is set. As the reference position, the range of the shape error at other positions with respect to the reference position is displayed together with the shape of the object.

Specifically, the error range of other points is displayed as a range in which the error obtained at the desired reference point is added to the error of other points.

FIG. 33 relates to a modification of the fourth embodiment.
It is explanatory drawing of the display example of the error of a three-dimensional shape different from an Example.

In the fourth embodiment, the error range of the shape is displayed together with the shape of the object, but in this modification, the error range is displayed numerically as shown in FIG.

In the fourth embodiment and the modified example, it is possible to easily observe a change in shape at a specific position on an object with respect to another position, and at the same time, reliability of the error in the magnitude of the change in shape. Can be easily known.

34 to 36 relate to the fifth embodiment of the present invention. FIG. 34 is a bird's-eye view of the autocorrelation function calculated and the autocorrelation function obtained. FIG. 35 is a relation between similar texture and autocorrelation function. And FIG. 36 is an explanatory diagram showing how the error range is estimated from the autocorrelation function.

The endoscope system of the fifth embodiment obtains the autocorrelation function of the image obtained by the image pickup means and estimates the error range of the shift map from the autocorrelation function.

In the endoscope system of the fifth embodiment, the means in any of the first, second and third embodiments may be used for estimating the three-dimensional shape, and the means for estimating the error relating to the estimated shape is used. Is different. Incidentally, the display format may be any of the formats of the above-mentioned respective embodiments or modified examples.

The system of this embodiment estimates the error of the shift map from the autocorrelation function instead of the error estimating means of the first, second and third embodiments which estimates the error by the deviation from the epipolar line. It has an error estimating means for
In the system of this embodiment, any one of the plurality of images captured in the first embodiment, or the second image
An error is estimated by the method described below using either one of the left and right images in the embodiment and one image in the third embodiment.

That is, in this embodiment, it is possible to estimate the error relating to the three-dimensional shape by using any one of the plurality of images. Therefore, the configuration of the present embodiment can be applied to any of the configurations of the first, second, and third embodiments, and the only difference from the configurations of these embodiments is the error estimation means. Therefore, only differences from the first to third embodiments will be described, and description of similar configurations and operations will be omitted.

As described in the template matching of the first embodiment, the position detecting means of this embodiment uses two different images and specifies the point to be detected as the same point from one of the images so as to be the center. Is a means for obtaining the position of a point such that the correlation value of the correlation function equation is maximized with respect to the other image by taking out the area of the template image.

As for the autocorrelation function, as shown in FIG. 34A, a template image taken out from one image is taken out and a correlation value is obtained for a specific area of the original image. That is, in the search area shown by the broken line in the figure, the autocorrelation function is obtained while moving the template. In this case, assuming that the correlation is obtained using the equation (A), the correlation function equations f and t are the same, that is, the correlation function is obtained in one image.

The autocorrelation function thus obtained, and
FIG. 34B shows a bird's-eye view showing the relationship with the amount of movement when the template image moves in the search area.

Now, let us consider a case in which an autocorrelation function in the image is obtained using a specific region as a template for the texture shown in FIG. If the structure or pattern of the object changes little when the template is moved vertically in the area indicated by the broken line, FIG.
As shown in, there is no difference in the obtained values. As described above, when textures having similar characteristics are present in the vertical direction, for example, detection of position in that direction easily causes a detection error.

As described above, the autocorrelation function corresponds to the feature of the texture in the image extracted as the template and the occurrence frequency of the shift map error. Therefore, as shown in FIGS. 36 (a) to 36 (c), an appropriate threshold value is set, and the peak value of the values of the autocorrelation function exceeding the threshold value and the same point detection by the position detecting means are performed. The range of the value of the autocorrelation function exceeding the threshold value is estimated as the range of the error of the shift map of the detected position while matching the detected position.

In the present embodiment, the range of error of the shift map can be estimated for each point from the shape of the autocorrelation function of the image obtained by the image pickup means.

[Supplementary Note 1] A predetermined calculation is performed based on the information obtained from the image of the object imaged by the image pickup means and the information about the image pickup means, and the three-dimensional shape of the object is estimated using the calculation result. The three-dimensional shape estimation means and the three-dimensional shape error estimation means for estimating an error included in the three-dimensional shape estimated by the predetermined calculation by the three-dimensional shape estimation means.

[Supplementary Note 2] The image processing apparatus according to claim 1, wherein the three-dimensional shape estimating means determines a predetermined portion in the image based on information obtained from the image of the object imaged by the image capturing means. A position detection unit that detects position information is included, and a relative three-dimensional shape of the target object is estimated based on information including the position information obtained by a relative relationship between the target object and the imaging unit. And a relative shape estimating means.

[Supplementary Note 3] The image processing apparatus according to claim 1, wherein the three-dimensional shape estimating means determines a predetermined value in the image based on information obtained from the image of the object imaged by the imaging means. An absolute value shape estimating means for estimating the absolute value three-dimensional shape of the object based on the detected position information and the absolute value relating to the image pickup means is included. Consists of.

[Supplementary Note 4] The image processing device according to claim 1, wherein the three-dimensional shape estimating means is configured to detect the object based on the information obtained by a relative relationship between the object and the imaging means. It comprises a relative shape estimation means for estimating the relative three-dimensional shape of an object.

[Supplementary Note 5] The image processing apparatus according to claim 1, wherein the three-dimensional shape estimating unit obtains information obtained by a relative relationship between the object and the image capturing unit and an absolute value regarding the image capturing unit. The absolute value shape estimating means for estimating the absolute value solid shape of the object based on the target value.

[Supplementary Note 6] The image processing apparatus according to [Supplementary Note 2], wherein the relative position information calculating means is a motion detecting means for detecting a motion of the image pickup means.

[Supplementary Note 7] In the image processing apparatus according to [Supplementary Note 3], the three-dimensional shape estimating means has the position information detected by the position detecting means and information about the image pickup means. The absolute value three-dimensional shape is estimated based on the specified absolute value arrangement information of the constituent elements constituting the.

[Appendix 8] [Appendix 2] or [Appendix 3]
In the image processing device described above, the three-dimensional shape error estimating means calculates an error included in the position information detected by the position detecting means, and estimates an error included in the three-dimensional shape based on the calculation result. .

[Appendix 9] [Appendix 2] or [Appendix 3]
The image processing device according to claim 1, wherein the position detecting unit is a position of a plurality of points in one image captured by the image capturing unit as the predetermined position information, or a position of the same point in the plurality of images. Then, the three-dimensional shape error estimating means obtains a detection error when the position is detected by the position detecting means, and estimates the three-dimensional shape error based on this detection error.

[Supplementary Note 9-1] In the image processing apparatus according to [Supplementary Note 9], the three-dimensional shape error estimating means determines the position by the position detecting means from the maximum error and the minimum error when estimating the shape. Estimate the detection error range at the time of detection.

[Supplementary Note 10-1] The image processing apparatus according to [Supplementary Note 9], wherein the three-dimensional shape error estimating means obtains a detection error when the position is detected by the position detecting means. The deviation amount from the line is calculated, and the error of the three-dimensional shape is estimated based on the deviation amount.

[Supplementary Note 10-2] The image processing apparatus according to [Supplementary Note 9], wherein the three-dimensional shape error estimating means sets a plurality of points in the one image in a specific area set at each point. On the other hand, the autocorrelation function is obtained, and the error of the three-dimensional shape is estimated based on the characteristic value of the autocorrelation function.

[Supplementary Note 10-3] In the image processing apparatus according to [Supplementary Note 9], the three-dimensional shape error estimating means obtains characteristic values for a plurality of points in the one image, and the values are calculated. Based on the above, the error of the three-dimensional shape is estimated.

[Supplementary Note 10-4] In the image processing device according to [Supplementary Note 9], the three-dimensional shape error estimating means detects a characteristic value of the autocorrelation function by setting an appropriate threshold value. Estimate the margin of error.

[Appendix 11] The image processing apparatus according to claim 1, further comprising image synthesizing means for synthesizing the estimated three-dimensional shape and an error included in the three-dimensional shape in a predetermined format.

[Supplementary Note 12] The image processing apparatus according to [Supplementary Note 11], wherein the image synthesizing unit is configured in such a manner that the stereoscopic shape and the estimation error included in the stereoscopic shape do not overlap on the same screen. To synthesize.

[Supplementary Note 13] In the image processing device according to [Supplementary Note 11], the image synthesizing means may form at least a part of the three-dimensional shape and an estimation error included in the shape in the form of an error range. Combine so that they overlap.

[Supplementary Note 14] In the image processing apparatus according to [Supplementary Note 11], the image synthesizing means may calculate an estimation error included in at least a part of the three-dimensional shape in the shape. It synthesizes so that it may overlap in the form of the error range compared with the reference point.

[Supplementary Note 15] In the image processing device according to [Supplementary Note 11], the image synthesizing means calculates, with respect to at least a part of the three-dimensional shape, an estimation error included in the shape as a numerical value. To synthesize.

[Appendix 16] The image processing apparatus according to claim 1, wherein the three-dimensional shape estimation means includes information obtained from an image of the object imaged by the imaging means and arrangement information regarding the imaging means. Based on the above, the relative distance or the absolute value distance between each point on the object and the image pickup means is obtained, and the three-dimensional shape of the object is estimated based on the obtained distance. The error estimation means calculates an error included in the distance obtained by the three-dimensional shape estimation means, based on information obtained from the image of the object imaged by the image pickup means and arrangement information about the image pickup means. The error included in the three-dimensional shape is estimated based on the calculation result.

[Supplementary Note 17] Position detecting means for detecting the position of the same point between images obtained by picking up the same object from a plurality of positions by the image pickup means, and position information of the same point detected by the position detecting means. From the position of the same point of the position detecting means and the movement of the image capturing means of the motion estimating means relative to each other to estimate the relative three-dimensional shape of the object. Geometric shape estimating means and a stereoscopic shape error estimating means for calculating an error included in the distance calculated by the relative shape estimating means and estimating an error range of the relative stereoscopic shape based on the error. ing.

In the image processing apparatus of appendix 17, the same object is imaged by the imaging means from a plurality of positions, and the relative shape of the object is estimated by utilizing the correlation between the images. The error range of the shape due to the error in the distance caused by the detection error in the position detection is estimated.

[Appendix 18] Position detecting means for detecting the position of the same point in a plurality of images having parallax taken by a plurality of image pickup means, and image pickup means including the parallax of the image pickup means Absolute to estimate the absolute value of the distance from the image pickup means to the object from the absolute value information and the position information of the same point by the position detection means, and to estimate the three-dimensional shape in the absolute value of the object from the obtained distance A dynamic shape estimating means and a solid shape error estimating means for calculating an error included in the distance calculated by the absolute shape estimating means and estimating the error range of the solid shape based on the error.

According to the image processing apparatus of Supplementary Note 18, the plurality of images obtained by the plurality of image pickup means from a plurality of positions having a parallax and the parallax of the image pickup means are used for the object. The absolute three-dimensional shape is estimated, and the error range of the three-dimensional shape due to the error in the distance caused by the detection error in detecting the position of the same point is estimated.

[Supplementary Note 19] Image obtained by picking up an object on which the pattern light is projected by an image pickup means having a projection means for projecting one or more pattern lights formed by light having two-dimensionally defined intervals. From the position detection means for detecting the position information of the pattern light, the position information detected by the position detection means and the absolute value information about the imaging means including each irradiation direction of the pattern light, on the object. Obtaining the distance to the irradiated pattern light, and obtaining an error included in the distance obtained by the absolute shape estimating means for estimating the absolute three-dimensional shape of the object from the obtained distance and the absolute shape estimating means. And a three-dimensional shape error estimating means for estimating the error range of the absolute three-dimensional shape based on this error.

In the image processing apparatus according to attachment 19, the absolute three-dimensional shape of the object is utilized by utilizing the position of the pattern light projected on the object on the image, the irradiation direction of the pattern light and the position of the image pickup means. And the error range of the three-dimensional shape is estimated based on the error of the distance caused by the detection error or the like when detecting the position of the pattern light.

[Supplementary Note 19-1] In the image processing apparatus according to [Supplementary Note 19], the projection means projects the pattern light composed of a plurality of strips of light arranged in a grid pattern at a prescribed interval. .

[Supplementary Note 19-2] In the image processing apparatus according to [Supplementary Note 19], the projection means forms one light spot or a plurality of spot lights arranged two-dimensionally at predetermined intervals in the pattern. Project as light.

[Supplementary Note 20] An endoscopic image processing apparatus having the image processing apparatus according to any one of Supplementary Note 17, Supplementary Note 18 or Supplementary Note 19, wherein the imaging means for capturing an endoscopic image and the estimated And a display unit for displaying the three-dimensional shape and the error range synthesized by the image synthesizing unit.

[0247]

According to the present invention, the object is imaged by the image pickup means, the shape of the object is estimated from the obtained image, and the degree of error in the estimated shape is grasped. Therefore, there is an effect that the reliability of the estimated shape can be evaluated.

[Brief description of drawings]

1 to 20 relate to a first embodiment of the present invention, and FIG. 1 is an overall configuration diagram of an electronic endoscope system including the first embodiment.

FIG. 2 is a block diagram showing a block configuration of an electronic endoscope apparatus.

FIG. 3 is a block diagram showing a configuration of an image processing apparatus.

FIG. 4 is a block diagram of image processing configured by a CPU.

FIG. 5 is a block diagram of an image processing apparatus configured by hardware.

FIG. 6 is a block diagram showing a configuration of a recording device.

FIG. 7 is a flowchart showing the processing contents of converting two-dimensional image data into three-dimensional data and estimating an error.

FIG. 8 is an explanatory diagram related to estimation of a three-dimensional shape.

FIG. 9 is an explanatory diagram of correction of distortion.

FIG. 10 is an explanatory diagram showing how a shift amount between a template image and a reference image is calculated.

FIG. 11 is an explanatory diagram showing how a shift map is calculated from a two-dimensional image data string.

FIG. 12 is a flowchart showing a concrete process of iterative processing of a motion vector by calculating a shift map.

FIG. 13 is an explanatory diagram showing a coordinate system of central projection.

FIG. 14 is an explanatory diagram showing a rotation axis and a rotation vector.

FIG. 15 is an explanatory diagram showing that three vectors are on the same plane.

FIG. 16 is an explanatory view showing the movement of the distal end of the endoscope.

FIG. 17 is a flowchart showing a state of iterative estimation.

FIG. 18 is an explanatory diagram of error estimation by an epipolar line.

FIG. 19 is an explanatory diagram regarding an error range of a position deviated from the epipolar line.

FIG. 20 is an explanatory diagram regarding display of an estimated three-dimensional shape and an error range.

21 to 24 relate to the second embodiment, and FIG. 21 is an overall configuration diagram of an electronic endoscope system.

FIG. 22 is an explanatory diagram of length measurement by a compound eye endoscope.

FIG. 23 is a block diagram of image processing configured by a CPU.

FIG. 24 is a block diagram of an image processing apparatus configured by hardware.

25 to 31 relate to the third embodiment, and FIG. 25 is an overall configuration diagram of a two-dimensional light spot type electronic endoscope system.

FIG. 26 is a configuration diagram of the distal end of the endoscope.

FIG. 27 is a configuration diagram of a two-dimensional spot light projection means.

FIG. 28 is an explanatory diagram of a principle of length measurement in the two-dimensional light spot method.

FIG. 29 is a block diagram of an image processing apparatus including a CPU.

FIG. 30 is a block diagram of an image processing apparatus configured by hardware.

FIG. 31 is an explanatory diagram related to detection of spot light projected on an object.

FIG. 32 is an explanatory diagram showing a display example of an estimated shape and an estimated error according to the fourth embodiment.

FIG. 33 is an explanatory diagram of a display example of a three-dimensional shape error different from that of the fourth embodiment according to the modification of the fourth embodiment.

34 to 36 relate to the fifth embodiment, and FIG. 34 is a bird's-eye view of the autocorrelation function calculated and the autocorrelation function obtained.

FIG. 35 is an explanatory diagram showing a relationship between a similar texture and an autocorrelation function.

FIG. 36 is an explanatory diagram showing how the error range is estimated from the autocorrelation function.

FIG. 37 is an explanatory diagram of error occurrence.

[Explanation of symbols]

47 ... Image processing means 50 ... Position detecting means 51 ... Position estimating means 52 ... Shape estimating means 53 ... Averaging processing means 54 ... Error estimating means 55 ... Image synthesizing means

Front page continuation (72) Inventor Tetsuo Nonami 2-43-2, Hatagaya, Shibuya-ku, Tokyo Olympus Optical Co., Ltd. (56) Reference JP-A-6-7289 (JP, A) JP-A-5 -180634 (JP, A) JP-A-63-244011 (JP, A) (58) Fields investigated (Int.Cl. ⁷ , DB name) G01B 11/00-11/30 A61B 1/00 G06T 7/00

Claims (3)

(57) [Claims] 1. Based on information obtained from an image of an object imaged by the imaging means and information on the imaging means,
A three-dimensional shape estimating means for estimating a three-dimensional shape of the object; and a three-dimensional shape estimating means for estimating the three-dimensional shape of the object.
Tables and three-dimensional shape error estimation means, the error estimated by the three-dimensional shape error estimating means for estimating an error of shape that occurs in the process of constant
An image processing apparatus comprising: an output unit that outputs to an indicator . 2. The same object by the image pickup means
For multiple images captured from multiple positions,
Position detecting means for detecting the position of the same point in the image, and on each of the images detected by the position detecting means.
From the same position information
Further comprising a position estimation means for constant, the information about the imaging means, by the position estimation means
The information of the estimated position is characterized in that
The image processing device described . 3. A projector for projecting one or more pattern lights.
The pattern light is projected by the imaging means having steps.
Position information of the pattern light from the image of the captured object.
Position information detecting means for detecting information , position information detected by the position information detecting means, and the pattern.
Absolute value information about the image pickup means including each direction of light emission
By, until the pattern light irradiated on the object
Obtain the distance, and from the obtained distance the absolute three-dimensional shape of the object
And an absolute three -dimensional shape estimating means for estimating
Calculate the output error, and based on this error the absolute three-dimensional shape
A three-dimensional shape error estimating means for estimating an error range and an error estimated by the three- dimensional shape error estimating means are displayed.
An image processing apparatus , comprising: an output unit that outputs to an indicator .