JPH10155110A - Image pickup method, device and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce the processing time by obtaining a parallax between images obtained by means of each image pickup system through the use of values such as a position of each image pickup system of the image pickup device having a plurality of the image pickup systems, its direction, and a focal distance and storing altogether the obtained parallax and the image picked up by the image pickup systems so as to extract corresponding points thereby improving the resulting parallax detection accuracy. SOLUTION: After application of power, a left side image pickup system 101 and a right side image pickup system 102 are initialized and set to a prescribed position, a release button is depressed and a photometry means 107 measures lightness of an object to decide an aperture and a shutter speed. A focus detection means 108 detects the focus of the left side image pickup system 101 and the right side image pickup system 102 so as to allow them to adjust the focus to an object via a system controller 106. The left side image pickup signal and the right side image pickup signal converted respectively into digital signals by A/D converter means 103, 104 are stored respectively in 1st and 2nd process memories 109, 111 and a corresponding point extract means 112 conducts extraction processing.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image pickup method and apparatus for picking up an image and a storage medium used for the image pickup method and apparatus.

[0002]

2. Description of the Related Art Conventionally, for example, Japanese Patent Application Laid-Open No.
A compound eye camera having two imaging systems as shown in JP-A-07-07 is known. Then, image processing is performed on the obtained left and right images. For example, one image process is a parallax extraction process. There is a parallax between the left and right images obtained by imaging the same subject from different viewpoints. If the parallax can be obtained by the processing, the subject can be cut out or the like using the obtained parallax.

Conventionally, in order to obtain parallax from left and right images, left and right images obtained by a compound-eye camera are transferred to a PC (personal computer), and processing for obtaining parallax between the left and right images is performed on the PC. I was Then, the subject is cut out or the like using the parallax.

[0004]

However, in the above-described conventional compound-eye camera, when an image is transferred to a PC and a process of obtaining parallax is performed on the PC, the compound-eye camera at the time of imaging of a focal length or the like is required. Since the situation is not known, there is a problem that the parallax extraction accuracy is deteriorated. In addition, when the aperture and shutter speed of the compound-eye camera are insufficiently set and the corresponding point between the left and right images cannot be accurately obtained, the image cannot be captured again. That is, there is a problem that it is not possible to immediately determine whether or not it is necessary to re-image. In addition, there is a problem that the right and left image pairs may be mistaken. Furthermore, there is a problem that even if the subject is cut out using such low-precision parallax, it is difficult to cut out the subject with high accuracy.

SUMMARY OF THE INVENTION The present invention has been made in view of the above-mentioned problems of the prior art, and a first object of the present invention is to enable a highly accurate parallax to be obtained and correct right and left parallax. It is an object of the present invention to provide an imaging method and apparatus which can obtain a parallax using an image pair and can record the obtained parallax together with an image.

It is a second object of the present invention to obtain a highly accurate parallax, shorten the processing time, and obtain a parallax using a correct pair of right and left images. An object of the present invention is to provide an imaging method and apparatus capable of recording the obtained parallax together with an image.

It is a third object of the present invention to be able to obtain a highly accurate parallax, to obtain a parallax using a correct pair of left and right images, and to obtain a parallax correctly. It is an object of the present invention to provide an imaging method and apparatus which can determine the time and immediately perform re-imaging.

It is a fourth object of the present invention to obtain a highly accurate parallax, shorten the processing time, and obtain a parallax using a correct pair of right and left images. It is an object of the present invention to provide an imaging method and apparatus capable of judging when parallax cannot be obtained correctly and performing re-imaging immediately.

A fifth object of the present invention is to obtain a highly accurate parallax, and to obtain a parallax using a correct pair of right and left images and to cut out a highly accurate subject. It is an object of the present invention to provide an imaging method and apparatus capable of performing the recording and recording information indicating a subject area in an image together with the image.

[0010] A sixth object of the present invention is to enable a highly accurate parallax to be obtained, a parallax to be obtained by using a correct pair of right and left images, and a parallax obtained by extracting corresponding points. The object of the present invention is to provide an imaging method and an apparatus which can select an object, can cut out a subject with high accuracy, and can record information indicating a subject area in the image together with the image. It is.

Further, a seventh object of the present invention is to provide a storage medium capable of smoothly controlling an imaging device.

[0012]

According to a first aspect of the present invention, there is provided an imaging method for imaging by an imaging apparatus having a plurality of imaging systems. A parallax extracting step of obtaining a parallax between images obtained by the respective imaging systems using a value of the focal length, and a parallax obtained by the parallax extracting step and an image captured by the imaging system are collectively stored. And a saving step.

According to a second aspect of the present invention, there is provided an imaging method according to the first aspect, wherein in the parallax extracting step, the position, orientation, and focal length of each imaging system are determined. The value is used to limit the search range of the corresponding point extraction processing.

According to a third aspect of the present invention, there is provided an imaging method for imaging by an imaging apparatus having a plurality of imaging systems.
A parallax extracting step of obtaining parallax between images obtained by the respective imaging systems using the values of the orientation and the focal length, and determining whether or not to retake the image based on the reliability of the parallax obtained in the parallax extracting step And a storing step of collectively storing the parallax obtained in the parallax extracting step and the image captured by the imaging system.

According to a fourth aspect of the present invention, in the imaging method of the third aspect, in the parallax extracting step, the position, orientation, and focal length of each imaging system are determined. The value is used to limit the search range of the corresponding point extraction processing.

According to a fifth aspect of the present invention, there is provided an imaging method according to the fifth aspect, wherein the parallax between images obtained by the respective imaging systems is provided. , An initial nucleus creating step of creating an initial nucleus for subject extraction from the disparity obtained by the parallax extracting step, and an image obtained by region growing the initial nucleus created by the initial nucleus creating step. A subject region extraction step for finding a subject region inside,
A storage step of collectively storing the subject area obtained by the subject area extraction step and the image captured by the imaging system.

According to a sixth aspect of the present invention, there is provided an imaging method according to the sixth aspect, wherein the position, orientation, and focal length of each imaging system are used to obtain an object. A representative parallax extracting step of obtaining the representative parallax up to and a comparing step of comparing the parallax obtained by the parallax extracting step with the representative parallax obtained by the representative parallax extracting step.

According to another aspect of the present invention, there is provided an image pickup apparatus having a plurality of image pickup systems, the position, direction, and focal length of each image pickup system being utilized. It has parallax extraction means for obtaining parallax between images obtained by each imaging system, and storage means for collectively storing the parallax obtained by the parallax extraction means and the image captured by the imaging system. And

In order to achieve the second object,
The imaging device according to claim 8 is the imaging device according to claim 7, wherein the parallax extracting unit includes a position of each imaging system,
It is characterized in that the values of the direction and the focal length are used to limit the search range of the corresponding point extraction processing.

In order to achieve the third object, an image pickup apparatus according to a ninth aspect of the present invention is an image pickup apparatus having a plurality of image pickup systems, wherein the position, direction, and focal length of each image pickup system are utilized. Parallax extracting means for obtaining parallax between images obtained by the respective imaging systems, determining means for determining whether or not to take a picture again from the reliability of the parallax obtained by the parallax extracting means, Storage means for collectively storing the parallax and the image captured by the imaging system.

According to a tenth aspect of the present invention, there is provided an imaging apparatus according to the ninth aspect, wherein the parallax extracting means includes a position, a direction, and a value of a focal length of each imaging system. Is used to limit the search range of the corresponding point extraction processing.

According to another aspect of the present invention, there is provided an image pickup apparatus having a plurality of image pickup systems, comprising: a parallax extracting means for obtaining a parallax between images obtained by the respective image pickup systems. An initial nucleus creating unit that creates an initial nucleus for subject extraction from the parallax obtained by the parallax extracting unit; and growing the initial nucleus created by the initial nucleus creating unit to obtain a subject region in the image. It is characterized by comprising a subject area extracting means, and a storage means for collectively storing the subject area obtained by the subject area extracting means and an image taken by the imaging system.

According to a twelfth aspect of the present invention, there is provided an imaging apparatus as defined in the twelfth aspect, wherein the position, orientation, and focal length of each imaging system are used to reach a subject. A representative parallax extracting means for obtaining the representative parallax of the above, and a comparing means for comparing the parallax calculated by the parallax extracting means with the representative parallax calculated by the representative parallax extracting means.

In order to achieve a seventh object, a storage medium according to claim 13 is a storage medium for storing a program for controlling an image pickup apparatus having a plurality of image pickup systems, wherein a position of each image pickup system, Orientation, a parallax extraction module that calculates the parallax between images obtained by each imaging system using the value of the focal length, and the parallax obtained by the parallax extraction module and the image captured by the imaging system are collectively obtained. A program having a storage module for storing is stored.

In order to achieve the seventh object,
According to a fourteenth aspect of the present invention, in the storage medium according to the thirteenth aspect, in the parallax extraction module, values of a position, an orientation, and a focal length of each imaging system are used to limit a search range of the corresponding point extraction processing. Features.

In order to achieve the seventh object,
The storage medium according to claim 15, which is a storage medium for storing a program for controlling an image pickup apparatus having a plurality of image pickup systems, wherein each of the image pickup systems uses a value of a position, an orientation, and a focal length of each image pickup system. A disparity extraction module for obtaining the disparity between the obtained images, a determination module for determining whether or not to retake the image based on the reliability of the disparity obtained by the disparity extraction module; and a disparity and imaging performed by the disparity extraction module. A storage module for collectively storing images captured by the system.

In order to achieve the seventh object,
The storage medium according to claim 16 is the storage medium according to claim 15, wherein the parallax extraction module uses the values of the position, orientation, and focal length of each imaging system to limit the search range of the corresponding point extraction processing. Features.

Further, in order to achieve the seventh object,
18. The storage medium according to claim 17, wherein the storage medium stores a program for controlling an image pickup apparatus having a plurality of image pickup systems, wherein a parallax extraction module for obtaining a parallax between images obtained by each image pickup system; An initial nucleus creation module for creating an initial nucleus for subject extraction from the parallax obtained by the extraction module, and a subject area extraction for obtaining a subject area in an image by growing the area of the initial nucleus created by the initial nucleus creation module A storage module that collectively stores a subject area obtained by the subject area extraction module and an image captured by an imaging system.

Further, in order to achieve the seventh object,
The storage medium according to claim 18, wherein the representative parallax extraction module for obtaining a representative parallax to a subject using values of a position, an orientation, and a focal length of each imaging system, and the parallax extraction. A comparison module that compares the parallax obtained by the module and the representative parallax obtained by the representative parallax extraction module.

[0030]

Embodiments of the present invention will be described below with reference to the drawings.

(First Embodiment) First, the first embodiment of the present invention will be described.
The embodiment will be described with reference to FIGS.
FIG. 1 is a block diagram showing the configuration of the imaging device according to the first embodiment of the present invention. In the figure, 101 is a left imaging system, 102 is a right imaging system, 103 is a left imaging system 10
A / D converter 104 for converting the image signal obtained by step 1 into a digital signal, and an A / D converter 104 for converting the image signal obtained by the right imaging system 102 into a digital signal.
D conversion means, 105 is a color signal processing means for generating luminance and color signals from RGB signals from a CCD (imaging element) (not shown) of the imaging system, 106 is a system controller, 107
Is a photometric unit that measures the brightness of the subject and determines an aperture value and a shutter speed according to the photometric value; 108, a focus detection unit that adjusts the focus of an imaging system; 109, a first process memory that holds an image , 110 is a second process memory for holding an image, 111 is a work memory used for corresponding point extraction, and 112 is a corresponding point extracting means for performing corresponding point extraction processing.

Next, the operation of the imaging apparatus according to this embodiment will be described with reference to the flowchart of FIG. First, when the power of the apparatus is turned on (ON) in step S201, the left imaging system 101 and the right imaging system 1 are set in the next step S202.
02 is performed. The respective lenses, the convergence angle, and the base line length are controlled so that each of them is located at a predetermined position. Further, since the initial position at that time becomes clear, even when the focal length or the like is changed, the value can be accurately determined from the number of pulses sent to a stepping motor (not shown). Next, when a release button (not shown) is pressed in step S203, in the next step S204, the photometric unit 1 is pressed.
At 07, the brightness of the subject is measured, and the aperture value and the shutter speed are determined according to the measured light value. Next, in step S205, the focus of the left imaging system 101 and the right imaging system 102 is adjusted with respect to the subject by the focus detection unit 108. These controls are performed via the system controller 106. The left imaging system 101 and the right imaging system 102 simultaneously image the subject with the aperture value and shutter speed obtained by the photometry unit 107 in step S206. I do. An image signal obtained by imaging the subject with the left imaging system 101 is a left A / A signal.
The signal is converted into a digital signal by the D conversion means 103. An image signal obtained by imaging the subject with the right imaging system 102 is converted into a digital signal by the right A / D conversion unit 104.

The right image signal converted into a digital signal by the right A / D converter 104 is held in the first process memory 109. A / D conversion means 1 for the left side
03 is converted to a left image signal in step S207.
Are processed by the color signal processing means 105 and stored in the second process memory 109. When that process ends,
The right image signal stored in the first process memory 109 is processed by the color signal processing unit 105 in step S207, and is stored again in the first process memory 109.

Next, at step S208, the corresponding point extracting means 1
A corresponding point (parallax) is obtained by performing a corresponding point extraction process using the processing of FIG. In order to obtain a corresponding point in the corresponding point extracting means 112, the right image signal held in the first process memory 109 and the left image signal held in the second process memory 109 are respectively processed by the work memory 111.
And a corresponding point extraction process is performed. At the same time, it is determined from the result in step S209 whether re-imaging is to be performed (whether to take a new image). If re-imaging is to be performed, a signal is output to the system controller 106 to set a flag for performing re-imaging. In step 209, the system controller 106 determines whether or not a flag for performing re-imaging is set. If the flag is set, the first process memory 109, the second process memory 110, and the work memory 111 are released. The process returns to step S203.

If the flag for performing re-imaging is not set in step 209, the parallax, which is the result of the corresponding point extraction held in the work memory 111, is used in step 210 when the corresponding point extraction processing is performed. Assuming that an image based on the right image is used as a reference, for example, the right image signal is held in the first process memory 109 in which the right image signal is held. Next, in step 211, the right image signal and the parallax as information indicating the attribute of the image are stored together.

Next, the corresponding point extracting process performed by the corresponding point extracting means 112 will be described. Corresponding point extraction refers to finding a corresponding point between a left image and a right image, and the difference in position between the corresponding pair of images corresponds to parallax. A method of performing the corresponding point extraction processing is called a corresponding point extraction method, and there are a plurality of types. Here, a general template matching method is used.

FIG. 3 shows the template matching method.
As shown in the figure, one point P at which a corresponding point in the right image 302 is to be obtained.
Consider a template 303 that surrounds
03 is moved in the left image 301, and the similarity at each point is calculated to determine a corresponding point.

As a typical evaluation function for obtaining the similarity, there is a function using a difference between luminance values as shown in the following equation (1).

[0039]

(Equation 1) In the above equation (1), F (i, j) represents the template 303 created from the right image 302, and A (i, j) represents the left image 3
01 is shown. That is, the above equation (1) indicates the similarity when the position of the template 303 is moved by (v, h) on the left image 301. Note that when the above equation (1) is used, the point at which E (v, h) becomes the minimum is the corresponding point, and the minimum value of E (v, h) becomes 0 in theory.

Next, the corresponding point extracting process based on the right image will be described with reference to the flowchart of FIG.
First, in step S401, the corresponding point extracting unit 112 selects one point in the right image, and notifies the system controller 106 of the coordinate value. Next, in step S402, the system controller 106 obtains a template area centered on the obtained coordinate values from the first process memory 109, and transfers it to the work memory 111 as a template. Next, at step S403, step S401 is performed.
Is obtained from the second process memory 110, and is transferred to the work memory 111 as the search range.

Next, at step S404, the work memory 11
3 is moved within the search range on the work memory 111 as described with reference to FIG. 3, and at each position, an evaluation function value is obtained by using the above equation (1). And At the same time, the reliability of the corresponding point is obtained.

FIG. 5 shows the evaluation function (residual sum) of the above equation (1).
FIG. Since the search range is actually two-dimensionally spread, the distribution of the evaluation function is actually two-dimensional.
It is a dimension.

FIG. 5 (a) schematically shows the distribution of the sum of the residuals when the corresponding points are obtained by using the feature points on the subject as a template and focusing accurately. If the focus is accurate, the exposure is correct, and the subject has a texture,
As shown in FIG. 5 (a), the sum of the residuals becomes close to 0 at the corresponding point, and other than that point, the sum of the residuals becomes large. However, when the focus is not accurate or the exposure is over-exposed or under-exposed, the distribution of the residual sum becomes smooth as a whole as shown in FIG. Become. Similarly, when the texture of the template area has no texture, the distribution of the residual sum is as shown in FIG. 5B. In such a case, it may be determined that there is no reliability of the corresponding point, so that the value of the evaluation function and the second minimum value of the evaluation function are obtained, and there is a difference between the evaluation function values of those points. If it is less than the threshold, it is determined that there is no reliability. Also, when a plurality of textures similar to the texture of the template exist in the search range, the difference between the evaluation function value at the point where the evaluation function value is the smallest and the evaluation function value at the second smallest point is equal to or less than a threshold value. At this time, since the reliability of the corresponding point is low, there is no problem even if it is determined that there is no reliability.

Next, in step S405, it is determined whether or not corresponding points have been obtained at all points to be obtained on the right image. If not, the process returns to step S401. If it is determined in step S405, the next step S4
Proceed to 06. In this step S406, the aforementioned step S
It is determined whether to perform re-imaging based on the reliability obtained in 404. Just because there is a point of unreliability,
That's not all out of focus or improper exposure. For example, as described in the step S404, the influence of the texture may occur. Accordingly, among the corresponding points obtained in the right image, the number of corresponding points having no reliability is obtained. If the number is more than half, it is determined that the focus is out of focus or the exposure is inappropriate. If the number of corresponding points having no reliability is more than half, that is, if re-imaging is to be performed, the system controller 106 is executed in step S407.
After outputting a signal to set a flag for performing re-imaging, the present processing operation ends. As described in step S209 in FIG. 2, when this flag is set, re-imaging is performed. In step S406, if the number of corresponding points having no reliability is less than half, that is, if re-imaging is not performed, step S407 is skipped and the present processing operation ends.

Next, the estimation of the search range will be described with reference to FIGS. 6, reference numeral 601 denotes a principal point of the left imaging system 101, 602 denotes a principal point of the right imaging system 102, 603 denotes a subject, 604 denotes one point P on the subject 603, 605 denotes an epipolar surface, 606 denotes a left image, and 607 denotes a left image. Epipolar line of left image 606, right image 608
09 is an epipolar line of the right image 608, 610 is an optical axis of the left imaging system 101, and 611 is an optical axis of the right imaging system 102.

The epipolar plane 605 and the one on the subject 603
This is a plane that connects and cuts the point P and the principal points 601 and 602. The epipolar line is a line where the epipolar plane 605 intersects the left image 606 and the right image 608. That is, one point P604 on the subject 603 in the right image 608
Is always on the left epipolar line 607. When one point in the right image 608 is determined, the epipolar line can be determined from the coordinates of the point, the focal length and the convergence angle of the left imaging system 101 and the right imaging system 102, and the rotation of each imaging system. For example, as shown in FIG. 7, the optical axis 610 of the left imaging system 101 and the optical axis 611 of the right imaging system 102 are parallel, there is no rotation around the optical axis, and the positions of the left imaging system 101 and the right imaging system 102 Assuming that the difference in the direction is only a parallel movement of the image only in the horizontal direction, both epipolar lines 607 and 608 are on a straight line. Therefore, it can be seen that the corresponding point of the point Qr on the right epipolar line can be obtained by moving the template on the left epipolar line 607.

Considering the case where the subject 603 is at infinity, when the imaging system is arranged as shown in FIG. 7, the position of the subject 603 in the left image 606 and the position in the right image 608 are mutually different. Matches. That is, the parallax becomes zero. When the subject 603 is at a finite distance, the subject 60
3 is the left image 607 from the position where it is imaged in the right image 608.
The position where the image is taken in is on the right side. That is, if the corresponding point of the point Qr in the right image 608 in FIG.
Assuming that the object 603 is at infinity, the point Ql1 in the left image 607 is obtained. If the object 603 is at a finite distance, the object 603 exists on the right side of the point Q11. From the distance of the subject 603 and the epipolar line, the point Q
It can be seen that the range for searching for the corresponding point of r only needs to be the shaded portion in the left image 607 of FIG. These can all be determined by information indicating the state of the imaging system of the compound-eye camera. The reason why the hatched portion in FIG. 7 has a larger area than the epipolar line is because errors in information from the camera are taken into account. Predicting the search range in this way,
By transferring only that part of the image, the amount of transfer is smaller than that of transferring the entire image, and the transfer time can be reduced. Similarly, since the search range is small, the processing time can be reduced.

Next, the storage of the right image and the parallax will be described. When finding a corresponding point, a search is performed in two dimensions in the v direction and the h direction, as can be seen from equation (1). That is, the position of the corresponding point is shifted in the v direction and the h direction with reference to the coordinates of the center point of the template. Therefore, in this description, it is assumed that the parallax has the v direction and the h direction. Then, for example, in FIG. 8, when FIG. 8A is the right image,
When the corresponding points are obtained at all points in the right image, FIG. 8B showing the amount of parallax in the v direction of the corresponding points of each point in the right image and FIG. 8C showing the amount of parallax in the h direction are shown in FIG. it can.

FIGS. 8B and 8C show that the image has the same size and the same parallax as that in the image. That is, when the parallax is stored together with the image as an attribute of the image, FIGS.
All of (c), that is, for example, the (i, j) -th element in the image is stored so as to have r, g, b values indicating luminance values and h, v values indicating parallax.

In FIGS. 8 (b) and 8 (c), arrows are shown instead of numbers, and the arrow in the v direction is upward and the arrow in the h direction is right. The length of the arrow indicates the amount of parallax. In such a case, one of the methods for storing the right image and the parallax is as follows.
As shown in FIG. 9A, there is a method of storing each of the right image, the v-direction parallax, and the h-direction parallax as one image. In this case, this method is convenient when displaying the right image or displaying the parallax in the h direction, and when looking at the same position from the origin O serving as the reference, the pixel value also becomes the parallax in the v direction. Both the value and the parallax value in the h direction are known. FIG. 9 (b)
There is also a method of storing a set of the pixel value, the disparity value in the v direction, and the disparity value in the h direction as shown in FIG. In this case, it is convenient to see the parallax value of one point in the right image.

The reason described with reference to FIGS. 8 and 9 is as follows.
FIG. 10 shows a case where the corresponding points are obtained in all the right images.
In some cases, parallax is obtained only from the representative points indicated by black points in FIG. In that case, the method of FIG. 9 described above may be used, but as shown in FIG. 10B, the coordinates (i,
It is efficient to store j) and the v-direction parallax and the h-direction parallax at that position. Further, the parallax of the corresponding point in the v direction can be set to 0 or regarded as 0 depending on the installation of the compound-eye camera. Therefore, there is also a method of storing only the right image and the parallax in the h direction.

As described above, in the compound-eye camera having the corresponding point extracting means, the search range can be limited by using the parameters such as the focal length of the camera, so that the parallax detection accuracy is improved and the processing time is shortened. It becomes possible. Further, an instruction to retake can be immediately given from the parallax detection result. Furthermore,
Since the obtained parallax is stored together with the image, it is possible to easily perform image processing using the parallax.

(Second Embodiment) Next, a second embodiment of the present invention will be described with reference to FIGS. In the present embodiment, a subject is extracted using the parallax used by the corresponding point extracting means.

First, the principle will be briefly described with reference to FIG. FIG. 11 schematically shows subject extraction. FIG. 11A shows a right image 1101 and an initial nucleus 1102.
FIG. 11B shows a right image 1101 and a mask 1104 (hatched portion). Using the parallax obtained from the corresponding point extracting means, a point considered to be a point on the subject 1103 is defined as an initial nucleus 1102. And
The initial nucleus 1102 is grown in area to finally obtain a mask 1104 showing the subject 1103. This is a simple flow of subject extraction.

The initial nucleus 1102 sets the flag at the same position as in the image when the image has the same size and the pixels in the image are considered to be pixels on the subject 1103. Whether or not the pixel is on the subject 1103 is determined from the parallax, and the parallax is calculated from the approximate distance of the subject 1103.
It is determined by comparing them. Then, the flag is extended to the surroundings (area is grown) and the mask 1104 is expanded.
Get.

FIG. 12 is a block diagram showing the configuration of an image pickup apparatus according to the second embodiment of the present invention. In FIG. 12, the same reference numerals are used for the same parts as those in FIG. 1 in the first embodiment. Is attached. 12 differs from FIG. 1 in that a subject extracting unit 1101 is added to the configuration in FIG. 1, and the other configuration is the same as that in FIG. The subject extracting unit 1101 extracts a subject from images captured by the imaging systems 101 and 102 using the parallax obtained by the corresponding point extracting unit 112.

Next, the processing procedure of the imaging apparatus according to the present embodiment will be described with reference to the flowchart of FIG. First, when the power of the apparatus is turned on (ON) in step S1301, the left imaging system 101 and the right imaging system 102 are initialized in the next step S1302. The respective lenses, the convergence angle, and the base line length are controlled so that each of them is located at a predetermined position. Further, since the initial position at that time becomes clear, even when the focal length or the like is changed, the value can be accurately determined from the number of pulses sent to a stepping motor (not shown). Next, step S1303
When a release button (not shown) is pressed, the brightness of the subject is measured by the photometric unit 107 in the next step S1304, and the aperture value and shutter speed are determined according to the photometric value. Next, in step S1305, the focus of the left imaging system 101 and the right imaging system 102 is adjusted by the focus detection unit 108 with respect to the subject. Note that these controls are performed via the system controller 106, and in step S1306, the photometric unit 1
The left imaging system 101 and the right imaging system 102 simultaneously image the subject, respectively, with the aperture value and the shutter speed obtained by 07. An image signal obtained by imaging the subject with the left imaging system 101 is converted into a digital signal by the A / D conversion means 103 for the left. An image signal obtained by imaging the subject by the right imaging system 102 is converted into a right A / D converter 104.
Is converted into a digital signal.

The right image signal converted into a digital signal by the right A / D converter 104 is held in the first process memory 109. A / D conversion means 1 for the left side
03 is converted to a left image signal in step S130.
In step 7, the signal is processed by the color signal processing unit 105 and stored in the second process memory 109. When the processing is completed, the right image signal held in the first process memory 109 is output to the color signal processing unit 105 in step S1307.
And stored again in the first process memory 109.

Next, in step S1308, a corresponding point extraction process is performed by the corresponding point extracting means 112 to obtain a corresponding point (parallax), and an initial nucleus is obtained using the parallax. In order to obtain a corresponding point in the corresponding point extracting means 112,
The right image signal held in the first process memory 109 and the left image signal held in the second process memory 109 are held in the work memory 111, respectively.
Corresponding point extraction processing is performed. Then, from the value of the parallax, it is determined whether or not the point exists on the subject, as described later in detail with reference to FIG. At the same time, it is determined in step S1309 whether re-imaging is to be performed (re-imaging) based on the result. If re-imaging is to be performed, a signal is output to the system controller 106 to set a flag for performing re-imaging. In step 1309, the system controller 106 determines whether or not a flag for performing re-imaging is set. If the flag is set, the first process memory 109, the second process memory 110, and the work memory 111 are released. The process returns to step S1303.

If the flag for re-imaging is not set in step 1309, step 131
The initial kernel of the corresponding point extraction result held in the work memory 111 at 0 is a right image signal, assuming that the image used as the reference when the corresponding point extraction processing is performed, for example, the right image is used as the reference. And the right image signal in the first process memory 109. Next, in step 1311, the work memory 111 is released.
The right image signal and the initial kernel held in the process memory 109 are held in the work memory 111. Then, in step 1313, the right image signal and the initial nucleus stored in the work memory 111 are extracted by the subject extracting means 1101. Then, the result is stored in the first process memory 109.
And the right image signal. Next, in step 1314, the right image signal and the subject signal are stored as attributes of the image.

Next, the corresponding point extracting process performed by the corresponding point extracting means 112 will be described. Basically, the first
This embodiment is the same as the embodiment described above, but different points will be described.

FIG. 14 is a flowchart showing the procedure of the corresponding point extraction processing based on the right image. First, in step S1401, camera parameters such as a subject distance used for autofocus are obtained from the system controller 106, and an approximate parallax value of the subject is obtained using the values. Specifically, the subject distance used for autofocusing is z, the focal length at that time is f,
Assuming that the pixel pitch of the CD (image sensor) is p and the base line length is b, the approximate parallax value d of the subject can be obtained by the following equation (2).

D = (f · b) / (z · p) (2) Next, in step S1402, the corresponding point extracting unit 112 selects one point in the right image and notifies the system controller 106 of the coordinate value. I do. Next, in step S1403, the system controller 106 obtains a template area centered on the obtained coordinate value from the first process memory 109, and transfers it to the work memory 111 as a template. Next, in step S1404, step S1 is performed.
An area where it is predicted that a point corresponding to the point selected in 402 exists is obtained from the second process memory 110, and is transferred to the work memory 111 as a search range. The prediction of the search range is as described in the first embodiment.

Next, in step S1405, the work memory 1
The template on 11 is moved in the search range on the work memory 111, and an evaluation function value is obtained at each position using the above equation (1), and a point having the minimum value is set as a corresponding point. At the same time, the reliability of the corresponding point is obtained. Next, step S
The parallax dd of the corresponding point obtained in 1406 and the aforementioned step S1
The absolute value of the difference with the approximate parallax d of the subject obtained in 401 is obtained, and if the value is smaller than a certain value, it is discarded as the initial kernel of the subject, and if it is larger, it is not the subject. Next, in step S1407, it is determined whether corresponding points have been obtained at all points to be obtained on the right image. If not, the process returns to step S1402. If it is determined in step S1407, the next step S1
Proceed to 408. In step S1408, it is determined whether or not to perform re-imaging based on the reliability obtained in step S1406. If the number of corresponding points having unreliability is half or more, that is, if reimaging is to be performed, a signal is output to the system controller 106 in step S1409, a flag for performing reimaging is set, and the processing operation is terminated. In step S1408, if the number of corresponding points having no reliability is less than half, that is, if re-imaging is not performed, step S1409 is skipped, and the process ends.

Next, the processing of the subject extracting means 1101 will be described with reference to the flowchart of FIG. Since the extraction of corresponding points is not good at places where there is no texture around the contour of the subject, if the subject is extracted only from the initial nucleus obtained by the corresponding point extracting means 112, the subject cannot be extracted neatly. As shown in FIG. 11, since the initial nucleus tends to be smaller than the subject, it is necessary to grow the region to the contour of the subject and extract it as a mask. Therefore, the subject extracting means 1101 obtains a mask by growing the area based on the initial nucleus and obtains a mask, thereby extracting the subject neatly. Specifically, the similarity of the image feature amount between the initial nucleus and its neighboring pixels is determined, and the mask is obtained by growing the initial nucleus using the result. That is, if the similarity is higher than a certain threshold value, the region is regarded as the same subject region and the region is grown. A description of the region growing process is omitted, and the flow of the process will be described with reference to FIG.

First, in step S1501, edges in the image of the right image signal are extracted. So to extract edges
Operators such as bel and Roberts are used. This is to make the region growth stop at the contour of the subject.
Next, in step S1502, one point is extracted from the initial nucleus.
Next, in step S1503, one point is extracted from the eight neighborhoods. Next, in step S1504, the process proceeds to step S1503.
It is determined whether the extracted point is on the edge or on the initial nucleus or mask, and if it is, the process returns to step S1503. If it is not a point on the edge, a point on the initial nucleus, or a point on the mask, the process advances to step S1505.

In step S1505, using the threshold value Ith of the luminance level, the values of the RGB components of pixels near eight and the similarity are determined. Next, in step S1506, the similarity determination is performed with the hue values near eight using the hue threshold value Hth. Next, in step S1507, when the difference absolute value of each of RGB is smaller than the threshold value Hth, the pixel is merged with the initial nucleus, and the region is grown as a mask. Next, in step S1508, it is determined whether or not the processing has been performed for all the eight neighboring points.
Return to 503. If the processing has been performed at all points in the vicinity of eight, it is determined in step S1509 whether or not the processing has been performed at all points of the initial nucleus. If not performed, the flow returns to step S1502. If the process has been performed at all points of the initial nucleus, step S1
In step 510, it is determined whether or not there is a pixel whose area is to be grown. If not, the process returns to step S1502. If there is, the process ends.

As described in detail above, in the compound eye camera having the corresponding point extracting means and the subject extracting means, the search range can be limited by using parameters such as the focal length of the camera, and the parallax detection accuracy can be improved and In addition, the corresponding point extraction processing time can be reduced. Then, an initial nucleus for subject extraction can be set based on the obtained parallax of each pixel and the parameters of the compound-eye camera, and highly accurate subject extraction can be performed by region growth. In addition, there is an effect that an instruction to retake can be given immediately from the parallax detection result.

(Third Embodiment) Next, a storage medium used in the image pickup apparatus of the present invention will be described with reference to FIGS.

As shown in FIG. 16, at least a “parallax extraction module” and a “storage module” are stored in a storage medium for storing a program for controlling an imaging apparatus that captures images having parallax from each other by a plurality of imaging systems. What is necessary is just to store the program code of a module.

Here, the "parallax extraction module" is a program module for obtaining the parallax between images obtained by each imaging system using the values of parameters such as the position, orientation, and focal length of each imaging system. . The “storage module” is a program module for collectively storing the parallax obtained by the “parallax extraction module” and the images captured by the respective imaging systems.

Further, as shown in FIG. 17, at least a “parallax extraction module”, a “judgment determination module”, and a “ module",
What is necessary is just to store the program code of each module of the "storage module".

Here, the "parallax extraction module" is a program module for obtaining the parallax between images obtained by each imaging system using the values of parameters such as the position, orientation, and focal length of each imaging system. . The “judgment module” is a program module for judging whether or not to retake an image based on the reliability of the parallax obtained by the “parallax extraction module”. The “storage module” is a program module for collectively storing the parallax obtained by the “parallax extraction module” and the images captured by the respective imaging systems.

As shown in FIG. 18, at least another parallax extracting module, a parallax extracting module and a parallax extracting module are stored in another storage medium for storing a program for controlling an image pickup apparatus for picking up images having parallax from each other by a plurality of image pickup systems. What is necessary is just to store the program code of each module of the "initial nucleus creation module", "subject region extraction module", and "save module".

Here, the “disparity extraction module” is a program module for obtaining the disparity between images obtained by each imaging system. The “initial nucleus creation module” is a program module for creating an initial nucleus for subject extraction from the parallax obtained by the “parallax extraction module”. "Subject area extraction module"
Is a program module for obtaining a subject area in an image by growing a region of the initial nucleus obtained by the "initial nucleus creation module". The “save module” is a program module for collectively saving the subject area obtained by the “subject extraction module” and the images captured by the respective imaging systems.

[0076]

As described above in detail, according to the imaging method and apparatus according to the first and seventh aspects of the present invention, it is possible to obtain a high-precision parallax, and to correct the parallax using a correct pair of left and right images. It is possible to obtain the parallax and to store the obtained parallax together with the image.

According to the imaging method and apparatus according to the second and eighth aspects of the present invention, a highly accurate parallax can be obtained, the processing time of corresponding point extraction can be shortened, and a correct left and right image can be obtained. The parallax can be obtained using the pair, and the obtained parallax can be stored together with the image.

According to the third and ninth aspects of the present invention, a highly accurate parallax can be obtained, and a parallax can be obtained by using a correct pair of right and left images. It is possible to determine when parallax is not successfully obtained, to immediately perform re-imaging, and to store the obtained parallax together with an image.

Further, according to the imaging method and apparatus according to the fourth and tenth aspects of the present invention, a highly accurate parallax can be obtained, the processing time of corresponding point extraction can be shortened, and correct right and left parallax can be obtained. The parallax can be obtained by using the image pair, and when the parallax cannot be obtained well, the re-imaging can be performed immediately, and the obtained parallax can be stored together with the image.

Further, according to the storage medium of the present invention, there is an effect that the above-described imaging device can be smoothly controlled.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of an imaging device according to a first embodiment of the present invention.

FIG. 2 is a flowchart showing a flow of processing of the imaging apparatus.

FIG. 3 is an explanatory diagram showing a corresponding point extraction method of the imaging device.

FIG. 4 is a flowchart showing a processing flow of a corresponding point extracting means of the imaging apparatus.

FIG. 5 is a diagram showing a change in a residual sum of the imaging apparatus.

FIG. 6 is a diagram showing an epipolar line and the like of the imaging apparatus.

FIG. 7 is a diagram showing a search range of the imaging device.

FIG. 8 is a diagram showing an image and parallax of the imaging device.

FIG. 9 is a diagram showing a method for storing an image and parallax of the imaging apparatus.

FIG. 10 is a diagram showing a method for extracting corresponding points of representative points and storing the results of the imaging apparatus.

FIG. 11 is a diagram showing an outline of subject extraction by the imaging apparatus.

FIG. 12 is a block diagram illustrating a configuration of an imaging device according to a second embodiment of the present invention.

FIG. 13 is a flowchart showing a processing flow of the imaging apparatus.

FIG. 14 is a flowchart showing a flow of processing of a corresponding point extracting means of the imaging apparatus.

FIG. 15 is a flowchart showing a flow of processing of a subject extracting unit of the imaging device.

FIG. 16 is a diagram showing a module of a program code stored in a storage medium used in the imaging apparatus of the present invention.

FIG. 17 is a diagram showing a module of a program code stored in a storage medium different from that of FIG. 16 used in the compound eye imaging apparatus of the present invention.

18 and 17 used in the imaging apparatus of the present invention.
FIG. 3 is a diagram showing modules of a program code stored in a storage medium different from FIG.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 101 Left imaging system 102 Right imaging system 103 Left A / D conversion means 104 Right A / D conversion means 105 Color signal processing means 106 System controller 107 Photometry means 108 Focus detection means 109 First process memory 110 Second process memory 111 Work memory 112 Corresponding point extracting means 1101 Subject extracting means

 ──────────────────────────────────────────────────続 き Continued on the front page (72) Inventor Nao Kurahashi 3-30-2 Shimomaruko, Ota-ku, Tokyo Inside Canon Inc. (72) Inventor Takeo Sakimura 3-30-2 Shimomaruko, Ota-ku, Tokyo Canon Inside the company

Claims (18)

[Claims] 1. An image pickup method for picking up an image by an image pickup apparatus having a plurality of image pickup systems, wherein a parallax for obtaining a parallax between images obtained by the respective image pickup systems by using values of a position, an orientation, and a focal length of each image pickup system. An imaging method, comprising: an extraction step; and a storage step of saving the parallax obtained by the parallax extraction step and an image captured by the imaging system all together. 2. The imaging method according to claim 1, wherein in the parallax extraction step, values of a position, an orientation, and a focal length of each imaging system are used to limit a search range of a corresponding point extraction process. 3. An image pickup method for picking up an image by an image pickup apparatus having a plurality of image pickup systems, wherein parallax for obtaining a parallax between images obtained by the respective image pickup systems by using values of a position, an orientation, and a focal length of each image pickup system. An extracting step, a judging step of judging whether or not to retake the image from the reliability of the parallax obtained by the parallax extracting step, and the parallax obtained by the parallax extracting step and the image captured by the imaging system in a lump. And a storing step of storing. 4. The imaging method according to claim 3, wherein, in the parallax extraction step, values of a position, an orientation, and a focal length of each imaging system are used to limit a search range of the corresponding point extraction processing. 5. An image pickup method for picking up an image by an image pickup apparatus having a plurality of image pickup systems, wherein a parallax extraction step for obtaining a parallax between images obtained by the respective image pickup systems, and a subject extraction from the parallax obtained by the parallax extraction step. An initial nucleus creating step for creating an initial nucleus for the subject, a subject area extracting step of growing a region of the initial nucleus created by the initial nucleus creating step to determine a subject area in the image, and an object nucleus extracting step. An image capturing method, comprising: a storing step of collectively storing the obtained subject region and an image captured by the image capturing system. 6. A representative parallax extracting step of obtaining a representative parallax to a subject using values of a position, an orientation, and a focal length of each imaging system, and a parallax obtained by the parallax extracting step and the representative parallax extracting step. 6. The imaging method according to claim 5, further comprising a comparing step of comparing the obtained representative parallax. 7. An image pickup apparatus having a plurality of image pickup systems, wherein parallax extraction means for obtaining parallax between images obtained by each image pickup system using values of the position, orientation, and focal length of each image pickup system; An image pickup apparatus comprising: a storage unit that collectively stores a parallax obtained by a parallax extraction unit and an image captured by the imaging system. 8. The imaging apparatus according to claim 7, wherein the parallax extraction unit uses the values of the position, orientation, and focal length of each imaging system to limit a search range of the corresponding point extraction processing. 9. An image pickup apparatus having a plurality of image pickup systems, wherein parallax extracting means for obtaining a parallax between images obtained by each image pickup system by using values of a position, an orientation, and a focal length of each image pickup system; Determining means for determining whether or not to retake from the reliability of the parallax obtained by the parallax extraction means, and storage means for collectively storing the parallax obtained by the parallax extraction means and the image captured by the imaging system. An imaging device comprising: 10. The imaging apparatus according to claim 9, wherein the parallax extraction unit uses the values of the position, orientation, and focal length of each imaging system to limit a search range of the corresponding point extraction processing. 11. An image pickup apparatus having a plurality of image pickup systems, wherein a parallax extracting means for obtaining a parallax between images obtained by the respective image pickup systems, and an initial kernel for extracting a subject from the parallax obtained by the parallax extract means. Initial nucleus creating means for creating, the subject area extracting means for obtaining a subject area in the image by area growing the initial nucleus created by the initial nucleus creating means,
An image pickup apparatus comprising: a storage unit that collectively stores a subject region obtained by the subject region extraction unit and an image captured by an imaging system. 12. A representative parallax extracting means for obtaining a representative parallax to a subject using values of a position, an orientation, and a focal length of each imaging system, and a parallax obtained by the parallax extracting means and the representative parallax extracting means. 12. The imaging apparatus according to claim 11, further comprising a comparison unit configured to compare the obtained representative parallax. 13. A storage medium for storing a program for controlling an image pickup apparatus having a plurality of image pickup systems, wherein an image obtained by each image pickup system using values of a position, an orientation, and a focal length of each image pickup system. A storage medium storing a program having a parallax extraction module for obtaining a parallax between the two, and a storage module for collectively storing the parallax obtained by the parallax extraction module and an image captured by the imaging system. . 14. The storage medium according to claim 13, wherein in the parallax extraction module, values of a position, an orientation, and a focal length of each imaging system are used to limit a search range of a corresponding point extraction process. 15. A storage medium for storing a program for controlling an image pickup apparatus having a plurality of image pickup systems, wherein an image obtained by each image pickup system using values of the position, orientation, and focal length of each image pickup system. A parallax extraction module for obtaining a parallax between, a determination module for determining whether or not to retake from the reliability of the parallax obtained by the parallax extraction module, and a parallax obtained by the parallax extraction module and an image captured by an imaging system. And a storage module for collectively storing the images. 16. The storage medium according to claim 15, wherein in the parallax extraction module, values of a position, an orientation, and a focal length of each imaging system are used to limit a search range of a corresponding point extraction process. 17. A storage medium for storing a program for controlling an image pickup apparatus having a plurality of image pickup systems, comprising: a parallax extraction module for obtaining parallax between images obtained by each image pickup system; An initial nucleus creation module for creating an initial nucleus for subject extraction from the obtained parallax; a subject area extraction module for growing a region of the initial nucleus created by the initial nucleus creation module to obtain a subject area in an image; A storage medium comprising: a storage module that collectively stores a subject area obtained by a subject area extraction module and an image captured by an imaging system. 18. A representative parallax extraction module for obtaining a representative parallax to a subject using values of the position, orientation, and focal length of each imaging system, and a parallax obtained by the parallax extraction module and the representative parallax extraction module. The storage medium according to claim 17, further comprising: a comparison module configured to compare the obtained representative parallax.