JP5889022B2 - Imaging apparatus, image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an imaging apparatus that includes a plurality of imaging optical systems, calculates distance information from an image acquired by an imaging element, and generates an in-focus image in a wide range from the front to the back based on the distance information. .

  Conventionally, an imaging device that can capture a right-eye image and a left-eye image with two imaging optical systems to generate a stereo image, a multi-eye imaging device that acquires images from two or more various viewpoint positions, etc. An imaging apparatus using a plurality of imaging optical systems is known. In such an imaging apparatus, various methods have been proposed for capturing a subject in various depths by focusing on the subject at various depth distances by changing the focal length of each camera.

  For example, after splitting incident light of an external scene, a person, etc. as a subject with a half mirror, this is imaged using two or more different TV imaging cameras with different focusing, and each pixel of each camera output signal is compared However, there is a television image pickup device having means for calculating and selecting a signal with a focus position. As a result, it is possible to generate an image with less blur that is focused on both a distant view and a close view (see Patent Document 1 below).

  In addition, a plurality of imaging optical systems provided with identification information n (n = 1, 2, 3,...) That do not overlap each other are provided, and the focus position of each focus lens in each imaging optical system is expressed as H / (2n− 1) By setting the lens position corresponding to the above-mentioned subject distance L and performing simultaneous imaging in which the light-receiving subject image of each imaging optical system is simultaneously converted into image data by the corresponding imaging element in accordance with the imaging instruction. There is an imaging apparatus that can obtain a focused image in a wider distance range in proportion to the number of imaging optical systems. Since images that are in focus at various distances can be acquired at the same time, a focused image can be taken regardless of the distance of the subject to be taken (see Patent Document 2 below).

  In addition, a first image sensor that provides a first sensor output and a first imaging unit having a first lens that expresses an image of a scene in the field of view on the first image sensor, a second image sensor that provides a second sensor output, and a field of view on the first image sensor A second imaging unit having a second lens that expresses a second image of the internal scene, and one of the outputs from the two imaging units is selected as an imaging output signal, while the subject focal position is detected on the other side There is a distance map generation type multi-lens camera that generates a distance map from binocular information. This makes it possible to focus at high speed because the imaging unit that is not used for video shooting is used when shooting still images and videos, and enables distance information to be calculated with two eyes (see below). (See Patent Document 3)

JP-A 63-224557 JP 2010-204483 A Special table 2010-524279

As described above, there have been many requests for clear shooting of subjects at various distances.
However, the technique described in Patent Document 1 uses a special configuration using a half mirror, and there are problems such as a reduction in the amount of light and a complicated configuration that makes it impossible to use a normal module.

  Further, in the technique described in Patent Document 2, it is possible to acquire an image focused on a subject at various distances, but it is necessary to specify a corresponding image from images captured by each imaging optical system, You can't get an image that focuses on all subjects.

  In the technique described in Patent Document 3, a plurality of imaging optical systems are provided, and the subject is quickly focused by separating the imaging optical system for photographing and the imaging optical system for focus adjustment. In addition to the problem that it takes some time to focus, it is difficult to obtain an image in which all subjects are in focus, as in Patent Document 2.

  The present invention has been made in order to solve the above-described problems, and provides an imaging device that acquires a clear image with no blur and with a simpler configuration and focused on a subject at all distances. The purpose is to do.

  According to an aspect of the present invention, at least two imaging units, a first imaging unit and a second imaging unit, are provided, and the respective focal positions of the first imaging unit and the second imaging unit are controlled. A focus control unit that performs image processing on image data including a first image and a second image captured by the first imaging unit and the second imaging unit, respectively. A distance information calculation unit that calculates distance information from the first image and the second image to the subject, and an image composition unit that combines the first image and the second image. The focus control unit controls the focal lengths of the first imaging unit and the second imaging unit to different in-focus positions, and the image synthesis unit selects an image based on the distance information. An imaging apparatus characterized by combining is provided.

  The image correction unit further includes an image correction unit that corrects the first image and the second image, the image correction unit corrects the subject to have the same size, and the first image and the second image are corrected. It is characterized by replacing.

  By leveling the epipolar lines of the first image and the second image, the difference in the angle of view is corrected, the size of the subject imaged by each imaging unit becomes the same size, and the corresponding points between the two images Are located on the same line.

  An image determination unit configured to determine which of the corrected first image and the corrected second image to be combined based on a focus range and the calculated distance information; And an in-focus image generation unit that generates an in-focus image based on the determination result.

  First, a subject in the foreground in focus is extracted from the image focused on the foreground using parallax information. Since the in-focus range of the subject is known in advance according to the focal position of the imaging apparatus at that time, the parallax value corresponding to this in-focus range is held. When the in-focus range is defined based on the parallax, the image determination unit can hold the parallax value and determine an area where the parallax value is equal to or greater than the predetermined value based on this value.

  Preferably, the image determination unit detects a focused state between the first image and the second image with respect to a peripheral portion of the subject, and extracts a focused image.

  The image determination unit detects the in-focus state of a plurality of images to be combined when determining an image to be combined in the vicinity of a boundary where the images are combined, and if there is no difference between the in-focus states, It is preferable to preferentially combine in-focus images.

  Since the parallax expansion region exists, the camera image is enlarged when the contrast is determined, so the edge positions do not match. Therefore, the area where the determination process is performed near the boundary to be combined is enlarged according to the enlargement ratio. As a result, it is possible to obtain a natural composite image in the vicinity of the composite boundary while suppressing the amount of calculation.

  The distance information calculation unit may further include an image reduction unit that reduces the image before calculating the distance information, and a parallax expansion unit that enlarges the image after the distance information is calculated.

  A distance map such as a parallax map calculated by reduction is enlarged by a parallax magnifying unit so as to correspond to the parallax of each pixel in the original image.

  The image composition unit enlarges and extracts an area focused on a foreground in the first or second image based on the distance information and the reduction ratio, and the image enlargement An image determination unit that performs a process of determining which image is to be combined in the vicinity of a combination boundary between the image enlarged in the unit and the image focused on the distant view of the first or second image; Preferably, the image processing unit includes a focused image generation unit that generates a composite image of the enlarged image and the unenlarged image according to the determination information determined by the image determination unit.

  When combining captured images of a plurality of imaging optical systems, it is possible to enlarge and combine an image focused on a near view.

  It is preferable that the image determination unit determines color information in addition to the in-focus state of a plurality of images to be combined when determining an image to be combined.

  Since the edges do not match due to the use of the enlarged image, there is a possibility of erroneous determination if it is determined which image is to be used simply by contrast. Therefore, the image determination unit uses color information in a form that prioritizes contrast. The difference in color information between the captured images is taken, and if the color information is equal to or smaller than a preset threshold value, the image with the higher contrast is determined. On the other hand, if there is a difference greater than or equal to the threshold, an image obtained by photographing a foreground is determined as a composite image. The difference in color information is the difference between RGB pixel values, and may be the sum of absolute values. Alternatively, the luminance and color difference signals may be divided and judged by the color difference, and there are various other methods.

  In this way, when there is a possibility that it cannot be determined only by contrast, it is possible to reliably select an image obtained by photographing a close-up by using color information, and the image becomes unnatural at the boundary to be combined. Can be prevented.

  It is preferable that the focus control unit includes an operation setting unit that switches control of the in-focus positions of the first imaging unit and the second imaging unit to the same or different states.

  Accordingly, when the 3D shooting mode is selected in the operation setting unit, the focus control unit sets the in-focus position of the first imaging device and the second imaging device to the same setting, and the image synthesis The image capturing apparatus can perform various operations such as 3D shooting without image synthesis and generation of an omnifocal image when the omnifocal camera mode is set.

  In addition, the present invention provides an imaging apparatus that includes a first imaging unit and a second imaging unit, and a focus control unit that controls a focal position of the first imaging unit and the second imaging unit, respectively. An image processing apparatus that performs image processing on image data including a captured first image and a second image, and calculates distance information from two images of the first image and the second image A distance information calculation unit that performs the above operation, and an image composition unit that synthesizes the two images of the first image and the second image, and the in-focus state of the input image captured by the imaging device is different. In this case, the distance information calculation unit calculates distance information from the first image and the second image, and the image composition unit calculates the first image and the first image based on the distance information. An image processing apparatus characterized by combining two images.

  According to another aspect of the present invention, at least two imaging units, a first imaging unit and a second imaging unit, are provided, and the respective focal positions of the first imaging unit and the second imaging unit are determined. An image processing method for performing image processing on image data including a focus control unit to be controlled, and a first image and a second image respectively captured by the first imaging unit and the second imaging unit. A distance information calculating step of calculating distance information from the first image and the second image to the subject, and an image combining step of combining the first image and the second image, The focus control unit controls the focal lengths of the first imaging unit and the second imaging unit to different in-focus positions, and synthesizes an image based on the distance information in the image synthesis step. An image processing method is provided.

  An image correction step for correcting the first image and the second image is further included, and the image correction step corrects the subject to have the same size, and the first image and the second image are corrected. Is preferably replaced.

  And an image determination step for determining which of the corrected first image and the corrected second image to be combined is based on a focus range and the calculated distance information, The image determination step detects a focused state between the corrected first image and the corrected second image with respect to the periphery of the subject, and extracts a focused image. It is preferable to determine the image.

  The present invention may be a program for causing a computer to execute the image processing method described above, or a computer-readable recording medium for recording the program.

  According to the imaging apparatus of the present invention, it is possible to generate a clear image focused on a subject at all distances with a simple configuration.

It is a functional block diagram showing an example of 1 composition of an imaging device by a 1st embodiment of the present invention. It is a figure which shows the outline | summary of a left camera image and a right camera image. It is a figure regarding description of an epipolar line. It is a figure which shows the outline | summary of the left-right camera image after correction | amendment. It is a figure which shows the outline | summary of the stereo matching by SAD. It is a figure which shows the example of parallax information. It is a functional block diagram which shows the example of 1 structure of an image synthetic | combination part. It is a figure explaining the parallax between right-and-left camera images. It is a figure which shows the boundary part at the time of the synthesis | combination of a left-right camera image. It is a figure for demonstrating occlusion. It is a flowchart figure which shows the flow of a process in an all-focus camera mode. It is a flowchart figure which shows the flow of the synthetic image creation process of FIG. It is a flowchart figure which shows the flow of a synthetic | combination process with the left image focused on the distant view and the right image of FIG. It is a functional block diagram which shows the example of 1 structure of the imaging device by the 2nd Embodiment of this invention. It is a functional block diagram which shows the example of 1 structure of the parallax calculation part in this Embodiment. It is a functional block diagram which shows the example of 1 structure of the image synthetic | combination part in this Embodiment. It is a figure which shows the effect of the expansion in an image synthetic | combination part. It is a flowchart figure which shows the flow of a parallax calculation process. FIG. 19 is a flowchart showing a detailed example of the image composition processing in FIG. 18.

Embodiments of the present invention will be described below with reference to the drawings.
(First embodiment)
FIG. 1 is a functional block diagram showing a configuration example of the imaging apparatus 100 according to the first embodiment of the present invention. FIG. 11 is a flowchart showing the flow of processing in the omnifocal camera mode. FIG. 12 is a flowchart showing the flow of the composite image creation process of FIG. FIG. 13 is a flowchart showing the flow of the composition process of the right image of FIG. 12 and the left image focused on a distant view. The image processing is executed by, for example, an image data processing apparatus excluding the imaging parts by the left camera 101 and the right camera 102 in the imaging apparatus 100 of FIG.

  In FIG. 1, the imaging device 100 is, for example, a mobile phone, a smartphone, a digital camera, or the like provided with two cameras. A fixed-focus lens and a focus adjustment function (focusing function), the left camera 101 arranged on the left side, the same function as the left camera 101, the right camera 102 arranged on the right side, and left and right camera images The image correction unit 103 that corrects the image, the focus control unit 104 that controls the focus by adjusting the lenses of the cameras 101 and 102, and the distance from the imaging device 100 to the subject from the corrected image of the left and right camera images A parallax calculation unit 105 that calculates parallax, which is an example of a distance information calculation unit that calculates information, an image synthesis unit 106 that combines left and right camera images, and an operation setting unit 107 that sets an operation mode of the imaging apparatus 100. Have.

  First, in order to start processing (FIG. 11: step S1), the operation setting unit 107 sets an operation mode. Here, the operation modes set by the operation setting unit 107 include a 3D shooting mode and an omnifocal camera mode. The 3D shooting mode will be described later. Hereinafter, the omnifocal camera mode will be described.

  When the omnifocal camera mode is set by the operation setting unit 107, the focus control unit 104 controls the lenses of the left camera 101 and the right camera 102 to change the focus position (FIG. 11: step S2). The left camera 101 and the right camera 102 have lens adjustment motors therein. In this embodiment, the control motor is a VCM (voice coil motor), but other motors such as a stepping motor and a piezo motor may be used. By controlling this VCM, the lens can be moved back and forth, and the in-focus position of the camera can be changed. The in-focus position is the position of the camera and the subject that can capture the subject clearly.

  The focus control unit 104 sets the in-focus position of the left camera 101 to the distant view and the in-focus position of the right camera 102 to the foreground. At this time, it is assumed that the lens position of each camera set by the VCM is a predetermined fixed value, and the focus control unit 104 holds this information, and moves the lens to the corresponding position.

  The left camera 101 outputs an image focused on a subject at a distance far from the camera, and the right camera 102 outputs an image focused on a subject at a distance close to the camera. FIG. 2 is a diagram illustrating an example of the captured image. 2A is a left camera image, and FIG. 2B is a right camera image. This shows that a subject 108 indicated by a square and a subject 109 indicated by a circle are captured on each camera image. A focused subject is indicated by a solid line, an unfocused subject that is blurred. Shown with dotted lines. Further, it is assumed that a uniform wall exists behind the subject 109 in the background portion, is sufficiently far away, and parallax is zero.

  Since the right camera 102 is focused on the near distance, the image distance becomes longer. The image distance is a length obtained by adding the focal length of the lens and the extension amount of the lens. The focal length is the distance from the center of the lens to the position at which the subject at infinity is in focus, and the closer the subject is, the longer the image distance becomes. As a result, the angle of view is reduced and the subject is imaged larger. The dotted line in the horizontal direction excluding the subject in the figure is for showing this. Focusing on a specific point on the subject on the left camera image, it can be seen that the corresponding point is not on the same line on the right camera image, and the subject has been photographed large.

  If there is a difference in the angle of view due to the difference in focus position, the parallax calculation processing performed by the parallax calculation unit 105 becomes complicated, and the image correction unit 103 performs epipolar line parallelization processing called rectification. Here, the epipolar line will be described with reference to FIG.

  The epipolar line is a plane (epipolar plane) 113 formed by three points of the feature point 110 in the space and the lens centers 111 and 112 of the two cameras as shown in FIG. 3 when there are two cameras. The intersecting lines with the image planes 114 and 115 of the respective cameras are the corresponding lines, and are the dotted lines denoted by reference numerals 116 and 117 in FIG. In the case of two cameras having a parallel arrangement in which the optical axes of the cameras are arranged in parallel and having the same height and characteristics, the epipolar lines 116 and 117 become horizontal straight lines having the same height, and two images are displayed. The corresponding points may be searched on one line. However, the epipolar lines do not become horizontal and do not lie on the same straight line when there is congestion instead of parallel arrangement, when the focal lengths are different, or when there is vertical displacement or inclination. Therefore, the image correcting unit 103 performs processing for correcting the image so as to be on the same horizontal straight line.

  This correction processing is performed using camera parameters. The camera parameters are internal parameters such as focal lengths and lens distortion parameters of each camera, and external parameters representing the positional relationship between the cameras. This information is assumed to be held in advance by the image correction unit 103. These camera parameters can be calculated using a process called calibration. Each parameter can be calculated by capturing a calibration pattern in which a grid pattern or equally spaced dots are printed, and extracting feature points. The image correction unit 103 performs geometric transformation (coordinate transformation) of the image using the camera parameters, and leveles the epipolar lines of the left camera image and the right camera image (FIG. 11: step S3). Accordingly, the difference in the angle of view shown in FIG. 2 is also corrected. As shown in FIG. 4, the size of the subject captured by each camera becomes the same size, and the corresponding points between both images are located on the same line. Will be. Then, the corrected left and right camera images are sent to the parallax calculation unit 105 and the image composition unit 106.

  The parallax calculation unit 105 calculates the parallax from the corrected left and right camera images sent from the image correction unit 103 (FIG. 11: step S4). The parallax is distance information that can be converted into a distance from the camera based on information such as the base line length and focal length of the camera. Here, an example of a parallax calculation method will be described. As a method for calculating parallax, a technique called stereo matching is generally known, and there is a technique called SAD (Sum of Absolute Differences) or SSD (Sum of Squared Difference).

  With reference to FIG. 5, block matching by SAD, which is a kind of stereo matching, will be described. In the left camera image 118 and the right camera image 119 shown in FIG. 5A, the epipolar lines coincide with each other by the correction in the image correction unit 103, so the corresponding points on the left and right camera images are on the same line in the horizontal direction. The corresponding point search is performed. In the corresponding point search, a window 120 having a 3 × 3 size is set around a pixel for which parallax is to be calculated on the left camera image 118. The window 120 is set to a position of L pixels from the right end of the left camera image 118. In the right camera image 119, the position of the window is shifted by one pixel from the window 121 at a distance of L + N from the right end of the image on the same line as the window 120 to the window 122 at the L position from the right end of the image. However, by calculating the correlation value of the left camera image with respect to the window 120, the corresponding point can be specified.

FIGS. 5B and 5C show windows 120 and 121, respectively. Symbols assigned with numerical values such as X ₀ and X ′ ₀ such as X ₀ and X ′ ₀ indicate the pixel value of G in each pixel. Yes. An SAD value that is a correlation value for each window is calculated according to the mathematical formula shown in FIG. The window 120 is fixed and the correlation value is calculated while moving the window on the right camera image. The position where the SAD value is the smallest becomes the corresponding point for the window 120, and the parallax is calculated from this position information. The By performing such an operation on all the images of the left camera image 118, a parallax map in the left camera image 118 can be calculated. The parallax map is a map in which the parallax is calculated for each pixel of the image.

  Although the window size has been described here as 3 × 3, a larger size such as 5 × 5 or 9 × 9 may be used, and the center of gravity of the window deviates from the center, but an even number such as 4 × 4. The window may be different in size in the vertical and horizontal directions, and is not limited to the size shown here.

N, which is the search range, can be arbitrarily set according to conditions such as the camera baseline length and the subject distance. The search direction is also the search from the window 121 to the window 122 here, but the search may be performed from the window 122 to the window 121. Since the search range is based on the left camera image 118, the search range is the range from the window 121 to the window 122. However, the search range is not limited to this range. Further, in FIG. 5 (b), the value of such X ₀ was pixel values of G, R (red) and B (blue) such as any one gloss sum of other colors, or a luminance value of each pixel In addition, values of all three colors of RGB may be used, but the present invention is not limited to this.

  The parallax calculation unit 105 searches for corresponding points by stereo matching and calculates parallax as described above. Here, the window size is assumed to be 15 × 15. When parallax is calculated by SAD, there is a possibility that the edge of the subject does not match the parallax switching position, and the parallax switching position is located outside the subject. This is because the larger the window size, the wider the range of the subject than the subject tends to be. An example of the parallax map calculated by SAD is shown in FIG. In FIG. 6A, the parallax value corresponding to the subject in the foreground is larger (the color is lighter and darker), and the parallax value of the subject far from the camera in the distance is smaller (the color is darker). The parallax value of the subject 108 (region 123) is 50, the parallax value of the subject 109 (region 124) is 10, and the parallax value of the background (region excluding the regions 123 and 124) is 0. FIG. 6B shows a comparison between the shooting size of the subject 108 after image correction shown in FIG. 4 and the size of the region 123 in which the parallax value corresponding to the subject 108 on the parallax map is calculated. As shown in FIG. 6B, the region 123 is larger than the subject 108, and it can be seen that the region where the parallax value is calculated expands with respect to the subject. The parallax calculation unit 105 sends the calculated parallax information (parallax map and window size) to the image synthesis unit 106.

  The image synthesis unit 106 acquires the corrected left camera image and right camera image from the image correction unit 103, and acquires parallax information from the parallax calculation unit 105. The image composition unit 106 uses the parallax information to generate one image focused on the subject at all distances from the corrected left camera image and right camera image (FIG. 11: step S5).

  Details of the image composition process will be described below. FIG. 7 is a functional block diagram illustrating a detailed configuration example of the image composition unit 106, and the image composition unit 106 includes an image determination unit 125 and a focused image generation unit 126. The image determination unit 125 determines which of the camera images is to be combined based on parallax information, contrast described later, and the like, and the focused image generation unit 126 combines the images according to the determination information that is output from the image determination unit 125. I do.

  First, from the right camera image focused on the foreground, the near-field subject focused on using the parallax information is extracted (FIG. 12: step S11). Since the focus distance of the subject is known in advance according to the focal length of the right camera, the parallax value corresponding to this focus range is held (FIG. 12: step S12). For example, if the focusing range of the right camera image is parallax 30 or more, the image determination unit 125 retains this parallax value, and determines an area having a parallax value of 30 or more based on this value (FIG. 12: Step S13). That is, the parallax region indicated by the region 123 in FIG. 6 is selected. Since the parallax map based on the left camera image is calculated as the parallax map, the right camera image is extracted by extracting the right camera image of the coordinates moved to the left by the parallax value from the coordinates of the area indicated by the area 123. It is possible to extract a subject in the foreground that is in focus from FIG. 12 (FIG. 12: step S13). FIG. 8 is a diagram for explaining this, with the upper diagram showing the left camera image and the lower diagram showing the right camera image. The parallax d between the images of the subject 108 shown in FIG. In this way, the foreground area is extracted from the right camera image and synthesized with the left camera image (FIG. 12: step S14).

  However, since the parallax map is expanded more than the subject as described in FIG. 6B, the image synthesized by performing the determination based only on the parallax information as described above is in focus. The blurred right camera image is synthesized instead of the left camera image. As shown in FIG. 9, the area around the subject 108 and the portion between the subject 108 and the parallax 50 area 123 are blurred as it is, and an unnatural image with a sense of incongruity is generated. Therefore, the image determination unit 125 performs the following processing to eliminate this unnaturalness.

  First, the processing is started (FIG. 13: Step S21), and the peripheral portion of the subject to be combined is discriminated between the left camera image and the right camera image before simple replacement. . Contrast is detected as a method for determining the in-focus state (FIG. 13: step S22). As a contrast detection method, there is a method in which an absolute value of a difference between a pixel value of a target pixel and a parallax value of surrounding pixels is taken and compared with a total value of these values. For example, the absolute value of the difference is calculated between the pixel values of the left and right and up and down of the target pixel and the pixel value of the target pixel, and the total value is calculated, and the left camera image and the right camera image are respectively compared. It can be determined that the image has a higher contrast. The contrast comparison area only needs to be compared with the expanded area, so the user may be able to freely set the 5 pixels inside the area extracted by the parallax value, etc. Accordingly, the width of the expansion area can be almost specified, and half of the window size tends to expand. Therefore, it is only necessary to compare half the window size or an area for the window size with a margin. By limiting the comparison area in this way, it is possible to prevent an increase in the amount of calculation and to prevent erroneous comparison and comparison in an unnecessary area. Here, it is assumed that the window size obtained from the parallax information is about half, and comparison is performed at 8 pixels.

  Contrast is compared at the 8 pixels around the subject, and the image with the higher contrast, that is, the focused image is extracted (FIG. 13: step S23). By thus giving priority to the in-focus area, it is possible to suppress blur around the subject. If there is no difference between the contrast values, the right camera image may be synthesized (FIG. 13: step S24). In order to determine whether or not there is a difference, there is a method in which a threshold value is determined in advance and it is determined that there is no difference when the difference is equal to or less than the threshold value. Note that the influence of occlusion can be reduced by giving priority to the right camera image when there is no difference in contrast value. The occlusion is caused by a difference in the viewpoint position of each camera, and is an area captured by only one camera. As shown in FIG. 10, a hatched area 127 that is not shown in the upper left camera image but shown in the lower right camera image is the occlusion area. In this occlusion area, the left camera image is the background, and the right camera image is the front side of the subject, so both may have a high contrast value in the contrast calculation. By giving priority to the composition, erroneous composition in this area can be prevented and a natural image can be obtained.

  When the 3D shooting mode is selected in the operation setting unit 107 in FIG. 1, the focus control unit 104 sets the in-focus positions of the left camera 101 and the right camera 102 to the same setting, and the image composition unit 106 sets the image. Assume that the parallax information, the left camera image, and the right camera image are output without combining. As a result, 3D shooting and parallax information can be calculated.

  As described above, by changing the focusing range of the two cameras and synthesizing the two images by parallax, it is possible to obtain an image with a deep depth of field focused in a wide range. In addition, it is possible to obtain a natural synthesized image by synthesizing an image with high contrast at the time of synthesis and giving priority to the right camera image when there is no difference.

  In this example, the image correction unit 103 performs rectification for parallax calculation. However, when the optical axes of the cameras are parallel and the heights in the vertical direction match, the left and right camera images are Only the size is different. Since it is possible to calculate in advance how much the size differs depending on each control value by the focus control unit, the reduction or enlargement ratio for alignment according to the lens position is held, The position may be corrected simply by enlargement / reduction. Even if the vertical heights are different, if the optical axes of the cameras are parallel, the two images are scaled to match the size, and then the image clipping position is changed in the vertical direction to absorb the difference in the height direction. It is possible to do so.

  In this embodiment, an example of using SAD for parallax calculation has been described. However, parallax calculation is not limited to SAD, and parallax is calculated for each area called an improved technique, SSD, or graph cut. Various methods such as a calculation method can be used.

  In this example, the camera has two fixed-focus lenses. However, the camera is not limited to a fixed-focus lens, and may be a two-lens camera using a zoom lens.

  As described above, according to the present embodiment, it is possible to generate a clear image focused on subjects at all distances with a simple configuration.

(Second Embodiment)
Next, a second embodiment of the present invention will be described with reference to the drawings. FIG. 14 is a functional block diagram illustrating a configuration example of the imaging apparatus 200 according to the present embodiment. In FIG. 14, the same symbols are assigned to components having the same functions as those in the first embodiment. 18 and 19 are flowcharts showing the flow of processing in the present embodiment.

  The imaging apparatus 200 according to the present embodiment includes a twin-lens camera unit 201 and an image processing unit 202 configured with a personal computer (PC). This image processing unit corresponds to the image processing apparatus A according to the first embodiment. The binocular camera unit 201 includes a left camera 101 having a focus adjustment function, a right camera 102 having a similar function, and a focus control unit 104 that adjusts the lens of each camera and controls the focus. The The image processing unit 202 includes an image correction unit 103 that corrects the left and right camera images, a parallax calculation unit 203 that calculates parallax from the corrected image of the left and right camera images, an image synthesis unit 204 that combines the left and right camera images, and an imaging device And an operation setting unit 107 that sets 100 operation modes.

  In the present embodiment, the two-lens camera unit 201 and the image processing unit 202 are divided and configured. However, the basic processing flow is the same as in the case of the first embodiment. Is omitted. The operations of the parallax calculation unit 203 and the image synthesis unit 204 that perform operations different from those of the first embodiment will be described below.

  When the process is started (FIG. 18: step S31), the parallax calculation unit 203 acquires the corrected left camera image and the corrected right camera image from the image correction unit 103 (FIG. 18: step S32). The parallax is calculated by stereo matching of these two videos as in the first embodiment. At this time, the parallax is calculated by reducing the image (FIG. 18: step S33).

  FIG. 15 is a functional block diagram illustrating a configuration example of the parallax calculation unit 203. The image reduction unit 205 reduces the image and the general parallax calculation unit 206 performs parallax map calculation. For example, if the input image size is 1920 × 1080 pixels, the image reduction unit 205 reduces the input image to 480 × 270 pixels. The reduction ratio at this time may be fixed in advance, or may be configured to be set by the user.

  As described in the first embodiment, the left camera image and the right camera image have different in-focus states depending on the distance. Therefore, stereo matching is performed between the blurred area of one camera and the clear area of the other camera. At this time, the parallax may not be calculated correctly due to blurring. In order to reduce this influence, image reduction processing is performed. By performing the reduction process, information of several pixels is integrated into one pixel, so that an edge of a blurred image becomes clear and parallax can be easily calculated, and an amount of calculation can be reduced. . On the other hand, however, the parallax calculation accuracy decreases due to the reduction. When the parallax is calculated with pixel accuracy, the parallax value 1 calculated with the image size of 480 × 270 is equivalent to calculating with the accuracy of 4 parallax values on 1920 × 1080, and ± 2 pixels. There will be an error. For example, if the parallax value at the original image size of the subject is 50, the parallax value on the reduced image is 12.5, which is 1/4 of the parallax value. However, when calculated with pixel accuracy, the parallax is calculated as 12 or 13, and when this is converted to the parallax value of the original image size, it becomes 48 or 52. Therefore, it is necessary to take this error into account when compositing images.

  The parallax calculation unit 203 enlarges the parallax map calculated by reduction using the parallax enlargement unit 207 so as to correspond to the parallax of each pixel in the original image (FIG. 18: step S34), and calculates the parallax information and the parallax map. Then, the reduction ratio is sent to the image composition unit 204 (FIG. 18: step S35). In this case, as the parallax value interpolation method, an interpolation method based on surrounding values such as nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation may be used as in the case of image enlargement. Thereby, the parallax map is the same as the image size of the original image. Also, the parallax value itself is enlarged. If the original image size is 1920 × 1080 and the parallax calculation image size is 480 × 270, the calculated parallax 1 corresponds to 4 on the original image size, so the parallax value is multiplied by four. Thereby, it can convert into the parallax value in an original image.

  The image synthesis unit 204 acquires the corrected left camera image and the corrected right camera image from the image correction unit 103, and combines the images using the parallax information acquired from the parallax calculation unit 203 and information on the reduction ratio. (FIG. 18: Step S36).

  FIG. 16 is a diagram illustrating a configuration example of the image composition unit 204, which includes an image enlargement unit 208, an image determination unit 209, and a focused image generation unit 210. FIG. 19 shows a flow of the image composition process, and the process is started in step S36-1.

  The image enlarging unit 208 extracts a region focused on the foreground based on the parallax information (step S36-2). This operation is the same as the operation performed by the image determination unit of the first embodiment. If the corrected image is FIG. 4 and the parallax is FIG. 6, an area having a parallax of 30 or more is specified. This region is cut out from the right camera image and sent to the image determination unit 209. At this time, the image is enlarged based on the information on the reduction ratio (step S36-3). As described above, since the parallax accuracy is reduced by the reduction process, an accurate parallax cannot be calculated, and therefore, there is an effect of preventing the composition position from being shifted and the image from becoming unnatural. For example, when there is a decrease in parallax accuracy of ± 2 pixels as described above, the image to be combined is enlarged and combined to a size that covers the error of ± 2 pixels. FIG. 17A is a diagram showing this, where an area 211 indicates the subject 108 extracted from the right camera image, and a dotted line area 212 indicates the blurred subject 108 of the left camera image. As shown in the figure, the edges may not match due to an error in parallax, and the composite image may become unnatural. However, as shown in FIG. This can be prevented by appropriately determining and synthesizing by the image determination unit 209. Note that enlarging the foreground image is effective even when the parallax map is not calculated by reducing the parallax map, and if there is an error in the parallax around the subject edge by enlarging and combining, This has the effect of preventing the composite image from becoming unnatural.

  The image determination unit 209 acquires the left camera image and the enlarged right-side camera image (step S36-4). A position for synthesizing the right camera image of the near view is determined from the parallax information (step S36-5). Then, a process of determining which image is to be synthesized around the synthesis boundary is performed. As in the first embodiment, since the parallax expansion region exists, the contrast is determined in the same manner. However, since the right camera image is enlarged, the positions of the edges do not match. For this reason, an area for performing the determination process is enlarged near the boundary to be combined according to the enlargement ratio (step S36-6). As a result, it is possible to obtain a natural synthesized image near the boundary of synthesis while suppressing the amount of calculation (step S36-7). In addition, since the edges do not match due to the use of the enlarged image, there is a possibility of erroneous determination if it is determined which image is to be used simply by contrast. Therefore, the image determination unit 209 uses color information in a form that prioritizes contrast. The difference between the color information is taken between the left camera image and the right camera image, and if the color information does not fall below a preset threshold value, the image with the higher contrast is determined. On the other hand, if there is a difference greater than or equal to the threshold, the image of the right camera image is determined as a composite image. The difference in color information is the difference between RGB pixel values, and may be the sum of absolute values. Alternatively, the luminance and color difference signals may be divided and judged by the color difference, and there are various other methods. In this way, if only the contrast information is determined, there is a focused background in the left camera image and an enlarged subject in the right camera image around the outer edge of the subject edge of the left camera image. Although there is a possibility that it cannot be determined, by using the color information, the right camera image can be reliably selected, and it becomes possible to prevent the image from becoming unnatural at the boundary to be combined.

  The focused image generation unit 210 combines the right camera image and the left camera image enlarged for each pixel in accordance with the determination information determined by the image determination unit 209, and the subject at all distances is in focus. It is possible to generate a simple image.

  Here, the image reduction process is performed to reduce the influence of blur. However, when the parallax is calculated using SAD, the window size may be increased instead. In this way, the amount of calculation increases, but the corresponding points are determined from information in a wide area, so that there is an effect that the corresponding points are easily detected.

  In each embodiment of the present invention, an example constituted by a two-lens camera has been shown. However, a system using a larger number of cameras such as a three-lens camera may be used. For example, when using images with different focus ranges with three eyes, one reference camera is determined, and parallax information is calculated for the other two cameras with respect to the reference camera. Then, based on the parallax information, it can be realized by discriminating which video of each camera is used in the same manner as in this embodiment. As the number increases, a composite image with a clearer subject can be generated.

  Moreover, although the method of calculating parallax information from a twin-lens camera is shown, as a camera that acquires distance information, a special camera that acquires distance information may be used to convert the distance information into parallax information. There is an infrared camera, etc. TOF (Time of flight) that obtains the distance to the subject from the time and speed of light until the light emitted from the infrared light is reflected by the object and reaches the sensor There is a law distance sensor. If the distance is known, the base line length and focal length of the camera can be converted into parallax information.

  In this example, the parallax value is reduced and calculated, and then the parallax value is multiplied by the reduction factor so that the parallax value of the original image size is obtained. You may do it. If the focusing range of the right camera is 30 or more in the original image size, the parallax value on the reduced image becomes 7.5 when the 1/4 is reduced in the case of the second embodiment, that is, the parallax value calculated on the reduced image is 8 or more. That is, the area having the parallax value is synthesized.

  In this example, the focus range of the right camera is 30 or more parallax, the parallax value of a near-distance subject is 50, and the entire short-distance subject is within the in-focus range. The subject may exist in the form. For example, when a subject such as a notebook on a plane is placed obliquely, the parallax value on the front side may be 35 and the parallax value on the far side may be 25. In such a case, if a part with a parallax value of 30 or more is extracted and synthesized, especially when a parallax value is calculated with a reduced image, there is a high possibility that a parallax error exists. There is a possibility of being. If there is a region where the parallax value continuously changes, this portion may be excluded from the synthesis processing target. By doing this, although some blurring remains, there is an effect of suppressing unnaturalness as a whole.

  In addition, when the image pickup apparatus corrects and outputs the image so that the size remains the same even if the in-focus position is changed, correction by image processing becomes unnecessary.

  Further, the present invention is not limited to the above-described embodiments, and various modifications are possible within the scope shown in the claims, and the invention can be obtained by appropriately combining the technical parts disclosed in different embodiments. Embodiments are also included in the technical scope of the present invention.

  In addition, a program for realizing the functions described in the present embodiment is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system and executed to execute processing of each unit. May be performed. The “computer system” here includes an OS and hardware such as peripheral devices.

  Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used.

  The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory in a computer system serving as a server or a client in that case, and a program that holds a program for a certain period of time are also included. The program may be a program for realizing a part of the above-described functions, or may be a program that can realize the above-described functions in combination with a program already recorded in a computer system.

  The present invention relates to an imaging device capable of capturing a right-eye image and a left-eye image by two imaging optical systems to generate a stereo image, and a multi-eye imaging device that acquires images from two or more various viewpoint positions. The present invention can be used for an imaging apparatus using a plurality of imaging optical systems and the image processing apparatus.

DESCRIPTION OF SYMBOLS 100 ... Imaging device 101 ... Left camera 102 ... Right camera 103 ... Image correction part 104 ... Focus control part 105 ... Parallax calculation part (distance information calculation part) 106 ... Image composition part 107 ... Operation setting part , 125 ... image determination unit, 126 ... focused image generation unit, 200 ... imaging device, 201 ... camera, 202 ... PC (image processing unit), 203 ... parallax calculation unit, 204 ... image composition unit, 205 ... image reduction unit , 206 ... parallax calculation unit, 207 ... parallax enlargement unit, 208 ... image enlargement unit, 209 ... image determination unit, 210 ... focused image generation unit.

Claims (7)

A focus control unit that has at least two imaging units, a first imaging unit and a second imaging unit, and controls respective focal positions of the first imaging unit and the second imaging unit; and the first An image processing apparatus that performs image processing on image data including a first image and a second image respectively captured by the image capturing section and the second image capturing section,
A distance information calculation unit for calculating distance information from the first image and the second image to the subject;
An image composition unit for compositing the first image and the second image;
An image determination unit that determines which of the first image and the second image is to be combined based on a focus range, the distance information, and a focus state;
A focused image generating unit that combines the first image and the second image to generate a focused image ;
The focus control unit controls the focal lengths of the first imaging unit and the second imaging unit to different in-focus positions,
The in- focus image generation unit generates an in-focus image based on a determination result by the image determination unit . The image determination unit
Respect periphery of the object, said detecting the focus state between the first image and the second image, according to claim 1, characterized in that to extract an image of a person who is in focus Imaging device. The image determination unit detects the in-focus state of a plurality of images to be combined when determining an image to be combined in the vicinity of a boundary where the images are combined, and if there is no difference between the in-focus states, the imaging apparatus according to claim 1 or 2, characterized in that to extract with priority the image is focused. The distance information calculation unit
It further includes an image reduction unit that reduces the image before calculating the distance information,
The image composition unit
An image enlarging unit for enlarging the first image or the second image based on the distance information and a reduction rate when the image is reduced in the image reducing unit ;
The focusing image generation unit, any areas that are focused on a near view of the first or second image from claim 1, characterized in that the synthesis is enlarged by the image enlargement unit to 3 The imaging apparatus of Claim 1 . First images captured by imaging devices each having a first imaging unit, a second imaging unit, and a focus control unit that controls a focal position of the first imaging unit and the second imaging unit. An image processing apparatus that performs image processing on image data including an image and a second image,
A distance information calculation unit that calculates distance information from two images of the first image and the second image;
An image composition unit for compositing two images of the first image and the second image;
An image determination unit that determines which of the first image and the second image is to be combined based on a focus range, the distance information, and a focus state;
A focused image generating unit that combines the first image and the second image to generate a focused image ;
When the in-focus state of the input image captured by the imaging device is different,
The distance information calculation unit calculates distance information from the first image and the second image,
The focused image generation unit synthesizes the first image and the second image based on a determination result by the image determination unit . A first imaging unit and the second imaging unit, a first captured respectively by an imaging device having a focus control unit for controlling the respective focal positions of the second imaging unit and the first image pickup unit An image processing method for image processing image data including a second image and a second image,
A distance information calculating step of calculating distance information from the first image and the second image to the subject, an image combining step of combining the first image and the second image;
An image determining step for determining which of the first image and the second image is to be combined based on a focus range, the distance information, and a focus state;
A focused image generation step of generating a focused image by combining the first image and the second image ,
Image processing characterized in that the focused image generation step generates a focused image based on the determination result of the image determination step when the in-focus state of the input image captured by the imaging device is different. Method. A program for causing a computer to execute the image processing method according to claim 6 .

Images