JP5900017B2 - Depth estimation apparatus, reconstructed image generation apparatus, depth estimation method, reconstructed image generation method, and program - Google Patents

Description

  The present invention relates to a depth estimation device, a reconstructed image generation device, a depth estimation method, a reconstructed image generation method, and a program.

  A technique for photographing a subject and estimating the depth of the subject is known. For example, in the technique described in Patent Document 1, a plurality of images are captured using a camera equipped with a sensor that measures the posture and the lens state, and the depth is obtained using parameters acquired by the sensor from the plurality of images. presume. At this time, it is necessary to measure the horizontal movement amount, pan angle, tilt angle, roll angle, zoom ratio, and focus amount of the camera by the sensor.

  In addition, there is known a technique for acquiring information related to the position, direction, and amount of light entering a lens of a camera from a subject by acquiring images of the subject taken from different viewpoints. In relation to such a technique, Patent Document 2 acquires a plurality of images (light field images) obtained by photographing a subject, and images of the subject in which focal length, depth of field, and the like are changed from the plurality of images. Is disclosed.

JP 2001-195580 A Special table 2008-515110 gazette

In order to estimate the depth of a subject using the technique described in Patent Document 1, a special camera including a physical sensor that measures the movement amount, pan angle, tilt angle, and roll angle of the camera is required.
On the other hand, no special physical sensor is required to acquire the light field image. Therefore, if the depth can be estimated from the light field image, the depth of the subject can be estimated without using a special physical sensor. If the depth of the subject can be estimated from the light field image, the estimation result can be used for noise removal of the reconstructed image. Patent Document 2 discloses a method for generating an image indicating the depth of the reconstructed image from the light field image. Is not disclosed.

  The present invention has been made in view of the above circumstances, and provides a depth estimation device, a reconstructed image generation device, a depth estimation method, a reconstructed image generation method, and a program capable of estimating the depth of a subject from a plurality of images obtained by photographing the subject. The purpose is to do. The present invention also provides a reconstructed image generating apparatus, a reconstructed image generating method, and a program capable of estimating the depth of an image reconstructed from a plurality of images obtained by photographing a subject and generating a reconstructed image with high image quality using the estimation result. For other purposes.

In order to achieve the above object, an aspect of the depth estimation apparatus according to the present invention is as follows.
A depth estimation device that estimates the depth of a subject from a light field image composed of a plurality of sub-images,
On the sub-image of the light field image, a pixel region defining means for defining a pixel area corresponding to the object,
And position on the target sub-image comprising a pixel area defined by the pixel area defining means, a position of the corresponding region corresponding to the picture element region on the peripheral sub-images arranged in the periphery of the target sub-images, the Depth estimation means for estimating the depth of the subject based on a positional shift;
Reconstructed image defining means for defining a reconstructed image obtained by reconstructing the image of the subject from the light field image and a reconstructed pixel that is a pixel of the reconstructed image;
Depth coefficient acquisition for obtaining a depth coefficient of a sub-pixel included in the pixel area based on a positional shift between the position of the pixel area on the target sub-image and the position of the corresponding area on the peripheral sub-image Means,
Bei to give a,
The depth estimation means estimates the depth of the subject by determining the depth coefficient of the reconstructed pixel of the reconstructed image based on the depth coefficient of each subpixel obtained by the depth coefficient acquisition means.
It is characterized by that.

In order to achieve the above object, one aspect of the reconstructed image generating apparatus according to the onset Ming,
A reconstructed image generating device that generates a reconstructed image focused on a predetermined reconstruction distance from a light field image composed of a plurality of sub-images ,
A pixel area defining means for defining a pixel area corresponding to a subject on the sub-image of the light field image; a position on the target sub-image including the pixel area defined by the pixel area defining means; A depth estimation device comprising: a depth estimation unit configured to estimate the depth of the subject based on a position shift of a corresponding area corresponding to a pixel area on a peripheral sub-image disposed around the image; and
Based on the difference in distance between the depth and the reconstruction distance of the object estimated by the depth estimation apparatus, a reconstruction image correcting means for correcting the pixel value of the reconstructed pixels constituting the reconstructed image,
Equipped with a,
It is characterized by that.
In order to achieve the above object, an aspect of the depth estimation method according to the present invention is as follows.
A depth estimation method for estimating the depth of a subject from a light field image composed of a plurality of sub-images,
Defining a pixel region corresponding to the subject on a sub-image of the light field image;
Based on the positional shift between the position on the target sub-image including the defined pixel area and the position of the corresponding area corresponding to the pixel area on the peripheral sub-image arranged around the target sub-image, Estimating the depth of the subject;
Defining a reconstructed image obtained by reconstructing the image of the subject from the light field image and a reconstructed pixel that is a pixel of the reconstructed image;
Obtaining a depth coefficient of a sub-pixel included in the pixel area based on a positional shift between the position of the pixel area on the target sub-image and the position of the corresponding area on the peripheral sub-image;
Including
In the step of estimating the depth, the depth of the subject is determined by determining the depth coefficient of the reconstructed pixel of the reconstructed image based on the depth coefficient of each of the sub-pixels obtained in the step of obtaining the depth coefficient. presume,
It is characterized by that.
In order to achieve the above object, an aspect of the reconstructed image generation method according to the present invention is:
A reconstructed image generation method for generating a reconstructed image focused on a predetermined reconstruction distance from a light field image composed of a plurality of sub-images,
Defining a pixel area corresponding to a subject whose depth is to be estimated on the sub-image of the light field image;
Based on the positional shift between the position on the target sub-image including the defined pixel area and the position of the corresponding area corresponding to the pixel area on the peripheral sub-image arranged around the target sub-image, Obtaining a depth coefficient of the subject for each pixel constituting a pixel area;
Extracting corresponding pixels on the light field image corresponding to the reconstructed pixels constituting the reconstructed image;
Setting a depth coefficient of the reconstructed pixel corresponding to the corresponding pixel based on the obtained depth coefficient;
Correcting the pixel value of the reconstructed pixel based on the difference between the distance to the subject indicated by the depth coefficient of the reconstructed pixel and the reconstructed distance;
Generating a reconstructed image using the corrected pixel value of the reconstructed pixel;
including,
It is characterized by that.
In order to achieve the above object, an aspect of the program according to the present invention is as follows.
To estimate the depth of a subject from a light field image composed of multiple sub-images,
A function for defining a pixel region corresponding to the subject on the sub-image of the light field image;
Based on the positional shift between the position on the target sub-image including the defined pixel area and the position of the corresponding area corresponding to the pixel area on the peripheral sub-image arranged around the target sub-image, A function for estimating the depth of the subject;
A function for defining a reconstructed image obtained by reconstructing the image of the subject from the light field image and a reconstructed pixel that is a pixel of the reconstructed image;
A function for obtaining a depth coefficient of a sub-pixel included in the pixel area based on a positional shift between a position of the pixel area on the target sub-image and a position of the corresponding area on the peripheral sub-image;
Realized,
In the function of estimating the depth, the depth of the subject is determined by determining the depth coefficient of the reconstructed pixel of the reconstructed image based on the depth coefficient of each subpixel obtained by the function of obtaining the depth coefficient. presume,
It is characterized by that.
In order to achieve the above object, an aspect of the program according to the present invention is as follows.
In order to generate a reconstructed image focused on a predetermined reconstruction distance from a light field image composed of a plurality of sub-images,
A function for defining a pixel region corresponding to a subject whose depth is to be estimated on a sub-image of the light field image;
Based on the positional shift between the position on the target sub-image including the defined pixel area and the position of the corresponding area corresponding to the pixel area on the peripheral sub-image arranged around the target sub-image, A function for obtaining a depth coefficient of the subject for each pixel constituting a pixel region;
A function of extracting corresponding pixels on the light field image corresponding to the reconstructed pixels constituting the reconstructed image;
A function of setting a depth coefficient of the reconstructed pixel corresponding to the corresponding pixel based on the obtained depth coefficient;
A function of correcting the pixel value of the reconstructed pixel based on a difference in distance between the distance to the subject indicated by the depth coefficient of the reconstructed pixel and the reconstructed distance;
A function of generating a reconstructed image using the corrected pixel value of the reconstructed pixel;
To realize,
It is characterized by that.

  According to the present invention, it is possible to estimate the depth of an image reconstructed from a plurality of images obtained by photographing a subject. In addition, a reconstructed image with high image quality can be generated using the estimation result.

It is a figure which shows the structure of the digital camera which concerns on Embodiment 1 of this invention. 1 is a diagram illustrating a configuration of an optical system of a digital camera according to Embodiment 1. FIG. FIG. 2 is a diagram illustrating an example of a light field image according to the first embodiment, where (a) illustrates a conceptual diagram of the light field image, and (b) illustrates an example of the light field image and the depth image. It is a figure which shows (a) physical structure and (b) functional structure of the image generation apparatus which concerns on Embodiment 1. FIG. 3 is a flowchart illustrating image output processing according to the first embodiment. 5 is a flowchart illustrating pixel area definition processing according to the first embodiment. 5 is a flowchart illustrating region integration processing according to the first embodiment. 4 is a flowchart illustrating depth coefficient calculation processing according to the first embodiment. It is a figure for demonstrating the depth coefficient calculation process which concerns on Embodiment 1, (a) is an example of an attention pixel area, (b) and (c) determines the area depth coefficient, (d) and ( e) is a diagram for explaining each of the depth coefficient voting processes. 5 is a flowchart showing region depth coefficient calculation processing according to the first embodiment. It is a figure which shows the example of the depth table which concerns on Embodiment 1. FIG. 5 is a flowchart illustrating depth coefficient determination processing according to the first embodiment. It is a figure for demonstrating the depth coefficient determination process which concerns on Embodiment 1, (a) is LFI, (b) is an example of a pixel histogram, (c) is another example of a pixel histogram, respectively. Show. 5 is a flowchart illustrating depth filter processing according to the first embodiment. It is a figure for demonstrating the depth filter process which concerns on Embodiment 1, (a) is a process which weights a depth coefficient from a depth image, (b) is a histogram of a voting result, (c) is a filter process the depth-out images after, respectively. 3 is a flowchart illustrating image generation processing according to the first embodiment. FIG. 6 is a diagram for explaining ray tracing according to the first embodiment. FIG. 3 is a diagram for explaining image generation processing according to the first embodiment, where (a) is an example of a light field depth image, (b) is a corresponding pixel on a sub-image, and (c) is a generated depth image. Examples of It is a figure which shows the functional structure of the image generation apparatus which concerns on Embodiment 2 of this invention. FIG. 10 is a diagram for explaining image generation processing according to the second embodiment, where (a) is an example of a reconstructed image before filtering, (b) is filtering processing using a depth image, and (c) is generated. Examples of the reconstructed images are shown below. 10 is a flowchart illustrating image generation processing according to the second embodiment.

  Hereinafter, a digital camera and an image generation apparatus according to embodiments for carrying out the present invention will be described with reference to the drawings. In the drawings, the same or corresponding parts are denoted by the same reference numerals.

(Embodiment 1)
The image generation apparatus 30 (depth estimation apparatus) according to the first embodiment is mounted on the digital camera 1 illustrated in FIG. The digital camera 1 shoots a subject to shoot a light field image composed of a plurality of sub-images, estimates the depth of the subject from the light field image, and reconstructs an image (depth image, reconstructed image) indicating the estimation result. ) Is generated and displayed. The image generation device 30 is responsible for a function of generating a depth image from a light field image composed of a plurality of sub-images.

  As shown in FIG. 1, the digital camera 1 includes an imaging unit 10, an information processing unit 20 including an image generation device 30, a storage unit 40, and an interface unit (I / F unit) 50. With such a configuration, the digital camera 1 acquires light ray information from the outside and displays an image representing the light ray information.

  The imaging unit 10 includes an optical device 110 and an image sensor 120, and performs an imaging operation.

  As shown in FIG. 2, the optical device 110 includes a shutter 111, a main lens ML, and a sub lens array SLA (micro lens array). The optical device 110 captures light rays from the outside by the main lens ML, and the sub lens array. An optical image obtained by using the optical center of each sub lens SL constituting the SLA as a viewpoint is projected onto the image sensor 120.

  The image sensor 120 converts an optical image projected by the optical device 110 into an electrical signal and transmits the electrical signal to the information processing unit 20. For example, an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). And a transmission unit that transmits an electrical signal generated by the image sensor to the information processing unit 20.

The shutter 111 controls the incidence and shielding of external light on the image sensor 120.
The main lens ML is composed of one or a plurality of convex lenses, concave lenses, aspherical lenses, and the like, and a virtual imaging surface between the main lens ML and the sub lens array SLA using the light of the subject OB at the time of photographing as an optical image. The image is formed on the MIP. Note that the subject OB at the time of shooting is assumed to be a plurality of components that are separated from the main lens ML by different distances as shown in FIG. In the present embodiment, it is assumed that the main lens ML is a single focus lens having a single focal length f _ML .

  The sub lens array SLA includes M × N sub lenses (micro lenses) SL arranged in a lattice pattern on a plane. The sub-lens array SLA is an image sensor that constitutes the image sensor 120 as an optical image obtained by observing an optical image formed by the main lens ML on the imaging plane MIP with the optical center of each sub-lens SL as a viewpoint. An image is formed on the imaging surface IE. A space formed by a plane formed by the main lens ML and a plane formed by the imaging surface IE is referred to as a light field.

For the main lens ML, a maximum diameter LD and an effective diameter ED can be defined. The maximum diameter LD is the physical diameter of the main lens ML. On the other hand, the effective diameter ED is a diameter of a region that can be used for photographing in the main lens ML. Of the main lens ML, the outside of the effective diameter ED captures and reconstructs an image because light beams input to and output from the main lens are blocked by various filters attached to the main lens ML and the physical structure around the main lens ML. This is an area that is not effective for the purpose (non-effective area).
The effective diameter ED can be defined by measuring a part where light rays are blocked by the physical structure such as the filter at the time of shipment from the factory.

  In the example of FIG. 2, the light beam from the portion POB where the subject OB is present passes through the portion (effective portion) forming the effective diameter ED of the main lens ML and is projected onto the plurality of sub lenses SL. In this way, an area in which light emitted from the portion POB of the subject OB passes through the effective portion of the main lens ML and is projected onto the sub lens array SLA is referred to as a main lens blur MLB of the portion POB. Among these, the part where the chief ray reaches is called the main lens blur center MLBC.

  The distance from the optical center of the main lens ML to the imaging plane MIP of the main lens is b1, the distance from the imaging plane MIP to the plane formed by the sub lens array SLA is a2, and the imaging plane IE of the image sensor from the sub lens array SLA. Let c2 be the distance.

With the above-described configuration, the imaging unit 10 captures a light field image (LFI) including information on all light rays that pass through the light field.
FIG. 3A shows an example of the LFI obtained by shooting the block-shaped subject OB.
This LFI is composed of images (sub-images SI, S _{11 to} S _MN ) corresponding to each of M × N sub-lenses SL (microlenses) arranged in a lattice pattern. For example, the upper left subimage S ₁₁ corresponds to an image obtained by photographing the object OB from the upper left, the sub-image S _MN the lower right corresponds to an image obtained by photographing the object OB from the lower right.

Each sub image is arranged at a position on the LFI corresponding to the position of the sub lens on which the sub image is formed.
The i-th row sub-images (horizontal row sub-images) Si1 to SiN are stereo images in which the image formed by the main lens ML is formed by the sub-lens SL arranged side by side in the i-th row of the sub-lens array SLA. Corresponds to an image. Similarly, in the j-th row sub-images (vertical one-row sub-images) S1j to SMj, the images formed by the main lenses ML are arranged vertically in the j-th column of the sub-lens array SLA (microlens array). This corresponds to a stereo image taken with the sub lens SL.

As shown in FIG. 4B, the digital camera 1 of the present embodiment estimates the depth of each subject when the subject is reconstructed as a main image from the LFI, and displays an image (depth image DI) indicating the estimation result. Generate. Each pixel of the depth image DI is set to be closer to black at a place where the estimated depth of the subject of the pixel is deep (the subject is far from the main lens ML), and closer to white as it is closer.
In the present embodiment, each sub image is a gray scale image, and each pixel constituting the sub image has a pixel value (scalar value).

  The information processing unit 20 shown in FIG. 1 is physically composed of a CPU (Central Processing Unit), a RAM (Random Access Memory), an internal bus, and an I / O port. It functions as the generation device 30 and the imaging control unit 220.

  The image processing unit 210 acquires an electrical signal from the image sensor 120 and converts the acquired electrical signal into image data based on imaging setting information stored in the imaging setting storage unit 410 of the storage unit 40. The image processing unit 210 transmits the image data and shooting setting information obtained by adding predetermined information to the shooting setting information to the image generation device 30.

  The imaging setting information stored in the imaging setting storage unit 410 will be described later.

The image generation device 30 generates an output image (depth image) using the LFI transmitted from the image processing unit 210. Processing for generating a depth image by the image generation device 30 will be described later.
The image generation device 30 stores the generated output image in the image storage unit 430 of the storage unit 40.

  The imaging control unit 220 controls the imaging unit 10 based on the imaging setting information stored in the imaging setting storage unit 410 of the storage unit 40 and images the subject OB using the imaging unit 10.

The storage unit 40 includes a main storage device including a RAM and the like, and an external storage device including a nonvolatile memory such as a flash memory and a hard disk.
The main storage device loads control programs and information stored in the external storage device and is used as a work area of the information processing unit 20.
The external storage device stores in advance a control program and information for causing the information processing unit 20 to perform processing to be described later, and transmits these control program and information to the main storage device in accordance with instructions from the information processing unit 20. Further, in accordance with an instruction from the information processing unit 20, information based on the processing of the information processing unit 20 and information transmitted from the interface unit 50 are stored.

  Functionally, the storage unit 40 includes an imaging setting storage unit 410, a reconstruction setting storage unit 420, and an image storage unit 430.

The imaging setting storage unit 410 stores imaging setting information. The imaging setting information includes a distance between the main lens ML and the sub lens array SLA, a focal length f _ML of the main lens, information for specifying an exposure time, an F value, a shutter speed, and the like as imaging parameters that can change during imaging. In addition, the imaging setting storage unit 410 stores information on the physical configuration of the digital camera 1 such as the position of each sub lens SL on the sub lens array SLA, the distance c2 between the sub lens array SLA and the imaging surface IE, and the like. .
The imaging setting storage unit 410 transmits imaging parameters to the imaging control unit 220.
Further, the imaging setting storage unit 410 adds information related to the physical configuration to the imaging setting information of the imaging unit 10 and transmits the information to the image processing unit 210 as imaging setting information.

The reconstruction setting storage unit 420 stores setting parameters for generating an image by reconstructing from the LFI, which is stored at the time of initial shipment or input by the user using the operation unit 530.
The reconfiguration setting information includes information indicating the specific contents of the reconfiguration process and reconfiguration parameters. Here, it includes information for generating a depth image from the LFI, and information for specifying the distance between the focus point of the new image and the main lens ML.

  The image storage unit 430 stores the output image generated by the image generation device 30. The image storage unit 430 transmits the stored image to the I / O unit 510 and the display unit 520 of the interface unit 50.

  An interface unit (described as I / F unit in the figure) 50 is a configuration relating to an interface between the digital camera 1 and its user (user) or an external device, and includes an I / O unit 510, a display unit 520, And an operation unit 530.

  The I / O unit (Input / Output unit) 510 is physically composed of a USB (Universal Serial Bus) connector, a video output terminal, and an input / output control unit. The I / O unit 510 outputs the information stored in the storage unit 40 to an external computer, and transmits the information transmitted from the outside to the storage unit 40.

  The display unit 520 includes a liquid crystal display device, an organic EL (Electro Luminescence) display, and the like, and has a screen for inputting imaging parameters stored in the imaging setting storage unit 410 and a screen for operating the digital camera 1. indicate. Further, the display unit 520 displays the image stored in the image storage unit 430.

  The operation unit 530 detects, for example, various buttons provided in the digital camera 1 and a touch panel provided in the display unit 520, information on operations performed on the various buttons and the touch panel, and the storage unit 40, the information processing unit 20, and the like. The information on the user operation is transmitted to the storage unit 40 and the information processing unit 20.

  As shown in FIG. 3B, the digital camera 1 captures an LFI, generates a depth image DI reconstructed from the LFI, and outputs it.

Next, the configuration of the image generation device 30 will be described with reference to FIG.
As shown in FIG. 4A, the image generation device 30 includes an information processing unit 31a, a main storage unit 32, an external storage unit 33, an input / output unit 36, and an internal bus 37.

  The information processing unit 31a includes a CPU and a RAM.

  The main storage unit 32 has the same physical configuration as the main storage device of the storage unit 40. The external storage unit 33 has the same physical configuration as the external storage device of the storage unit 40 and stores a program 38. The input / output unit 36 has the same physical configuration as the I / O unit 510. The internal bus 37 connects the information processing unit 31a, the main storage unit 32, the external storage unit 33, and the input / output unit 36.

  The information processing unit 31a, the main storage unit 32, the external storage unit 33, the input / output unit 36, and the internal bus 37 are the internal circuit of the information processing unit 20 of the digital camera 1, the storage unit 40, and the interface unit. 50 may be a functional block realized by.

  The image generation device 30 copies the program 38 and data stored in the external storage unit 33 to the main storage unit 32, and the information processing unit 31 a executes the program 38 using the main storage unit 32. Processing for generating and outputting an image to be described later is executed.

  The image generation device 30 functions as an input unit 310, a depth estimation unit 320, and a depth image generation unit 330, as shown in FIG.

The input unit 310 receives an LFI acquisition unit 3110 that acquires an LFI from the image processing unit 210, a shooting setting acquisition unit 3130 that acquires shooting setting information when the LFI is acquired, and an output image (depth image) from the LFI. A reconstruction setting acquisition unit 3120 that acquires a reconstruction setting including parameters used for generation.
In addition to the imaging parameters (the distance between the main lens ML and the sub-lens array SLA, the focal length f _ML of the main lens, the information specifying the exposure time, the F value, and the shutter speed), the shooting setting information includes: The existing values include the effective diameter ED of the main lens, the position information of each sub lens SL constituting the sub lens array SLA, and the distance between the sub lens array SLA and the imaging surface IE of the image sensor 120. Further, a distance c1 from the main lens ML to the sub lens array SLA, a distance c2 from the sub lens array SLA to the surface of the image sensor (imaging surface IE), a diameter of each sub lens, a relative position of each image sensor to the sub lens, Information.

The reconstruction setting acquisition unit 3120 stores reconstruction setting information for reconstructing the image stored in the reconstruction setting storage unit 420. The reconstruction setting information includes information indicating specific contents of the reconstruction process and information necessary for the image generation process described below (information indicating a reconstruction parameter, a depth table, and a threshold value). The reconstruction parameter includes information for specifying the distance (reconstruction distance, distance a1) between the focus point of the new image and the main lens ML.
In this embodiment, a case where a depth image is generated on the reconstruction plane indicated by the reconstruction setting from the LFI will be described. At this time, the reconstruction setting information includes information for generating a depth image.

The depth estimation unit 320 and the depth image generation unit 330 include an output image (depth image) that matches the reconstruction setting acquired by the reconstruction setting acquisition unit 3120, a reconstruction setting, an LFI acquired by the LFI acquisition unit 3110, and It is generated using the shooting setting information acquired by the shooting setting acquisition unit 3130.
Among these, the depth estimation unit 320 estimates the depth of the subject corresponding to each pixel (subpixel) constituting the subimage, and generates a light field depth image (LFDI) representing the estimation result. The depth is the estimated distance between the main lens ML and the subject OB.

  In order to execute such processing, the depth estimation unit 320 performs the region definition unit 3210, the attention region selection unit 3220, the peripheral sub-image selection unit 3230, the SSD calculation unit 3240, the depth coefficient voting unit 3250, A pixel depth coefficient selection unit 3260 and a depth filter execution unit 3270 are included.

  The area defining unit 3210 defines an area (pixel area) composed of pixels estimated to correspond to the same subject on the basis of pixel values for each sub-image. One or a plurality of pixel regions are defined for one sub-image. For the process of defining the pixel region, any known method for dividing an image for each corresponding subject can be used. Here, the pixel area is defined using the process described later, but as another example, a difference image is calculated, and a part surrounded by a part where the difference value is larger than the threshold value is used as the pixel area. It is also possible to do.

  The attention area selection unit 3220 sequentially selects pixel areas on the sub-image defined by the area definition unit 3210 as attention pixel areas.

A peripheral sub-image is defined for the sub-image including the target pixel region selected by the target region selection unit 3220 (target sub-image). The peripheral sub-image is another sub-image included in a predetermined area (peripheral area) determined by reconstruction settings centered on the target sub-image.
The peripheral sub-image selection unit 3230 sequentially selects the peripheral sub-images as the target peripheral image.

  The SSD calculation unit 3240 defines a corresponding candidate region that is a candidate for the corresponding region corresponding to the target pixel region in the target peripheral image, and shifts the corresponding candidate region within a range corresponding to the assumed depth in the setting, A coefficient indicating the certainty of correspondence with the target pixel region in is calculated. Here, the sum of squares (SSD) of pixel value differences between the target pixel region and the corresponding candidate region is calculated as a coefficient. A specific processing method will be described later. The peripheral sub-image selection unit 3230 to the SSD calculation unit 3240 sequentially set the peripheral sub-images defined in the pixel-of-interest region as the target peripheral sub-image, and execute the above processing for all.

The depth coefficient voting unit 3250 obtains a coefficient indicating a depth that minimizes the sum of SSDs (SSSD) based on the SSDs calculated for all peripheral sub-images. Then, the obtained coefficient is set as a candidate of information (candidate coefficient) indicating the depth of the sub-pixel included in the target pixel area and the corresponding area. Further, the candidate coefficients are voted on the depth histograms (FIG. 9 (e)) of these pixels. The depth histogram and voting process will be described later.
At this time, the region corresponding to the candidate coefficient is a region (corresponding region) that is estimated to correspond to the target pixel region in the peripheral sub-image. Therefore, the depth coefficient voting unit 3250 determining the depth coefficient that minimizes the SSSD for the target pixel area is equivalent to defining the area on the peripheral sub-image indicated by the depth coefficient as the corresponding area of the target pixel area. In other words, the depth coefficient voting unit 3250 defines a corresponding area, and registers (votes) a shift in the position of the corresponding area in the depth histogram in order to estimate the depth of the pixel of the target pixel area and the corresponding pixel area. .

  Vote for the depth factor in the depth histogram. When the voting process is completed for all peripheral sub-images of the target pixel area, the target area selection unit 3220 selects the next target pixel area. Then, the attention area selection unit 3220 and the depth coefficient voting unit 3250 work together to vote the depth coefficient for all the pixel areas.

  The sub-pixel depth coefficient selection unit 3260 selects a depth coefficient indicating the depth of the subject corresponding to the sub-pixel of the sub-image constituting the LFI based on the depth histogram of the voting result. As a result of the selection, a depth coefficient indicating the estimated depth for each pixel of the LFI is obtained. An image having a pixel value determined by the depth coefficient is called LFDI. A specific selection method will be described later.

  The depth filter execution unit 3270 performs processing for applying an image filter (depth filter) described later to the LFI depth image generated by the sub-pixel depth coefficient selection unit 3260, and adjusts the edges of the image. This depth filter will be described later. Instead of the processing described later, any known image filter that adjusts the edge of the image and removes noise may be used.

  The depth image generation unit 330 reconstructs an output image (depth image) from the LFI depth coefficient. The depth image generation unit 330 includes a template definition unit 3310, a target pixel selection unit 3320, a corresponding pixel extraction unit 3330, and a pixel value determination unit 3340 for such processing.

  Based on the reconstruction setting acquired by the reconstruction setting acquisition unit 3120, the model definition unit 3310 arranges the prototype (model) of the depth image on the surface (reconstruction surface) at a distance a1 from the main lens ML. Here, the template is an image having all pixel values of 0 (or not set). Then, the defined prototype is transmitted to the target pixel selection unit 3320.

  The pixel-of-interest selection unit 3320 selects one of the pixels (depth pixel, reconstruction pixel) of the depth image on the reconstruction plane defined by the template definition unit 3310 as the pixel of interest. Then, information on the target pixel is transmitted to the corresponding pixel extraction unit 3330.

  The corresponding pixel extraction unit 3330 performs ray tracing from the target pixel selected by the target pixel selection unit 3320, and extracts a pixel (corresponding pixel) on the LFI to which light from the target pixel has arrived for each sub lens.

The pixel value determination unit 3340 determines the pixel value of the target pixel based on the depth coefficient of the corresponding pixel. Specific contents of this processing will be described later.
Pixel value determining unit 3340 from the target pixel selecting section 3320 determines the pixel value as a sequential pixel of interest including Murrell all reconstructed pixels in the depth image to generate a depth image.

  The output unit 340 outputs the depth image generated by the depth image generation unit 330 to the image storage unit 430.

  Here, processing (image output processing) for reconstructing and outputting an image (depth image) from the LFI executed by the image generation device 30 will be described with reference to FIGS. 5 to 17.

  When the user shoots an LFI using the digital camera 1 and the image processing unit 210 generates the LFI and transmits it to the image generating apparatus 30 together with the shooting setting information, the image generating apparatus 30 starts the image output process shown in FIG. .

In the image output process, first, the LFI acquisition unit 3110 acquires the LFI (step S101).
Then, the reconstruction setting acquisition unit 3120 acquires the reconstruction setting, and the imaging setting acquisition unit 3130 acquires the imaging setting (step S102).

  Then, the region definition unit 3210 executes processing (pixel region definition processing) for defining one or a plurality of pixel regions that can be assumed to correspond to the same subject in the sub-image (step S103).

The pixel area definition process executed by the area definition unit 3210 will be described with reference to FIG. Each process of the pixel area definition process is executed by the area definition unit 3210.
In the pixel area definition process, first, a sub image of interest is selected from the sub images (step S201). At this time, a label (pixel area label) that defines the first pixel area is defined.

  Then, from among the sub-pixels of the target sub-image, a target pixel that is a pixel that has not been determined yet to which pixel region (a pixel to which a pixel region label has not been attached, an unlabeled pixel) is selected (step) S202). Then, the current pixel area label is attached to the target pixel.

  Then, one unlabeled pixel is selected from the pixels located in the vicinity region centered on the target pixel, and set as the vicinity pixel. Then, pixel values of neighboring pixels are acquired (step S203). Although the neighborhood region can be freely set, here, a region formed by 8 pixels adjacent to the target pixel is defined as the neighborhood region.

  The pixel value of the target pixel is compared with the pixel values of the neighboring pixels acquired in step S203, and it is determined whether the difference between the pixel values is equal to or less than a threshold value (step S204).

  If the difference between the pixel values is equal to or smaller than the threshold value (step S204: YES), the neighboring pixels are incorporated into the current pixel area on the assumption that the neighboring pixels and the target pixel correspond to the same subject (step S205). ). Specifically, the current pixel area label is pasted on the neighboring pixels. Then, the neighboring pixel is set as the next pixel of interest (step S206), and the process is repeated from step S203.

On the other hand, if the difference is larger than the threshold value the pixel value (step S204: NO), since the target pixel and neighboring pixels may be estimated to correspond to a different object, the neighboring pixels do not transfer to the current pixel region. And it is discriminate | determined whether the unlabeled pixel which has not yet performed the discrimination | determination process of step S204 in the vicinity area | region of an attention image (step S207). If there is an unlabeled pixel that has not been subjected to the discrimination process (step S207; NO), the process is repeated from step S203 for the next unlabeled pixel in the vicinity region.

  On the other hand, if it is determined that the above processing has been executed for all unlabeled pixels in the vicinity area (step S207; YES), it can be determined that there is no unlabeled pixel that should belong to the current pixel area in the vicinity of the target pixel. Therefore, the average value of the pixel values belonging to the current pixel area is determined as the representative pixel value of the pixel area (step S208), and the process is terminated for the current pixel area.

Next, it is determined whether or not an unprocessed sub pixel (unlabeled pixel) remains in the target sub image (step S209).
If it is determined that unlabeled pixels remain (step S209; NO), the pixel area label is updated to define the next pixel area (step S210), and the process from step S202 is repeated for the next pixel area.

On the other hand, if it is determined that there are no unlabeled pixels remaining (step S209; YES), it is labeled which pixel region all the subpixels of the target subimage belong to.
Therefore, a process of integrating the current area (area integration process) is executed to integrate the pixel areas estimated to correspond to the same subject (step S211).

  The area integration process executed in step S211 will be described with reference to FIG. In the region integration process, one pixel region of interest is selected from the current pixel region (step S301).

Then, all adjacent pixel areas that are adjacent to the target pixel area are extracted (step S302). Note that the adjacent pixel region is a pixel region including a pixel adjacent to a pixel belonging to the target pixel region.
Then, the extracted representative pixel value of the adjacent pixel region is compared with the representative pixel value of the target pixel region, and it is determined whether there is an adjacent pixel region in which the difference between the representative pixel values is equal to or less than the threshold (step S303).

  If it is determined that there is an adjacent pixel area where the difference between the representative pixel values is equal to or less than the threshold (step S303; YES), the adjacent pixel area is integrated into the target pixel area (step S304). Specifically, the pixel area label of the pixel area of interest is overwritten with the pixel area label of the adjacent pixel area. Further, an average value of all the pixels in the new area is set as a new representative pixel value. Then, the process is repeated from step S302 with the new area as the target pixel area.

On the other hand, if there is no adjacent pixel area whose representative pixel value is less than or equal to the threshold (step S303; NO), it is next determined whether all the pixel areas have been processed (step S305). Specifically, it is determined whether or not there is a pixel area that has not been subjected to the determination process in step S303 until the present since the last integration of the areas. If there is (step S305; NO), the pixel area is determined as the target pixel. The process is repeated from step S301 as an area.
On the other hand, when there is no pixel area (step S305; YES), it can be determined that there is no more pixel area to be integrated, so the area integration process is terminated.

  Returning to FIG. 6, when the region integration processing is completed in step S211, it is determined whether pixel regions have been defined for all sub-images (step S212). If an unprocessed subimage remains (step S212; NO), the process is repeated from step S201 for the next target subimage. On the other hand, when all the sub-images have been processed (step S212; YES), the pixel area definition process ends.

  Returning to FIG. 5, when the region definition unit 3210 defines the pixel region, the attention region selection unit 3220 to the depth filter execution unit 3270 define the depth coefficient of each pixel of the LFI (all subpixels included in the LFI). Processing for generating a light field depth image (LFDI) (depth coefficient calculation processing) is executed (step S104).

The depth coefficient calculation process executed in step S104 will be described with reference to FIG. In step S104, the attention area selection unit 3220 starts the depth coefficient calculation process shown in FIG.
In the depth coefficient calculation process, first, the attention area selection unit 3220 selects an attention sub-image from the LFI sub-image (step S401). Further, one of the pixel areas on the target sub-image is selected as the target pixel area (step S402). An example of the target pixel region is shown as a region surrounded by a dotted line in FIG.

  Then, the SSD calculation unit 3240 starts a process of calculating the depth coefficient of the target pixel area (area depth coefficient calculation process, FIG. 10). Here, the peripheral sub-images are eight sub-images (SI1 to SI8) excluding the central target sub-image among the sub-images on the LFI shown in FIG. 9B.

  In step S403, the area depth coefficient calculation process shown in FIG. 10 is started. In the region depth coefficient calculation process, first, the peripheral sub-image selection unit 3230 selects a target peripheral sub-image from the peripheral sub-images (step S501).

  Next, one of the possible depth coefficients determined in the setting is selected as a target depth coefficient (step S502). The depth coefficient is associated with an image shift value (image parallax). In the present embodiment, as shown in FIG. 11, the depth coefficient is associated with a representative value of the image shift value (representative shift value). Hereinafter, unless otherwise specified, parallax means image parallax.

Then, the SSD calculation unit 3240 calculates the SSD for the corresponding area candidate at the position corresponding to the target depth coefficient on the target peripheral sub-image (step S503).
For example, when the target depth coefficient corresponds to the image shift 0, a region (corresponding candidate region) that is a candidate for the corresponding region is arranged in the same portion of the target peripheral subimage (here, S I 1). A sum of squares (SSD) of pixel value differences between corresponding pixels in the target pixel region and the corresponding candidate region is calculated as a coefficient indicating the certainty of correspondence.
On the other hand, when the target depth coefficient corresponds to a predetermined image deviation value, the direction parallel to the straight line passing through the centers of the target subimage and the target peripheral subimage and away from the target subimage (for S I 1, FIG. 9). Corresponding candidate areas are arranged at positions shifted by the image shift value (in the direction of arrow d1 in (b)), and the SSD of each position is calculated (step S503). Here, FIG. 9C shows the relationship (image shift-SSD graph) between the amount of shift (image shift) on the target peripheral sub-image and the SSD (image shift-SSD graph).

Here, the horizontal axis of the image shift-SSD graph represents the depth coefficient corresponding to the image shift. Further, it is assumed that the depth coefficient is defined in the depth table shown in FIG. 11 included in the reconstruction setting. The depth table is a table that records a depth coefficient, a representative displacement value representing image displacement, and a reconstruction distance that is a depth range in association with each other. In addition, the depth corresponding to the image shift includes the position of the sub lens SL corresponding to the target sub image, the position of the sub lens SL corresponding to the peripheral sub image, the focal length f _ML of the main lens ML, and the main lens ML. It is determined by the distance between the sub lens array SLA and the distance between the sub lens array SLA and the imaging surface IE.

  The larger the positional deviation (degree of deviation) between the target pixel area and the corresponding area, the more the image of the target pixel area and the corresponding area is located closer to the main lens ML (and the sub lens SL). Therefore, the corresponding reconstruction distance is smaller as the depth coefficient has a larger image shift.

  When the SSD calculation unit 3240 calculates the SSD for the corresponding candidate area shifted by the number of pixels of the representative shift value corresponding to the target depth coefficient of the target peripheral sub-image, the SSD is calculated for each SSD defined for the depth coefficient. (SSSD) is added (step S504).

  Then, it is determined whether or not the above processing has been executed for all depth coefficients determined in the setting (step S505). If there is an unprocessed depth coefficient (step S505; NO), the next unprocessed depth coefficient is determined. The process is repeated from step S502 for the target peripheral sub-image.

On the other hand, when all the depth coefficients have been processed (step S505; YES), the SSD calculation unit 3240 determines whether there are SSDs calculated for the target peripheral sub-image that are equal to or smaller than a threshold determined by the reconstruction setting. (Step S506).
If no subthreshold SSD (step S506; NO), since the attention surrounding the subimage noise is strong valid calculation Target not equal can be determined, eliminating the attention surrounding the subimage from the calculation (step S507). That is, the SSD calculated for the target peripheral sub-image is subtracted from the SSSD of each depth value.

  On the other hand, when there is an SSD below the threshold (step S506; YES), it can be determined that the target peripheral sub-image holds valid information, and thus step S507 is skipped.

  Then, it is determined whether or not the above processing has been performed on all the sub-images in the peripheral area (step S508). If there is an unprocessed peripheral sub-image (step S508; NO), the next peripheral sub-image is selected as the target peripheral sub-image. The process is repeated for the image from step S501.

  On the other hand, when all the peripheral sub-images have been processed (step S508; YES), the depth coefficient voting unit 3250 compares the SSSDs of the respective depth coefficients, and determines the depth coefficient having the smallest SSSD (MinD in FIG. 9C). The depth coefficient of the target pixel region is set (step S509).

  Returning to FIG. 8, when the depth coefficient of the target pixel area is calculated, the depth coefficient voting unit 3250 defines a corresponding area in the peripheral sub-image. Specifically, the corresponding region is defined at a position shifted from the portion corresponding to the target pixel region in the outward direction (d1, d2, etc. in FIG. 9B) by a positional shift (representative shift value) indicated by the depth coefficient. (FIG. 9 (d)). It should be noted that no target region is defined for the peripheral sub-image (SI7 in the example of FIG. 9D) that is not the addition target in step S504. Then, the calculated depth coefficient is voted on a histogram for determining the depth coefficient of the pixel belonging to the target pixel area and the pixel belonging to the corresponding area (step S404).

  Examples of voting are shown in FIGS. 9 (d) and 9 (e). As shown in FIG. 9E, a histogram (depth histogram) storing a depth coefficient and the number voted for the depth coefficient is defined for each sub-pixel. For example, when the depth coefficient of the target pixel region selected in step S402 is 1, the depth coefficient 1 is set as a candidate coefficient. Then, the candidate coefficient is voted, and the depth coefficient voting unit 3250 calculates the depth coefficient 1 of the histogram (FIG. 9E) of the pixels (for example, white dots in FIG. 9D) included in the corresponding area. Increase the number of votes by one. Similar processing is executed for all the target pixel areas and the pixels included in the corresponding area defined for the target area.

  When the voting ends in step S404, it is then determined whether or not the process of step S404 has been completed for all pixel regions of the target sub-image (step S405). If there is an unprocessed pixel area (step S405; NO), the process is repeated from step S402 for the next unprocessed pixel area.

  On the other hand, if the process of step S404 has been completed for all pixel regions of the target sub-image (step S405; YES), it is then determined whether the above-described process has been completed for all sub-images (step S406). If there is an unprocessed sub-image (step S406; NO), the process is repeated from step S401 for the next unprocessed sub-image.

  On the other hand, when the processing for the target sub-image has been completed (step S406; YES), the sub-pixel depth coefficient selection unit 3260 starts processing for determining the depth coefficient of each sub-pixel (depth coefficient determination processing). (Step S407).

The depth coefficient determination process executed in step S407 will be described with reference to FIG.
In the depth coefficient determination process, first, the sub-pixel depth coefficient selection unit 3260 selects a target pixel from all the sub-pixels included in the LFI (step S601). Then, the sub-pixel depth coefficient selection unit 3260 calculates the reliability coefficient tp of the target pixel (step S602). The reliability coefficient is a coefficient indicating how reliable the voting result of the histogram of the target pixel is for determining the depth coefficient of the target pixel. Here, the maximum number of votes of the depth coefficient on the histogram is nMAX, and the total histogram The number of votes is nALL, and nMAX is divided by nALL.

  Note that the reliability coefficient can be replaced with any numerical value indicating the degree of reliability of the depth coefficient that has acquired the maximum number of votes of the histogram of interest compared to the number of votes of other depth coefficients. For example, the inverse of the variance of the histogram may be used.

  Then, the sub-pixel depth coefficient selection unit 3260 determines whether or not the calculated reliability coefficient tp is equal to or greater than a threshold value determined in setting (step S603). If it is equal to or greater than the threshold (step S603; YES), the depth coefficient of the most frequently obtained vote is set as the depth coefficient of the pixel of interest (step S604).

  On the other hand, if it is smaller than the threshold (step S603; NO), the voting result does not have sufficient reliability to determine the depth coefficient, so the depth coefficient of the pixel of interest is set to UNKOWN (step S605).

  An example of this processing will be described with reference to FIG. Among the pixels on the LFI in FIG. 13A, in the histogram of the pixel i shown in FIG. 13B, the total number of votes is 11, and the depth coefficient 2 has obtained the maximum number of votes (6). Since the reliability coefficient tp (6/11) is equal to or greater than the threshold th (0.5), the depth coefficient dp of the pixel i is 2. On the other hand, in the histogram of the pixel ii shown in FIG. 13C, the total number of votes is 13, and the highest number of votes is the number of votes (4) with the depth coefficient 7. At this time, since the reliability coefficient tp (4/13) is smaller than the threshold th (0.5), the depth coefficient dp of the target pixel is UNKOWN indicating that it is unknown.

  Once the depth coefficient of the pixel of interest has been determined, it is next determined whether or not the above processing has been completed for all sub-pixels (step S606). If there is an unprocessed subpixel (step S606; NO), the process is repeated from step S601 for the next unprocessed subpixel. On the other hand, when there is no unprocessed subpixel (step S606; YES), the depth coefficient is defined for all the subpixels. FIG. 18A shows an example of a light field depth image (LFDI) in which the pixel values of all the pixels on the LFI are values indicating the depth coefficient. Since the depth boundary of the LFDI at this time is disturbed, a process of applying a filter (depth filter process) is executed in step S607 to improve the accuracy of the image.

  The depth filter process executed in step S607 will be described with reference to FIG. FIG. 15 shows a specific example of depth filter processing. When the processing reaches step S607, the depth filter execution unit 3270 starts the depth filter processing shown in FIG. In the depth filter process, first, the depth filter execution unit 3270 selects a pixel on the LFDI as a target pixel (step S701). In FIG. 15, the target pixel is indicated by a square surrounded by a thick line.

  Then, the depth filter execution unit 3270 defines a peripheral region around the target pixel. The peripheral area can be arbitrarily set, but here it is assumed to be an area composed of eight pixels adjacent to the target pixel (dotted line area in FIG. 15A). The depth filter execution unit 3270 selects a target peripheral sub-pixel from pixels in the peripheral area (peripheral sub-pixel) (step S702).

  Then, the depth filter execution unit 3270 determines whether or not the depth coefficient of the target peripheral sub-pixel is UNKOWN (step S703). If it is UNKOWN (step S703; YES), the pixel is not used as a reference, so the process skips to step S708. In the example of FIG. 15, the black pixel indicated by i in FIG. 15A is a pixel that is UNKOWN. This pixel is not subject to later processing.

  On the other hand, if it is not UNKOWN (step S703; NO), the depth filter execution unit 3270 next determines whether or not the difference between the pixel values of the target pixel and the target peripheral sub-pixel is equal to or larger than a threshold value set in the setting (step). S704).

If it is equal to or greater than the threshold (step S704; YES), the weight of this combination is set to 1 based on the determination that the correspondence between the target pixel and the target peripheral subpixel is weak (step S706). In the example of FIG. 15A, white pixels indicated by ii and hatched pixels indicated by iv are pixels that are equal to or greater than the threshold value. For each pixel, one vote corresponding to the weight is voted on the histogram.
If it is smaller than the threshold value (step S705; NO), the weight of this combination is set to 2 based on the judgment that the correspondence between the target pixel and the target peripheral subpixel is strong (step S706). In the example of FIG. 15A, the vertical and horizontal pixels indicated by iii are pixels smaller than the threshold value. For each pixel, two votes are voted on the histogram.
The weighting setting is not limited to the above, and can be replaced with an arbitrary weighting setting that exerts a stronger influence when the correspondence between the target pixel and the target peripheral subpixel is strong.

  Then, the depth coefficient is voted to the new histogram of the target pixel by the number of weights. (Step S707). An example of a histogram of voting results is shown in FIG.

  Next, it is determined whether or not the above processing has been completed for all the pixels in the peripheral region (step S708). If there are unprocessed peripheral subpixels (step S708; NO), the processing is repeated from step S702 for the next peripheral subpixel. .

  On the other hand, if all the peripheral sub-pixels have been processed (step S708; YES), the LFDI depth coefficient to be output next is updated based on the vote result. Specifically, the median value of the voted histogram is set as a new depth coefficient of the pixel of interest (step S709). In the histogram of FIG. 15B, 10 votes are given, and the median (the median value when the voted samples are sorted by depth) is 4. Therefore, the depth pixel value of the target pixel is set to 4 as shown in FIG. Note that the update result may be reflected after all the filtering processes are completed.

Then, it is determined whether or not the filtering process has been completed for all the sub-pixels (step S710). If there is an unprocessed sub-pixel (step S710; NO), the process is repeated from step S701 with the next pixel as the target pixel.
On the other hand, if all the sub-pixels have been processed (step S710; YES), the depth filter process is terminated.

Returning to FIG. 12, when the depth filter processing is finished in step S607, the depth coefficient determination processing is finished.
Returning to FIG. 8, when the depth coefficient determination process ends in step S407, the depth coefficient calculation process ends.

  Returning to FIG. 5, when the generation of the LFDI is completed, a process (image generation process) for reconstructing an output image (depth image) from the LFDI is executed (step S105). In the present embodiment, the image generation process is the image generation process 1 shown in FIG.

Image generation processing 1 executed in step S105 will be described with reference to FIG.
In step S105, the depth image generation unit 330 starts the image generation process shown in FIG. First, the depth image generation unit 330 acquires reconstruction settings (step S801). Then, the model definition unit 3310 specifies the distance between the main lens specified by the reconstruction setting and the reconstruction surface RF, and defines the model (prototype) of the reconstruction image on the reconstruction surface RF (step S802). .

  Then, the pixel-of-interest selection unit 3320 sets one of the reconstructed pixels that are pixels constituting the reconstructed image as the pixel of interest (step S803). At this time, the target pixel corresponds to the target site P on the reconstruction plane RF in FIG.

  Further, the corresponding pixel extraction unit 3330 extracts a pixel (corresponding pixel) on the sub-image corresponding to the target pixel by ray tracing (step S804).

Ray tracing will be described with reference to FIG.
The light beam from the site of interest P passes through the main point of the main lens and reaches the arrival position of the microlens array (MLBC on the sub lens in FIG. 17). The position on the sub lens of the MLBC can be obtained based on the shooting setting. A range (main lens blur MLB, mesh area in FIG. 17) in which light from the target site P reaches centered on the MLBC is obtained from the lens characteristics. The diameter of the main lens blur MLB the distance between the main lens ML and the reconstruction plane RF a1, (calculated from focal length _{f ML} of a1 and the main lens) distance b1 between the main lens and the image plane MIP, imaging plane MIP And the distance a2 between the sub lens array SLA and the effective diameter ED of the main lens.

  Corresponding pixel extraction unit 3330 specifies a sub-lens SL in which a part or all of sub-lenses SL included in sub-lens array SLA are included in main lens blur MLB. Then, the corresponding pixel extraction unit 3330 sequentially selects the identified sub lenses SL as the target lens.

  Next, the corresponding pixel extraction unit 3330 extracts a pixel (corresponding pixel) on the sub image at a position where the light beam from the target pixel is imaged by the selected sub lens SL.

Specifically, the corresponding pixel (corresponding to the reaching point PE) is calculated according to the following procedure.
First, the distance b1 to the focal plane of the main lens corresponding to the reconstruction plane RF uses the focal length f _ML of the known numerical a1 and the main lens, can be calculated from the following equation (1).

Further, a2 can be obtained by subtracting b1 calculated using the equation (1) from the known numerical value c1.
Further, using the subject distance a1, the distance b1, and the known numerical value x (the distance between the site of interest P and the optical axis OA) in the following equation (2), the site of interest P is imaged through the main lens ML. A distance x ′ between (imaging point PF) and the optical axis OA is calculated.
x ′ = x · b1 / a1 (2)
Further, the distance d ₁ from the optical axis OA to the principal point of the target sub lens SL, the distance x ′ calculated using the above formula (2), the distance c 2 from the sub lens array SLA to the imaging surface IE, and the distance The distance x ″ between the arrival point PE and the optical axis OA is calculated using a2 in the following equation (3).

  The same calculation is performed in the Y-axis direction, and the position of the reaching point PE is specified. An example of LFDI is shown in FIG. FIG. 18B shows an example of the arrangement of corresponding pixels on each LFDI sub-image SI. FIG. 18B shows that the central sub-image is a sub-image corresponding to MLBC, and the corresponding pixels indicated by white squares on the peripheral sub-images are shifted on the sub-image. ing.

  When the corresponding pixel is extracted in step S804, the depth coefficient of each corresponding pixel is acquired (step S805). Then, the pixel value of the pixel of interest is determined from the depth coefficient of the corresponding pixel acquired by the pixel value determination unit 3340 (step S806). Specifically, the mode value of the extracted depth coefficient of the corresponding pixel is selected, and the pixel value corresponding to the mode value is set as the pixel value of the target pixel.

Then, it is determined whether pixel values have been determined for all the reconstructed pixels (step S807). If there is an unprocessed reconstructed pixel (step S807; NO), the process is repeated from step S803 for the next reconstructed pixel. .
On the other hand, in the case of all reconstruction pixels processed (step S807; YES), as shown in FIG. 18 (c), so to generate the depth image DI that indicates the depth of the object on the reconstruction plane, The image generation process 1 ends.

  Returning to FIG. 5, when the reconstruction of the output image (depth image DI) is completed, the output unit 340 outputs the output image to the image storage unit 430 (step S106).

  Thereafter, the digital camera 1 outputs the reconstructed image stored in the image storage unit 430 to the display unit 520. Alternatively, it may be output to an external device using the I / O unit 510.

As described above, according to the image generation apparatus 30 of the present embodiment, the depth of the subject on the LFI is estimated, and a depth image indicating the depth of the subject on the reconstruction plane is generated and output based on the estimation result. I can do it.
The subject depth estimation results can be used for various purposes such as detecting the presence of objects that are too close, as well as the construction of a 3D model of the subject, as well as the use of noise removal and blurring in reconstructed images. it can.

  The image generation apparatus 30 according to the present embodiment defines an area that is considered to correspond to the same subject on the sub-image, and determines the degree of deviation of the subject on the sub-image for each area, and a depth coefficient for each area. Calculate to determine the corresponding area. Then, a depth coefficient is calculated from the deviation between the pixel area and the corresponding area to generate a depth image. Therefore, the depth estimation can be performed with higher accuracy even when there are factors that make it difficult to estimate the depth, such as noise, occlusion, and non-Lambertian surface, as compared with the case of obtaining the depth in pixel units.

  Further, the image generation apparatus 30 calculates an SSD for each of the regions (corresponding candidate regions) at positions corresponding to the depth coefficient of the peripheral sub-image, and defines the corresponding region using the SSD. For this reason, the depth coefficient can be calculated based on the difference between the pixel values of the pixel area and the corresponding candidate area. An area having a small difference from the area pixel value can be selected as the corresponding area from the corresponding candidate areas. Therefore, the depth coefficient reflecting the pixel value can be obtained.

  Further, in the present embodiment, the region depth coefficient is calculated using an SSSD obtained by adding the SSD calculated for each depth coefficient. For this reason, it is possible to calculate the depth coefficient of the region by comprehensively judging information obtained from a plurality of peripheral sub-images, resulting in high accuracy of depth estimation.

  Furthermore, since the depth coefficient of the LFDI is smoothed using the depth filter, the boundary line of the depth coefficient can be smoothed on the LFDI. Therefore, noise in the depth image of the output image can be reduced.

(Embodiment 2)
A digital camera and an image generation apparatus according to Embodiment 2 of the present invention will be described. The digital camera 1 of the present embodiment is the same as the digital camera 1 according to the first embodiment, except that the image generation apparatus is the image generation apparatus 31 shown in FIG. In this embodiment, it is assumed that the reconstruction setting includes information indicating that a reconstructed image of the subject is output, corrected based on the depth estimation result.

  As illustrated in FIG. 19, the image generation apparatus 31 according to the present embodiment includes an input unit 310, a depth estimation unit 320, a reconstructed image generation unit 331, and an output unit 340. The input unit 310, the depth estimation unit 320, and the output unit 340 are the same as parts having the same names in the image generation apparatus 30 according to the first embodiment.

  The reconstructed image generation unit 331 generates a reconstructed image in which the subject is reconstructed based on the LFI, the reconstruction setting, the shooting setting, and the depth (LFDI) of each pixel of the LFI estimated by the depth estimation unit 320. . For such processing, the reconstructed image generation unit 331 includes a template definition unit 3310, a target pixel selection unit 3320, a corresponding pixel extraction unit 3330, a depth coefficient determination unit 3341, and a pixel value calculation unit 3350. A reconstruction filter execution unit 3360.

  The template definition unit 3310, the target pixel selection unit 3320, and the corresponding pixel extraction unit 3330 are the same as the parts having the same names according to the first embodiment.

  The depth coefficient determination unit 3341 determines the depth coefficient of the reconstructed pixel by the same method as the pixel value determination unit 3340 according to the first embodiment determines the depth coefficient of the depth image.

The pixel value calculation unit 3350 calculates the pixel value of the target pixel based on the pixel value of the target pixel extracted by the corresponding pixel extraction unit 3330. Specifically, the pixel value calculation unit 3350 calculates the pixel value according to the following procedure in conjunction with the target pixel selection unit 3320 and the corresponding pixel extraction unit 3330.
1) The position (MLBC in FIG. 17) where the light beam from the pixel of interest passes through the main point of the main lens and reaches the sub lens array is specified.
2) The main lens blur MLB corresponding to the reconstruction surface is calculated around the specified position.
3) A sub lens in which a part or all of the sub lenses included in the sub lens array are included in the main lens blur is specified.
4) Select one of the specified sub-lenses.
5) The area where the selected sub-lens and the main lens blur overlap is calculated and divided by the area of the microlens to obtain a weighting coefficient.
6) A pixel value is acquired on the sub-image at a position where the light beam from the target pixel is imaged by the selected sub-lens.
7) A correction pixel value is obtained by multiplying the acquired pixel value by the above weighting coefficient.
8) A correction pixel value is calculated over all of the sub-lens part or all of which are included in the main lens blur, and the sum is obtained.
9) Divide the sum of the corrected pixel values by the sum of the overlapping areas to obtain the pixel value of the image to be reconstructed.

  The reconstruction filter execution unit 3360 executes a filter process (reconstruction filter) according to the difference between the distance between the reconstruction surface specified by the reconstruction setting and the main lens, and the depth of each reconstruction pixel.

  An example of filter processing executed on the reconstructed image will be described with reference to FIG. FIG. 20A shows a reconstructed image before filter processing. RI1 is an image with the front (lower left note) portion as a reconstruction plane, and RI2 is an example with the upper background as a reconstruction image. In RI1, noise is generated in the back portion, and in RI2, noise is generated in the front portion.

  FIG. 20B shows a result (depth image DI) of estimating the depth for RI1 and RI2. Here, it is assumed that the same depth image DI is obtained by RI1 and RI2. The right is an enlarged view. In the depth image, the pixel whose depth coefficient is in front is brighter. In FIG. 20B, among the three pixels indicated by white squares, the first pixel is the farthest object, the second pixel is the central part, and the third pixel is the foreground object. It is estimated to correspond.

The result of performing blur addition on each pixel of the reconstructed image is images RI3 and RI4 in FIG. RI3 is an image obtained by applying a filter corresponding to the reconstruction distance to RI1 and RI4 is applied to RI2.
And Oh standing to filter, with reference to the depth table, and depth coefficient reconstruction distance corresponding, when the difference between the depth factor of the reconstructed pixel is a predetermined threshold or more, the blur adding processing according blur intensity set I do. For example, a Gaussian filter is used for the blur addition process.

  When adding blur, only pixels behind the target pixel are used for blurring, so that pixels behind the reconstruction distance are not blurred by pixels behind, and the pixels before this are not used. For example, when performing blur addition on the pixel 2, among the pixels in a predetermined blur addition application region (the large square region on the right side of FIG. 20B), the pixel corresponding to the back (the first pixel) is blurred. However, the pixel in front (the third pixel) is not added to the blur. Further, the intensity of blur addition is adjusted so as to add blur so that the pixel farther from the reconstruction distance is strong and the pixel near is weak. Then, the reconstructed pixel after filtering is transmitted to the output unit 340.

  Here, a process (image output process, FIG. 5) for generating and outputting a reconstructed image from the LFI executed by the image generating apparatus 31 will be described. The image output process executed by the image generation apparatus 31 is the same as the process of the same name according to the first embodiment except that the image generation process executed in step S105 is the image generation process 2 shown in FIG.

  Image generation processing 2 according to the present embodiment will be described with reference to FIG. In the image generation process 2, the reconstructed image generation unit 331 executes steps S901 to S904 in the same manner as steps S801 to S804 of the image generation process 1 in FIG.

  Thereafter, the pixel value of the corresponding pixel is acquired in step S905, and the depth coefficient of the corresponding pixel is acquired in step S906.

Then, the depth coefficient determination unit 3341 determines the depth coefficient of the reconstructed pixel by the same method in which the depth coefficient of the pixel of the depth image is determined in step S805 of the image generation process 1 in FIG. 16 (step S907).
Further, the pixel value calculation unit 3350 calculates the pixel value of the target pixel by the method using the above-described weighted addition (step S908).

Next, it is determined whether or not the above process has been completed for all the reconstructed pixels (step S909). If there is an unprocessed pixel (step S909; NO), the process is repeated from step S903 for the next pixel.
On the other hand, if all the pixels have been processed (step S909; YES), the reconstruction filter execution unit 3360 executes the filter processing by the above-described method.
Then, the image generation process 2 ends.

  As described above, according to the image generation device 31 of the present embodiment, the depth of the subject corresponding to the reconstructed pixel is estimated, and the filtering process is performed using the estimation result. Therefore, a reconstructed image with high image quality can be generated and output.

In particular, the difference between the reconstruction distance and the depth of the reconstructed pixel is obtained, and the blur is not added to the portion where the difference is small, and the blur is added to the portion where the difference is large. Distant parts can be blurred. For this reason, it is possible to generate an image with a high degree of user satisfaction by adding a blur according to the depth.

In addition, the greater the difference in distance, the stronger the blur and the weaker the blur when it is smaller, so it is possible to generate an image with high user satisfaction that reflects the difference between the depth and the reconstruction plane and has a different blur. .
Furthermore, when blurring is added, the pixel in front is not added to the blur. For this reason, the color of the object in front is not blurred, and an image with high image quality reflecting the depth can be generated and output.

(Modification)
The present invention is not limited to the above embodiment, and various modifications can be made.
For example, in the first and second embodiments, the correspondence between the image shift and the SSD (graph in FIG. 9C) is obtained for the assumed range of the peripheral sub-image, and the corresponding region is defined by obtaining the SSSD for each depth coefficient. . However, the method of defining the corresponding area is not limited to this, and the depth coefficient of each pixel may be voted with the area corresponding to the depth coefficient having the smallest SSD for each peripheral sub-image as the corresponding area.

  In the first and second embodiments, the depth filter is applied to LFDI, and then a reconstructed image is generated. However, a filter related to the depth coefficient on the reconstructed image is not applied to the LFDI. Processing may be executed.

In the first and second embodiments, the main lens is a single focal lens. However, the main lens is not limited to this, and may be a lens whose focal length can be changed. At this time, a plurality of depth tables may be prepared and the depth table corresponding to the shooting parameter may be used. Specifically, a table in which the focal length of the main lens indicated by the shooting setting is associated with the depth table used in that case is prepared, and the optimum depth table with the acquired shooting setting as an argument for each image output process Shall be extracted and used.

  In the above embodiment, the image is described as a grayscale image. However, the image to be processed according to the present invention is not limited to a grayscale image. For example, the image may be an RGB image in which three pixel values of R (red), G (green), and B (blue) are defined for each pixel. In this case, the pixel value is similarly processed as an RGB vector value. Alternatively, the above processing may be performed using each of R, G, and B values as independent grayscale images. According to this configuration, a reconstructed image that is a color image can be generated from a light field image that is a color image.

  In addition, the hardware configuration and the flowchart described above are merely examples, and can be arbitrarily changed and modified.

  A central part that performs processing for generating a reconstructed image composed of the information processing unit 31a, the main storage unit 32, the external storage unit 33, and the like is not based on a dedicated system, but using a normal computer system. It is feasible. For example, a computer program for executing the above operation is stored and distributed in a computer-readable recording medium (flexible disk, CD-ROM, DVD-ROM, etc.), and the computer program is installed in the computer. You may comprise the center part which performs the process for image reconstruction. Further, the reconstructed image generating apparatus may be configured by storing the computer program in a storage device included in a server device on a communication network such as the Internet, and downloading it by a normal computer system.

  When realizing the function of the reconfiguration generation apparatus by sharing the OS (operating system) and application program, or by cooperating between the OS and application program, store only the application program part in a recording medium or storage device. Also good.

  It is also possible to superimpose a computer program on a carrier wave and distribute it via a communication network. For example, the computer program may be posted on a bulletin board (BBS: Bulletin Board System) on a communication network, and the computer program may be distributed via the network. The computer program may be started and executed in the same manner as other application programs under the control of the OS, so that the above-described processing may be executed.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to the specific embodiment which concerns, This invention includes the invention described in the claim, and its equivalent range It is. Hereinafter, the invention described in the scope of claims of the present application will be appended.

(Appendix 1)
A depth estimation device that estimates the depth of a subject from a light field image composed of a plurality of sub-images,
Pixel area defining means for defining a pixel area corresponding to a target subject to be subjected to the depth estimation on a sub-image of the light field image;
The position of the pixel area defined by the pixel area defining means on the target sub-image including the pixel area and the corresponding area corresponding to the pixel area on the peripheral sub-image arranged around the target sub-image. And a depth estimation means for estimating the depth of the target subject based on the positional deviation of the position,
A depth estimation apparatus comprising:

(Appendix 2)
A corresponding area extracting unit that extracts a corresponding area on the peripheral sub-image based on a pixel value of a pixel area of the target sub-image and a pixel value of the peripheral sub-image;
The depth estimation means includes
The depth of the target subject is estimated based on a positional shift between the position of the pixel area on the target sub-image and the position of the corresponding area extracted by the corresponding area extraction unit on the peripheral sub-image. ,
The depth estimation apparatus according to supplementary note 1, wherein:

(Appendix 3)
Reconstructed image defining means for defining a reconstructed image obtained by reconstructing an image of a subject from the light field image and a reconstructed pixel that is a pixel thereof;
Depth coefficient acquisition for obtaining a depth coefficient of a sub-pixel included in the pixel area based on a positional shift between the position of the pixel area on the target sub-image and the position of the corresponding area on the peripheral sub-image Means,
Further comprising
The depth estimation means includes
Estimating the depth of the subject by determining the depth coefficient of the pixel of the reconstructed image based on the depth coefficient of each sub-pixel obtained by the depth coefficient acquisition means;
The depth estimation apparatus according to Supplementary Note 1 or 2, wherein:

(Appendix 4)
Corresponding pixel extraction means for extracting corresponding pixels on the light field image corresponding to the reconstructed pixels,
The depth estimation means includes
Estimating the depth of the subject by determining the depth coefficient of the reconstructed pixel based on the depth coefficient obtained by the depth coefficient acquisition means for the corresponding pixel extracted by the corresponding pixel extraction means;
The depth estimation apparatus according to Supplementary Note 3, wherein

(Appendix 5)
Corresponding coefficient acquisition for defining a corresponding candidate area at a plurality of positions according to parallax on the peripheral sub-image and obtaining a corresponding coefficient indicating the certainty of correspondence with the pixel area of the sub-image for each of the corresponding candidate areas Further comprising means,
The corresponding area extracting means includes
Extracting a corresponding region on the peripheral sub-image based on the corresponding coefficient obtained by the corresponding coefficient acquiring means;
The depth estimation apparatus according to Supplementary Note 2, wherein

(Appendix 6)
The corresponding coefficient acquisition means is
Defining the corresponding candidate area for each of the plurality of peripheral sub-images for each different parallax, and determining the corresponding coefficient for each of the corresponding candidate areas;
The corresponding area extracting means includes
A likelihood coefficient indicating the likelihood of the corresponding coefficient is obtained for each parallax from the corresponding coefficient determined for each of the corresponding candidate areas, and the corresponding candidate area corresponding to the corresponding coefficient with the most likely likelihood coefficient is the corresponding Extract as a region,
The depth estimation apparatus according to Supplementary Note 5, wherein

(Appendix 7)
The correspondence coefficient obtained by the correspondence coefficient acquisition unit is a coefficient indicating a difference in pixel values between the pixel area and the correspondence candidate area.
The depth estimation apparatus according to Supplementary Note 5 or 6, characterized in that:

(Appendix 8)
The depth coefficient acquisition means includes
Correcting the depth coefficient of the sub-pixel based on the depth coefficient of the sub-pixel included in the predetermined area from the sub-pixel;
The depth estimation apparatus according to supplementary note 3 or 4, characterized in that:

(Appendix 9)
A reconstructed image generating device that generates a reconstructed image focused on a predetermined reconstruction distance from a light field image,
The depth estimation apparatus according to any one of appendices 1 to 8,
Reconstructed image correcting means for correcting a pixel value of a reconstructed pixel constituting the reconstructed image based on a difference in distance between the depth of the subject estimated by the depth estimating device and the reconstructed distance;
A reconstructed image generating apparatus comprising:

(Appendix 10)
The reconstructed image correcting unit reconstructs a reconstructed pixel in which a difference between the depth of the corresponding subject and the reconstructed distance is larger than a predetermined threshold as a correction within a predetermined range from the reconstructed pixel. Add blur based on pixel value of pixel,
The reconstructed image generating apparatus according to Supplementary Note 9, wherein

(Appendix 11)
The reconstructed image correcting means does not add the blur to the reconstructed pixel of the subject whose depth of the corresponding subject is in front of the reconstruction distance when adding the blur. ,
The reconstructed image generating apparatus according to Supplementary Note 10, wherein:

(Appendix 12)
A depth estimation method for estimating the depth of a subject from a light field image composed of a plurality of sub-images,
Defining a pixel area corresponding to a target subject to be subject to the depth estimation on a sub-image of the light field image;
The position of the defined pixel area on the target sub-image including the pixel area and the position of the corresponding area corresponding to the pixel area on the peripheral sub-image arranged around the target sub-image Estimating the depth of the target subject based on the deviation of
The depth estimation method characterized by including.

(Appendix 13)
A reconstructed image generation method for generating a reconstructed image focused on a predetermined reconstruction distance from a light field image composed of a plurality of sub-images,
Defining a pixel area corresponding to a target subject to be subject to the depth estimation on a sub-image of the light field image;
The position of the defined pixel area on the target sub-image including the pixel area and the position of the corresponding area corresponding to the pixel area on the peripheral sub-image arranged around the target sub-image Determining the depth coefficient of the target subject for each pixel constituting the pixel area based on
Extracting corresponding pixels on the light field image corresponding to the reconstructed pixels constituting the reconstructed image;
Setting a depth coefficient of the reconstructed pixel corresponding to the corresponding pixel based on the determined depth coefficient;
Correcting the pixel value of the reconstructed pixel based on the difference between the distance to the subject indicated by the depth coefficient of the reconstructed pixel and the reconstructed distance;
Generating a reconstructed image using the corrected pixel value of the reconstructed pixel;
A reconstructed image generation method comprising:

(Appendix 14)
To estimate the depth of a subject from a light field image composed of multiple sub-images,
A function of defining a pixel region corresponding to a target subject to be subjected to the depth estimation on a sub-image of the light field image;
The position of the defined pixel area on the target sub-image including the pixel area and the position of the corresponding area corresponding to the pixel area on the peripheral sub-image arranged around the target sub-image A function of estimating the depth of the target subject based on the deviation of
A program characterized by realizing.

(Appendix 15)
In order to generate a reconstructed image focused on a predetermined reconstruction distance from a light field image composed of a plurality of sub-images,
A function of defining a pixel region corresponding to a target subject to be subjected to the depth estimation on a sub-image of the light field image;
The position of the defined pixel area on the target sub-image including the pixel area and the position of the corresponding area corresponding to the pixel area on the peripheral sub-image arranged around the target sub-image A function for obtaining the depth coefficient of the target subject for each pixel constituting the pixel area based on the deviation of
A function of extracting corresponding pixels on the light field image corresponding to the reconstructed pixels constituting the reconstructed image;
A function of setting a depth coefficient of the reconstructed pixel corresponding to the corresponding pixel based on the obtained depth coefficient;
A function of correcting the pixel value of the reconstructed pixel based on the difference in distance between the distance to the subject indicated by the depth coefficient of the reconstructed pixel and the reconstructed distance;
A function of generating a reconstructed image using the corrected pixel value of the reconstructed pixel;
A program characterized by realizing.

DESCRIPTION OF SYMBOLS 1 ... Digital camera, 10 ... Imaging part, 110 ... Optical apparatus, 111 ... Shutter, 120 ... Image sensor, 20 ... Information processing part, 30 ... Image generation apparatus, 31 ... Image generation apparatus, 31a ... Information processing part, 32 ... Main storage unit 33 ... external storage unit 36 ... input / output unit 37 ... internal bus 38 ... program 210 ... image processing unit 220 ... imaging control unit 310 ... input unit 3110 ... LFI acquisition unit 3120 ... Reconstruction setting acquisition unit, 3130 ... Shooting setting acquisition unit, 320 ... Depth estimation unit, 3210 ... Area definition unit, 3220 ... Region of interest selection unit, 3230 ... Peripheral sub-image selection unit, 3240 ... SSD calculation unit, 3250 ... Depth coefficient Voting unit, 3260 ... sub-pixel depth coefficient selection unit, 3270 ... depth filter execution unit, 330 ... depth image generation unit, 331 ... reconstructed image generation unit, 310 ... Template definition unit, 3320 ... Target pixel selection unit, 3330 ... Corresponding pixel extraction unit, 3340 ... Pixel value determination unit, 3341 ... Depth coefficient determination unit, 3350 ... Pixel value calculation unit, 3360 ... Reconstruction filter execution unit, 340: output unit, d1 to d8: deviation direction, MinD: depth coefficient with minimum SSSD, 40 ... storage unit, 410 ... imaging setting storage unit, 420 ... reconstruction setting storage unit, 430 ... image storage unit, 50 ... interface Part (I / F part), 510 ... I / O part, 520 ... display part, 530 ... operation part, LFI ... light field image, LFDI ... light field depth image, OA ... optical axis, OB ... subject, POB ... subject , P ... attention site, ML ... main lens, PF ... imaging point, MIP ... main lens imaging plane, PE ... arrival point, IE ... imaging plane, SLA ... Zuarei, SL ... sub-lens, MLB ... the main lens blur, MLBC ... the main lens blur center, SI ... _{sub-images, S} 11 ~S _{MN ...} sub-image, SI1~SI8 ... sub-image, DI ... depth image, RI1, RI2 ... Reconstructed image before execution of reconstruction filter, RI3, RI4 ... Reconstructed image after execution of reconstruction filter, RF ... Reconstruction plane

Claims (14)

A depth estimation device that estimates the depth of a subject from a light field image composed of a plurality of sub-images,
On the sub-image of the light field image, a pixel region defining means for defining a pixel area corresponding to the object,
And position on the target sub-image comprising a pixel area defined by the pixel area defining means, a position of the corresponding region corresponding to the picture element region on the peripheral sub-images arranged in the periphery of the target sub-images, the Depth estimation means for estimating the depth of the subject based on a positional shift;
Reconstructed image defining means for defining a reconstructed image obtained by reconstructing the image of the subject from the light field image and a reconstructed pixel that is a pixel of the reconstructed image;
Depth coefficient acquisition for obtaining a depth coefficient of a sub-pixel included in the pixel area based on a positional shift between the position of the pixel area on the target sub-image and the position of the corresponding area on the peripheral sub-image Means,
Bei to give a,
The depth estimation means estimates the depth of the subject by determining the depth coefficient of the reconstructed pixel of the reconstructed image based on the depth coefficient of each subpixel obtained by the depth coefficient acquisition means.
A depth estimation apparatus characterized by that. The corresponding region on the peripheral sub-images, further comprising: a pixel value of the pixel area of the target sub-images, the pixel values of the peripheral sub-images, the corresponding area extracting unit that extracts, based on,
The depth estimation means includes
Wherein the position on the target sub-image of the pixel area, the corresponding area and position on the peripheral sub-images of said corresponding area extracted by the extraction means, based on the deviation of the position of, estimating the depth of the subject ,
The depth estimation apparatus according to claim 1. Corresponding pixel extraction means for extracting corresponding pixels on the light field image corresponding to the reconstructed pixels,
The depth estimation means includes
Wherein for said corresponding pixel extracted by the corresponding pixel extracting means based on the depth coefficient is the depth factor acquisition means determined by determining the depth coefficient of the reconstructed pixels to estimate the depth of the subject,
The depth estimation apparatus according to claim 1 or 2 , wherein Define the corresponding candidate region into a plurality of positions corresponding to the parallax on the peripheral sub-images, wherein for each of the corresponding candidate area corresponding coefficient acquiring seeking corresponding coefficient indicating the correspondence between certainty of a pixel region of the sub-images Further comprising means,
The corresponding area extracting means includes
Based on the corresponding coefficients the corresponding coefficient acquiring means is determined, and extracts the corresponding region on the peripheral sub-images,
The depth estimation apparatus according to claim 2. The corresponding coefficient acquisition means is
Defines the corresponding candidate region to each of the plurality of the peripheral sub-images for different parallax calculated the corresponding coefficients for each of said corresponding candidate region,
The corresponding area extracting means includes
Said corresponding candidate region said from the corresponding coefficients obtained for each of the corresponding candidate area, obtains a likelihood factor indicating the probability of the corresponding coefficient for each of the parallax, the likelihood factor corresponding to the most probable said corresponding coefficient Is extracted as the corresponding region,
The depth estimation apparatus according to claim 4 . Wherein the corresponding coefficients corresponding coefficient acquiring means finding is a coefficient indicating a difference in pixel value between the pixel region and the corresponding candidate area,
The depth estimation apparatus according to claim 4 or 5 , characterized by the above. The depth coefficient acquisition means includes
The depth factor of the sub-pixel is corrected based on the depth factor of the sub-pixel where the contained sub-pixel in a predetermined area,
The depth estimation apparatus according to any one of claims 1 to 6 . A reconstructed image generating device that generates a reconstructed image focused on a predetermined reconstruction distance from a light field image composed of a plurality of sub-images ,
A pixel area defining means for defining a pixel area corresponding to a subject on the sub-image of the light field image; a position on the target sub-image including the pixel area defined by the pixel area defining means; A depth estimation device comprising: a depth estimation unit configured to estimate the depth of the subject based on a position shift of a corresponding area corresponding to a pixel area on a peripheral sub-image disposed around the image; and
Based on the difference in distance between the depth and the reconstruction distance of the object estimated by the depth estimation apparatus, a reconstruction image correcting means for correcting the pixel value of the reconstructed pixels constituting the reconstructed image,
Equipped with a,
A reconstructed image generating apparatus characterized by that. The reconstructed image correcting means, as the correction, the reconstructed pixels difference is greater than a predetermined threshold value of the distance between depth and the reconstruction distance of the corresponding object is in the predetermined range from the reconstructed pixel re Add blur based on the pixel values of the constituent pixels.
The reconstructed image generating apparatus according to claim 8 . The reconstructed image correction means, when adding the blur, depth of the corresponding object, said the reconstructed pixels of the subject is short of the reconstruction distance, and processed by the process of adding the blur do not do,
The reconstructed image generating apparatus according to claim 9 . A depth estimation method for estimating the depth of a subject from a light field image composed of a plurality of sub-images,
Defining a pixel region corresponding to the subject on a sub-image of the light field image;
And position on the target sub-image comprising a pixel area of the defined, based on the position and, in the position of displacement of the corresponding region corresponding to the picture element region on the target sub-peripheral sub images that are arranged around the image Estimating the depth of the subject ;
Defining a reconstructed image obtained by reconstructing the image of the subject from the light field image and a reconstructed pixel that is a pixel of the reconstructed image;
Obtaining a depth coefficient of a sub-pixel included in the pixel area based on a positional shift between the position of the pixel area on the target sub-image and the position of the corresponding area on the peripheral sub-image;
Only including,
In the step of estimating the depth, the depth of the subject is determined by determining the depth coefficient of the reconstructed pixel of the reconstructed image based on the depth coefficient of each of the sub-pixels obtained in the step of obtaining the depth coefficient. presume,
A depth estimation method characterized by the above. A reconstructed image generation method for generating a reconstructed image focused on a predetermined reconstruction distance from a light field image composed of a plurality of sub-images,
On the sub-image of the light field image, comprising the steps of defining a pixel area corresponding to the object of interest to estimate the depth,
And position on the target sub-image comprising a pixel area of the defined, based on the position and, in the position of displacement of the corresponding region corresponding to the picture element region on the target sub-peripheral sub images that are arranged around the image Te, determining a depth coefficient of the object for each pixel constituting the picture element region,
Extracting corresponding pixels on the light field image corresponding to the reconstructed pixels constituting the reconstructed image;
A step of, based on calculated meth the depth factor, to set the depth coefficient of the reconstructing pixel corresponding to the corresponding pixels,
Correcting the pixel value of the reconstructed pixel based on the difference between the distance to the subject indicated by the depth coefficient of the reconstructed pixel and the reconstructed distance;
Generating a reconstructed image using the corrected pixel value of the reconstructed pixel;
Including,
A reconstructed image generation method characterized by the above. To estimate the depth of a subject from a light field image composed of multiple sub-images,
A function for defining a pixel region corresponding to the subject on the sub-image of the light field image;
And position on the target sub-image comprising a pixel area of the defined, based on the position and, in the position of displacement of the corresponding region corresponding to the picture element region on the target sub-peripheral sub images that are arranged around the image A function for estimating the depth of the subject ,
A function for defining a reconstructed image obtained by reconstructing the image of the subject from the light field image and a reconstructed pixel that is a pixel of the reconstructed image;
A function for obtaining a depth coefficient of a sub-pixel included in the pixel area based on a positional shift between a position of the pixel area on the target sub-image and a position of the corresponding area on the peripheral sub-image;
Realized ,
In the function of estimating the depth, the depth of the subject is determined by determining the depth coefficient of the reconstructed pixel of the reconstructed image based on the depth coefficient of each subpixel obtained by the function of obtaining the depth coefficient. presume,
A program characterized by that. In order to generate a reconstructed image focused on a predetermined reconstruction distance from a light field image composed of a plurality of sub-images,
A function for defining a pixel region corresponding to a subject whose depth is to be estimated on a sub-image of the light field image;
And position on the target sub-image comprising a pixel area of the defined, based on the position and, in the position of displacement of the corresponding region corresponding to the picture element region on the target sub-peripheral sub images that are arranged around the image Te, function of obtaining a depth coefficient of the object for each pixel constituting the picture element region,
A function of extracting corresponding pixels on the light field image corresponding to the reconstructed pixels constituting the reconstructed image;
Based on the determined metadata depth factor, the ability to set the depth coefficient of the reconstructing pixel corresponding to the corresponding pixels,
Function based on said difference in distance of the distance to the subject indicated by the depth factor of reconstructed pixel and the reconstruction distance, corrects the pixel value of the reconstructed pixels,
Function of generating a reconstructed image by using the pixel values after compensation of the reconstructed pixel,
To realize ,
A program characterized by that.