JP5644468B2 - IMAGING DEVICE AND IMAGING DEVICE CONTROL PROGRAM - Google Patents

Description

  The present invention relates to an image arithmetic device, an imaging device, and an image arithmetic program.

A technique is known in which scene depth information is obtained from a single piece of image data by photographing a scene with a camera in which a structured aperture composed of RGB color filters is arranged at the aperture position of the lens optical system ( For example, Non-Patent Document 1).
[Prior art documents]
[Non-Patent Document 1] Depth estimation and foreground mat extraction using a color filter for the diaphragm of a camera (Visual Computing / Graphics and CAD Joint Symposium 2008)

  As described above, in order to obtain depth information, a structured aperture that is a mask having at least a special aperture shape is arranged in the optical system. Therefore, it is necessary to retract the structured aperture from the subject light beam when photographing a normal photographed image for viewing and the like, and to insert the structured aperture into the subject light beam when photographing a photographed image for obtaining distance information. It was.

  In order to solve the above-described problem, an image calculation apparatus according to the first aspect of the present invention includes an acquisition unit that acquires an infrared image obtained by imaging a subject irradiated with an infrared light pattern, and a red image captured in the infrared image. A calculation unit that calculates depth information of the subject based on edge information of the external light pattern.

  In order to solve the above-described problem, an imaging device according to a second aspect of the present invention is an imaging device including the above-described image calculation device, an infrared irradiation unit that irradiates an infrared light pattern, and an infrared light wavelength. And an imaging device having an infrared light receiving pixel having sensitivity in a band.

  In order to solve the above-described problem, an image calculation program according to the third aspect of the present invention includes an acquisition step of acquiring an infrared image obtained by imaging a subject irradiated with an infrared light pattern, and red captured in the infrared image. Based on the edge information of the external light pattern, the computer executes a calculation step for calculating the depth information of the subject.

  It should be noted that the above summary of the invention does not enumerate all the necessary features of the present invention. In addition, a sub-combination of these feature groups can also be an invention.

It is principal part sectional drawing of the single-lens reflex camera which concerns on this embodiment. It is explanatory drawing of the pixel filter arrange | positioned on the pixel of an image pick-up element. It is a figure which shows the relationship between the wavelength range which the pixel of an image pick-up element has a sensitivity, and the wavelength range of the infrared light pattern which a pattern irradiation part irradiates. 1 is a block diagram schematically showing a system configuration of a single-lens reflex camera. It is explanatory drawing which shows the mode of pattern irradiation. It is explanatory drawing explaining the difference in the mode of pattern irradiation in a depth direction. It is a figure explaining a blur map production | generation process. It is a processing flowchart explaining an imaging sequence. It is sectional drawing of the interchangeable lens camera which has arrange | positioned the image sensor for infrared light, and the image sensor for visible light.

  Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. In addition, not all the combinations of features described in the embodiments are essential for the solving means of the invention.

  FIG. 1 is a cross-sectional view of a main part of a single-lens reflex camera 200 according to the present embodiment. The single-lens reflex camera 200 functions as an imaging device by combining a lens unit 210 that is a photographing lens and a camera unit 230 that is a camera body.

  The lens unit 210 includes a lens group 211 arranged along the optical axis 201. The lens group 211 includes a focus lens 212 and a zoom lens 213. The lens unit 210 includes a lens system control unit 216 that controls and calculates the lens unit 210 such as driving the focus lens 212. Each element constituting the lens unit 210 is supported by the lens barrel 217.

  The lens unit 210 includes a lens mount 218 at a connection portion with the camera unit 230, and engages with the camera mount 231 included in the camera unit 230 to be integrated with the camera unit 230. Each of the lens mount 218 and the camera mount 231 includes an electrical connection portion in addition to a mechanical engagement portion, and realizes power supply from the camera unit 230 to the lens unit 210 and mutual communication.

  The camera unit 230 includes a main mirror 232 that reflects a subject image incident from the lens unit 210 and a focus plate 234 on which the subject image reflected by the main mirror 232 is formed. The main mirror 232 rotates around the rotation axis 233 and can take a state of being obliquely provided in the subject light flux centered on the optical axis 201 and a state of being retracted from the subject light flux. When the subject image is guided to the focus plate 234 side, the main mirror 232 is provided obliquely in the subject light flux. Further, the focus plate 234 is disposed at a position conjugate with the light receiving surface of the image sensor 243.

  The subject image formed on the focusing screen 234 is converted into an erect image by the pentaprism 235 and observed by the user via the eyepiece optical system 236. An AE sensor 237 is disposed above the exit surface of the pentaprism 235, and detects the luminance distribution of the subject image.

  The region near the optical axis 201 of the main mirror 232 in the oblique state is formed as a half mirror, and a part of the incident light beam is transmitted. The transmitted light beam is reflected by the sub mirror 238 that operates in conjunction with the main mirror 232, and is guided to the AF optical system 239. The subject luminous flux that has passed through the AF optical system 239 enters the AF sensor 240. The AF sensor 240 detects a phase difference signal from the received subject light beam. The sub mirror 238 retracts from the subject light beam in conjunction with the main mirror 232 when the main mirror 232 retracts from the subject light beam.

  A focal plane shutter 241, an optical low-pass filter 242, and an image sensor 243 are arranged along the optical axis 201 behind the oblique main mirror 232. The focal plane shutter 241 takes an open state when the subject light flux is guided to the image sensor 243, and takes a shielding state at other times. The optical low-pass filter 242 plays a role of adjusting the spatial frequency of the subject image with respect to the pixel pitch of the image sensor 243. The image sensor 243 is a photoelectric conversion element such as a CMOS sensor, for example, and converts the subject image formed on the light receiving surface into an electric signal.

  The electrical signal photoelectrically converted by the image sensor 243 is processed into image data by the image processing unit 246 which is a DSP mounted on the main board 244. In addition to the image processing unit 246, a camera system control unit 245, which is an MPU that integrally controls the system of the camera unit 230, is mounted on the main board 244. The camera system control unit 245 manages the camera sequence and performs input / output processing of each component.

  A display unit 247 such as a liquid crystal monitor is disposed on the back surface of the camera unit 230, and a subject image processed by the image processing unit 246 is displayed. The display unit 247 displays not only a still image after shooting, but also an EVF image as a viewfinder, various menu information, shooting information, and the like. The camera unit 230 houses a detachable secondary battery 248 and supplies power to the lens unit 210 as well as the camera unit 230.

  A flash 249 is provided in the vicinity of the pentaprism 235 to irradiate the subject under the control of the camera system control unit 245. In addition to the flash 249, the camera unit 230 includes a pattern irradiation unit 250 in the vicinity of the pentaprism 235. The pattern irradiation unit 250 is a unit that irradiates the subject with an infrared light pattern. The pattern irradiation unit 250 includes an infrared light emission unit 251 that emits infrared light, a pattern mask 252 on which a lattice pattern to be irradiated to a subject is printed, and an irradiation lens 253.

  In addition, a wavefront coding optical element (WCOE) 254 is disposed in the irradiation light flux of the infrared light pattern, particularly in the vicinity of the pupil of the irradiation lens 253. The WCOE 254 is an optical element that makes the depth of focus of the infrared light pattern significantly deeper than that of the irradiation lens 253 alone. That is, by disposing the WCOE 254 in the irradiation light beam of the infrared light pattern, blurring of the infrared light pattern projected on the defocus plane can be suppressed. The WCOE 254 has a phase curved surface defined by a cubic function, for example.

  The pattern irradiation unit 250 further includes a plurality of WCOEs 254 outside the irradiation light beam of the infrared light pattern, and a WCOE replacement unit 255 for replacing one of them with the WCOE 254 in the irradiation light beam. The plurality of WCOEs 254 have different characteristics. Specifically, each WCOE 254 has a different expansion factor. The expansion magnification is the ratio of the depth of focus when the WCOE 254 is disposed to the depth of focus of the irradiation lens 253 alone.

  FIG. 2 is an explanatory diagram of pixel filters arranged on the pixels of the image sensor 243. As shown in the figure, the pixel filter array in this embodiment includes four pixels as a set, and an R pixel filter, a G pixel filter, a B pixel filter, and an IR pixel filter are provided on each pixel. The IR pixel filter is a filter that transmits the wavelength band of infrared light. Therefore, the wavelength band in which each pixel has sensitivity is regulated by the pixel filter provided for each pixel. As the entire image sensor 243, each of the two-dimensionally arranged pixels is discretely provided with one of an R pixel filter, a G pixel filter, a B pixel filter, and an IR pixel filter. It can be said that the incident subject luminous flux is detected separately for each wavelength band. In other words, the image sensor 243 separates the subject image formed on the light receiving surface into four wavelength bands of RGB + IR and performs photoelectric conversion.

  FIG. 3 is a diagram illustrating the relationship between the wavelength band in which the pixels of the image sensor 243 have sensitivity and the wavelength band of the infrared light pattern irradiated by the pattern irradiation unit 250. In the figure, the vertical axis represents transmittance (%), and the horizontal axis represents wavelength (nm). It represents that the light having a higher transmittance reaches the photodiode constituting the pixel.

  B curve 301 indicates the sensitivity of the pixel provided with the B pixel filter, similarly G curve 302 indicates the sensitivity of the pixel provided with the G pixel filter, and R curve 303 indicates the sensitivity of the pixel provided with the R pixel filter. The IR curve 304 indicates the sensitivity of a pixel provided with an IR pixel filter. The wavelength band of the infrared light pattern irradiated by the pattern irradiation unit 250 is represented by an arrow 311.

  That is, the infrared light pattern reflected by the subject is received only by the infrared light receiving pixels provided with the IR pixel filter. In other words, the RGB pixel provided with the R pixel filter, the G pixel filter, and the B pixel filter cannot capture an infrared light pattern. That is, the infrared light pattern does not affect the RGB pixels that receive the visible light wavelength band.

  Then, if only the outputs of the infrared light receiving pixels are collected, a subject image in which the infrared light pattern is reflected can be formed. The subject image is an infrared image, and depth information can be acquired by performing arithmetic processing described later on the image.

  At the same time, by collecting the outputs of the RGB pixels provided with the R pixel filter, the G pixel filter, and the B pixel filter, it is possible to form a color subject image that is not affected by the infrared light pattern. That is, it is possible to acquire a main photographic image as a normal subject image that can endure viewing. Therefore, it is possible to acquire both the infrared image in which the infrared light pattern is reflected and the actual captured image that is not affected by the infrared light pattern by a single photographing operation.

  Next, the system configuration of the single-lens reflex camera 200 according to the present embodiment will be described. FIG. 4 is a block diagram schematically showing the system configuration of a single-lens reflex camera. The system of the single-lens reflex camera 200 includes a lens control system centered on the lens system control unit 216 and a camera control system centered on the camera system control unit 245 corresponding to the lens unit 210 and the camera unit 230, respectively. Is done. The lens control system and the camera control system exchange various data and control signals with each other via a connection portion connected by the lens mount 218 and the camera mount 231.

  The image processing unit 246 included in the camera control system generates image data as an actual captured image from an imaging signal composed of RGB pixel outputs photoelectrically converted by the imaging device 243 in accordance with a command from the camera system control unit 245. In addition, the image processing unit 246 processes an imaging signal formed from the output of the infrared light receiving element to generate an infrared image. The generated infrared image is transferred to the blur map calculation unit 261, and the blur map calculation unit 261 generates a blur map by a method described later. The blur map expresses depth information for the actual captured image. The image processing unit 246 performs blur processing on the image data of the actual captured image with reference to the generated blur map. The display unit 247 displays the image data that has been subjected to the blur processing. Thereafter, the image processing unit 246 processes the image data into a predetermined image format, and the camera system control unit 245 records the image data in the external memory via the external connection IF 264.

  The camera memory 262 is a non-volatile memory such as a flash memory, for example, and plays a role of recording a program for controlling the single-lens reflex camera 200, various parameters, and the like in addition to a temporary recording place of the generated image data. . The work memory 263 is a memory that can be accessed at a high speed, such as a RAM, and has a role of temporarily storing image data being processed.

  The release switch 265 has a two-stage switch position in the push-in direction, and the camera system control unit 245 detects phase difference information from the AF sensor 240 when detecting that the first-stage switch SW1 is turned on. get. Then, the WCOE replacement unit 255 is driven based on the acquired phase difference information, and the selected WCOE 254 is arranged in the irradiation light beam of the infrared light pattern. Also, the exposure value is determined by obtaining the luminance distribution of the subject image from the AE sensor 237. Further, when detecting that the second-stage switch SW2 is turned on, the camera system control unit 245 emits the infrared light emitting unit 251 to irradiate the subject with the infrared light pattern, and is set in advance. The photographing operation is executed according to the processing flow. A specific processing flow will be described later.

  The lens system control unit 216 receives various control signals from the camera system control unit 245 and executes various operations. The lens memory 221 stores information unique to the lens unit 210. For example, the lens memory 221 provides the focal length of the lens unit 210, open aperture information, and the like to the camera system control unit 245 via the lens system control unit 216.

  FIG. 5 is an explanatory diagram showing the pattern irradiation. FIG. 5A shows the state of the subject before pattern irradiation. As shown in the figure, a table 501 on which an object is placed, a rear wall surface 502, and a blind 503 that covers a window provided on the wall surface 502 exist as subjects.

  FIG. 5B shows a state where the subject is irradiated with the infrared light pattern 510 by the pattern irradiation unit 250. Under the control of the camera system control unit 245, the pattern irradiation unit 250 emits an infrared light emitting unit 251 configured by, for example, a high-intensity LED, and irradiates the subject with a pattern defined by the pattern mask 252. In the present embodiment, a subject is irradiated with a lattice pattern. In addition, since the infrared light pattern 510 is a pattern using infrared light, a person included as a subject is not allowed to recognize pattern light that makes the person feel uncomfortable in the visible light wavelength band. Further, the infrared image is acquired as shown in FIG. 5B, and the actual captured image is acquired as shown in FIG.

  It is difficult to acquire phase difference information from the AF sensor 240 for objects such as a blind 503 in which a plurality of horizontal lines continue in the vertical direction and a wall surface 502 having no contrast on the surface. However, if the grid pattern is actively irradiated as in the present embodiment, depth information can be acquired for most subjects by the processing described later. Further, although there is a considerable depth difference between the table 501 and the blind 503, the diffusion of the lattice pattern is kept within a certain range by irradiating the infrared light pattern 510 with an appropriate WCOE 254 interposed. That is, the lattice pattern can be observed over the depth direction of the scene. Here, “observed” means that the image can be acquired as an infrared image rather than being visually recognized by a human. In FIG. 5B, the line width of the lattice pattern is expressed as a constant for convenience, but actually the line width and contrast change according to the depth as described below.

  FIG. 6 is an explanatory diagram for explaining a difference in pattern irradiation in the depth direction. The infrared light pattern irradiated to the scene from the pattern irradiation unit 250 of the single-lens reflex camera 200 is focused on the first plane that is the focus plane. That is, the infrared light pattern has the maximum contrast in the first plane. If an object exists on the first plane, an infrared light pattern is observed on the surface with the maximum contrast.

  In the defocus plane deviated from the first plane in the depth direction, the contrast of the infrared light pattern gradually decreases as the distance from the first plane increases. Therefore, a thinner infrared light pattern is observed on the third plane farther from the first plane than on the second plane, compared to the infrared light pattern observed on the second plane.

  That is, as in the scene shown in FIG. 5, if there are a plurality of objects in the depth direction, infrared light patterns having different contrasts according to the depths are observed on the surfaces of the respective objects.

  The infrared light pattern has the smallest line width in the first plane that is the focus surface. The line width gradually increases as the distance from the first plane increases. That is, when there are a plurality of objects in the depth direction, infrared light patterns having different line widths according to the depths are observed on the surfaces of the respective objects.

  In the present embodiment, depth information is calculated by paying attention to a change in contrast as edge information of an infrared light pattern. FIG. 6A shows a state in which a part of the vertical line is cut out from the pattern observed in each plane. As schematically shown in FIG. 6A, the pattern observed on the first plane is clear. As it moves away from the second plane and the third plane, the pattern gradually becomes blurred. This can be expressed as a luminance value at A-A as shown in FIG. That is, the pattern in the first plane has a large luminance value and shows a steep rise at the boundary of the pattern. On the other hand, it can be seen that as the distance from the second plane and the third plane increases, the maximum luminance value gradually decreases, and the pattern boundary gradually decreases.

  When a differential value is calculated with respect to the luminance value at A-A in FIG. 6B, it is expressed as shown in FIG. In the case of a pattern observed on the first plane, a steep change at the boundary appears as a narrow and high convex shape. The differential value is defined as positive when going from the non-pattern part to the pattern part, and minus when going from the pattern part to the non-pattern part. Each shape appears.

  On the other hand, as the distance from the second plane and the third plane increases, the base of the convex shape gradually widens and the height is observed to be low. That is, the contrast of the infrared light pattern can be evaluated by the differential value. In other words, the depth information can be acquired by evaluating the differential value of the infrared light pattern. More simply, the depth information can be acquired from the maximum differential value that is the height of the convex shape. If a calculation is performed for each area of the two-dimensional plane corresponding to the infrared image using the maximum value of the differential value as an evaluation value, a blur map expressing the blur in the depth direction as a numerical value can be generated. A specific blur map generation process will be described below.

  FIG. 7 is a diagram for explaining the blur map generation process. First, an acquired image shown in FIG. 7A is acquired as an infrared image. Here, it is assumed that there are three rectangular objects in the depth direction. The infrared image is present in the farthest position, the first region corresponding to the infrared light pattern irradiated to the first object existing closest to the subject image, the second region corresponding to the second object existing in the middle. And a third region corresponding to the third object. The other area is an infinite background where the infrared light pattern does not reach. Here, it is assumed that the irradiated plane of the first object is the focus plane.

  A differential value is calculated in the x direction, y direction, and xy ± 45 degrees direction with respect to the infrared light pattern of the acquired infrared image, and the absolute values thereof are added. The absolute value is used in the same way even when the non-pattern part moves from the pattern part to the non-pattern part. The purpose of calculating the differential value from a plurality of directions is to extract changes in the pixel value more reliably by sampling so as to include the direction intersecting the edge of the infrared light pattern. . When this value is assigned to each pixel and visualized, a differential image shown in FIG. 7B is obtained.

  Next, as shown in FIG. 7C, an operation window having a side w around the pixel Pi of interest is extracted from the differential image. Then, for all the pixels included in this calculation window, the values are histogrammed. Then, as shown in FIG. 7 (d), a histogram of differential values is obtained according to the frequency of counting the number of pixels with respect to the differential values divided for each fixed range. Then, the differential value Vi indicating the mode value in this histogram is determined as the evaluation value for the pixel Pi. By such processing, a representative differential value V of a minute region including the peripheral portion, not the differential value itself of the pixel Pi, can be determined as the evaluation value of the pixel Pi. Then, this process is executed for all the pixels while shifting the calculation window vertically and horizontally. Then, in the region corresponding to the infrared light pattern showing the same degree of contrast, the evaluation value calculated for each pixel included in the region corresponds to almost the same value.

In the example of the figure, each pixel in the first region shows an evaluation value V ₁ that is substantially the same over the entire region, similarly, the second region has an evaluation value V ₂ , and the third region has an evaluation value V _3. Indicates. Then, the contrast of the infrared light pattern in the first region is high, and it is low in the third region. Therefore, the magnitude relationship between the evaluation values is V ₁ > V ₂ > V ₃ . It should be noted that the evaluation value of the other area that is an infinite background is 0.

  In this way, different evaluation values are determined for each region corresponding to different depth directions. The evaluation value is a value corresponding to the contrast of the infrared light pattern. In the first place, the evaluation value is a value resulting from a physical phenomenon in which the point image is gradually blurred as the distance from the focus surface increases in the depth direction. Therefore, the numerical value array generated by assigning the evaluation values for each region of the two-dimensional plane corresponding to the infrared image can be said to be a blur map.

  Since each region of the infrared image has a one-to-one correspondence with each region of the main captured image, it can be said that the blur map represents depth information of the subject appearing in the main captured image. Then, if the blur map is included as the depth information together with the actual captured image data, it can be applied to the following processing.

  First, the amount of blur according to the depth can be given later by image processing. In other words, if each region is provided with a blur amount linked to the evaluation value of the blur map, natural image data that has a small blur at a depth close to the focus surface and a large blur at a far depth and that is visually unnatural can be acquired. For example, in a compact digital camera using a small image sensor, the depth of field is inevitably deep, and generally only images close to pan focus can be obtained. However, good image data with a blur can be generated by image processing using the blur map.

  Further, by referring to the blur map, it is possible to separate the subject appearing in the actual captured image for each depth. Then, it is possible to track the movement of the specific subject in a plurality of continuously captured real image data, or to apply a special effect only to the specific subject.

  Furthermore, if an evaluation value at a known distance in the depth direction is acquired in advance by simulation or the like and a correspondence table between the evaluation value and the subject distance is prepared, the blur map can be converted into a depth map. The depth map is equivalent to a so-called distance image. In other words, if a blur map is generated together with the acquisition of the actual captured image data, the absolute distance to each subject appearing in the actual captured image can be acquired.

  Next, a series of shooting processes for generating a blur map and adjusting the blur amount of the main shot image with reference to the blur map will be described. FIG. 8 is a process flow diagram for explaining a photographing sequence. A series of processing in this flow is started when the single-lens reflex camera 200 is switched to the photographing mode.

  In step S101, the camera system control unit 245 waits for a shooting preparation instruction from the user. Specifically, it waits for release switch 265 to be operated by the user and SW1 to be turned on. When SW1 is turned on, the process proceeds to step S102, and the camera system control unit 245 drives the AF sensor 240 to execute AF processing. Specifically, the camera system control unit 245 acquires phase difference information in the focus detection area from the output of the AF sensor 240, and the lens system so that the defocus amount is 0 with respect to the main focus detection area. The focus lens 212 is moved via the control unit 216.

  As a result, the subject is specifically focused. That is, at this time, the first plane described with reference to FIG. 6 is determined. Further, since a plurality of focus detection areas are discretely set with respect to the object scene, defocus information can be acquired for each of a plurality of subjects having different positions in the depth direction. When blur maps are generated for these subjects, the evaluation values from the foreground subject to the farthest subject are calculated. Therefore, the imaging range L is calculated from the defocus amount for the foremost subject and the defocus amount for the farthest subject. The imaging range L is calculated in consideration of the lens information of the lens unit 210. Specifically, the focal length information of the lens unit 210 and the movement position information of the focus lens 212 are obtained from the lens system control unit 216 and calculated. Since the focus detection area is discrete, the distance information calculated from the defocus information is also discrete.

  When calculating the evaluation value over the imaging range L, the diffusion amount of the infrared light pattern within this range must be within a predetermined range. This is because if the irradiated infrared light pattern is too diffused, the calculated differential value becomes a small value and the error becomes large. Therefore, in step S103, the camera system control unit 245 drives the WCOE replacement unit 255 to insert the WCOE 254 optimal for the imaging range L into the irradiation light beam of the infrared light pattern.

  Specifically, when the position Fz of the first plane is calculated, the focal depth Sp is calculated from the focal length and F number of the pattern irradiation unit 250. Then, a magnification β obtained by dividing the imaging range L by the focal depth Sp is calculated. On the other hand, the plurality of WCOEs 254 are prepared as elements for multiplying the depth of focus by αi which is a different expansion magnification. Therefore, the camera system control unit 245 selects the WCOE 254 having the expansion magnification αi closest to the calculated magnification β as a characteristic. Then, the camera system control unit 245 inserts the WCOE 254 into the irradiation light beam of the infrared light pattern.

  The camera system control unit 245 proceeds to step S104 and waits for a shooting instruction from the user. Specifically, it waits for release switch 265 to be operated by the user and SW2 to be turned on. When SW2 is turned on, the process proceeds to step S105, and the camera system control unit 245 causes the infrared light emitting unit 251 to emit light and irradiates the subject with an infrared light pattern. In step S106, exposure control of the image sensor 243 is performed during irradiation of the infrared light pattern.

  When the exposure is completed in step S <b> 106, the camera system control unit 245 performs the processing as the actual captured image and the processing for generating the blur map in parallel or in order. In step S <b> 107, the camera system control unit 245 reads RGB pixels from the image sensor 243 and delivers the read RGB image signal to the image processing unit 246. In step S108, the image processing unit 246 generates actual captured image data from the received RGB image signal.

  In step S <b> 109, the camera system control unit 245 reads out the infrared light receiving pixels of the image sensor 243 and delivers the read infrared image signal to the image processing unit 246. The image processing unit 246 generates an infrared image from the received infrared image signal and passes it to the blur map calculation unit 261. The blur map calculation unit 261 generates a blur map by the procedure described with reference to FIG.

  When the main captured image data and the blur map are generated together, the process proceeds to step S111, and the image processing unit 246 applies blur filter processing by applying a different blur amount to each region of the main captured image data while referring to the blur map. . In step S <b> 112, the camera system control unit 245 displays the captured image data on which the blurring process has been performed on the display unit 247. At this time, reduction processing may be performed in advance according to the size of the display unit 247.

  The user visually recognizes the blurring-processed main captured image data displayed on the display unit 247 and confirms whether the image is finished as a favorite image. If the desired image is not finished, the operation member is operated to re-execute the blur process. If the camera system control unit 245 receives a blur re-execution instruction from the user in step S113, the camera system control unit 245 returns to step S111. If an OK instruction is received instead of a re-execution instruction, the process advances to step S114 to cause the image processing unit 246 to execute predetermined format processing or the like on the blurring-processed main image data. Then, the camera system control unit 245 records in the external memory via the external connection IF 264. Thus, a series of processing ends.

  Next, variations on the arrangement of the image sensor will be described. FIG. 9 is a cross-sectional view of an interchangeable lens camera 900 in which an infrared light imaging device 944 and a visible light imaging device 943 are arranged. As shown in the figure, a dedicated infrared light image sensor 944 that receives an infrared light wavelength band and a visible light image sensor 943 that receives a visible light wavelength band are arranged with an infrared light wavelength band and a visible light wavelength of a subject light flux. The dichroic mirrors 932 that divide the band are independently arranged at conjugate positions. The optical element that divides the infrared light wavelength band and the visible light wavelength band is not limited to the dichroic mirror 932 and may be an element such as a dichroic prism.

  Further, depending on the photographing optical system, the focal plane of the infrared light wavelength band and the focal plane of the visible light wavelength band may be shifted. In this case, the infrared light imaging element 944 is the focal point of the visible light imaging element. It is preferable to arrange at a correction position in which this deviation amount is added to a position conjugate with the surface. Alternatively, for example, a correction lens 933 may be disposed in the vicinity of the infrared light imaging device 944 so as to cancel out the shift amount. Alternatively, a day / night lens in which the focal plane of the infrared light wavelength band coincides with the focal plane of the visible light wavelength band can be used, including the above-described embodiment.

  In the present embodiment described above, the pattern irradiation unit 250 is provided as an irradiation unit that irradiates an infrared light pattern. However, when the camera includes a projector unit that projects a captured image or the like, the projector unit can be used as an irradiation unit. In this case, when an infrared light pattern is projected, the light emitting unit is switched to, for example, an infrared light emitting diode, and a pattern mask and a WCOE are inserted into the irradiation light beam. When an active light projection system is provided as AF auxiliary light, the light projection system may be used as an irradiation unit.

  Moreover, in the above-mentioned embodiment, the lattice-shaped infrared light pattern was irradiated. However, the pattern is not limited to the lattice shape, and may be another pattern. Further, although an example in which WCOE is used as an element for increasing the depth of focus of an infrared light pattern has been described, a method of adding a phase of a pupil plane using a spatial light modulator as a phase plate may be employed.

  In the above embodiment, the single-lens reflex camera 200 has been described as an example, but the imaging apparatus is not limited to a single-lens reflex camera. As described above, a large effect is exhibited even in a compact digital camera having a small image sensor and a portable terminal. The present invention can also be applied to a mirrorless camera that does not include an optical viewfinder, a video camera that mainly shoots moving images, and the like. For example, when applied to a video camera, a blur map corresponding to each frame can be continuously generated.

  In the above embodiment, the imaging apparatus provided with the imaging unit has been described as including an arithmetic device that calculates depth information. However, if an infrared image in which an infrared light pattern is reflected can be acquired, a blur map can be generated and depth information can be calculated by an external device such as a PC, not an imaging device. In this case, an external device such as a PC functions as an image calculation device that calculates depth information. Further, even with an external device, if the actual captured image is acquired together with the infrared image, the actual captured image can be blurred while referring to the generated blur map.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be added to the above-described embodiment. It is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

  The order of execution of each process such as operations, procedures, steps, and stages in the apparatus, system, program, and method shown in the claims, the description, and the drawings is particularly “before” or “prior to”. It should be noted that the output can be realized in any order unless the output of the previous process is used in the subsequent process. Regarding the operation flow in the claims, the description, and the drawings, even if it is described using “first”, “next”, etc. for convenience, it means that it is essential to carry out in this order. It is not a thing.

200 SLR camera, 201 optical axis, 210 lens unit, 211 lens group, 212 focus lens, 213 zoom lens, 216 lens system control unit, 217 lens barrel, 218 lens mount, 221 lens memory, 230 camera unit, 231 camera Mount, 232 main mirror, 233 rotation axis, 234 focus plate, 235 pentaprism, 236 eyepiece optical system, 237 AE sensor, 238 sub mirror, 239 AF optical system, 240 AF sensor, 241 focal plane shutter, 242 optical low pass filter, 243 Image sensor, 244 main board, 245 camera system control unit, 246 image processing unit, 247 display unit, 248 secondary battery, 249 flash, 250 pattern irradiation unit, 2 DESCRIPTION OF SYMBOLS 1 Infrared light emission part, 252 Pattern mask, 253 Irradiation lens, 254 WCOE, 255 WCOE replacement part, 261 Blur map calculation part, 262 Camera memory, 263 Work memory, 264 External connection IF, 265 Release switch, 301 B curve, 302 G curve, 303 R curve, 304 IR curve, 311 arrow, 501 table, 502 wall surface, 503 blind, 510 infrared light pattern, 900 lens interchangeable camera, 932 dichroic mirror, 933 correction lens, 943 imaging device for visible light, 944 Imaging device for infrared light

Claims (12)

An image sensor having an infrared light receiving pixel having sensitivity in an infrared light wavelength band;
An infrared irradiation unit for irradiating an infrared light pattern;
A acquisition unit that acquires an infrared image captured by the image pickup device of a subject which is irradiated with the infrared light pattern,
The infrared irradiation unit is an imaging apparatus including a wavefront coding optical element disposed in an irradiation light beam of the infrared light pattern. The imaging apparatus according to claim 1, further comprising: a calculation unit that calculates depth information of the subject based on edge information of the infrared light pattern reflected in the infrared image. The imaging apparatus according to claim 2 , wherein the calculation unit evaluates an edge blur amount as the edge information . The imaging apparatus according to claim 3 , wherein the calculation unit evaluates the blur amount based on a change amount of a pixel value with respect to a plurality of pixels sampled so as to include a direction intersecting the edge . The imaging device according to claim 3 or 4 , wherein the calculation unit calculates the depth information with reference to a correspondence table prepared in advance indicating a correspondence relationship between a subject distance and the blur amount . The infrared irradiation unit, a selected one of a plurality of the wavefront coding optical element according to any one of claims 1 to 5 having a insertion mechanism for inserting and removing into interchangeably the irradiation light beam Imaging device. An AF sensor;
The imaging apparatus according to claim 6 , further comprising: a selection unit that selects one of the plurality of wavefront coding optical elements based on an output of the AF sensor. The imaging apparatus according to claim 7 , wherein the selection unit selects one of the plurality of wavefront coding optical elements in consideration of photographing lens information. The infrared light receiving pixels in the image sensor, the visible light receiving pixels having a sensitivity in a visible light wavelength band, according to any one of claims 1 to 8, are arranged two-dimensionally and discretely Imaging device. 10. The imaging according to claim 9 , further comprising: an image generation unit configured to generate the infrared image and the visible image from each of the output of the infrared light receiving pixel and the output of the visible light receiving pixel by one exposure to the imaging element. apparatus. At least a projector unit for projecting a captured image,
The infrared irradiation unit, an imaging apparatus according to any one of 10 constructed according to claim 1 by the projector unit. An infrared irradiation step of irradiating an object with an infrared light pattern from an infrared irradiation unit having a wavefront coding optical element disposed in the irradiation light beam;
A control program of an imaging apparatus to execute the steps of acquiring an infrared image obtained by imaging the object which is irradiated with the infrared light pattern to a computer.