JP7043209B2 - Control device, image pickup device, control method, and program - Google Patents

Description

The present invention relates to an image pickup apparatus capable of detecting unnecessary light before shooting and reducing it at the time of shooting.

In recent years, image pickup devices such as digital still cameras and digital video cameras equipped with image pickup elements such as CCD image sensors and CMOS image sensors have become widespread. When an image is taken by an image pickup device, a part of the light incident on the image pickup optical system may be reflected by the interface of the lens or a member holding the lens, and may reach the image pickup surface as a light flux different from the image pickup light flux. Luminous flux different from the luminous flux that reaches the image pickup surface (hereinafter referred to as unnecessary light) forms a dense spot image or covers a wide area of the subject image, and is photographed as an unnecessary component called ghost or flare. Appears in the image.

Further, in a telephoto lens or the like, a diffractive optical element may be used for the lens closest to the object, for example, for correcting axial chromatic aberration and chromatic aberration of magnification. At this time, the light from a high-intensity luminance object such as the sun existing outside the imaged angle of view hits the diffractive optical element, so that a vague light flux different from the captured light flux may appear in the entire captured image. In addition, in fixed-point shooting such as time-lapse shooting and interval shooting, a light source or an object that reflects the light source appears in or near the angle of view, and unnecessary components are generated in the shot image, resulting in an unintended failure image of the photographer. Is assumed.

Conventionally, for ghosts and flares, a method of estimating the direction of a light source and optically reducing unnecessary light and a method of removing unnecessary components by digital image processing have been proposed. Patent Document 1 discloses a method of acquiring shooting condition information regarding a focal length and an aperture value of an imaging optical system and determining unnecessary image components based on the shooting condition information. Patent Document 2 discloses a method of accurately detecting a ghost image by acquiring a plurality of viewpoint images, calculating and correcting the amount of image shift due to the parallax of the subject images between the viewpoint images.

Japanese Unexamined Patent Publication No. 2013-179564 Japanese Unexamined Patent Publication No. 2011-205531

According to the methods disclosed in Patent Document 1 and Patent Document 2, unnecessary components are used by using a plurality of viewpoint images generated by an image sensor that photoelectrically converts light flux passing through different regions of a single photographing optical system. Can be removed. However, in the method disclosed in Patent Document 1, in order to acquire shooting condition information regarding the focal length and aperture value of the imaging optical system after shooting and determine whether or not to remove unnecessary components, unnecessary components are removed before shooting. It is not possible to determine if it is possible to remove it. In the method disclosed in Patent Document 2, since the image shift correction due to the parallax between a plurality of viewpoint images is performed after shooting to remove unnecessary components, is it possible to similarly remove unnecessary components before shooting? It cannot be determined whether or not.

Therefore, the present invention presents a control device, an image pickup device, a control method, and a program capable of acquiring an image with reduced unnecessary components by performing a shooting process based on the detection result of unnecessary components detected before shooting. The purpose is to provide.

The control device as one aspect of the present invention is based on the detection means for detecting the unnecessary component of the first image acquired at the first timing and the detection result of the unnecessary component by the detection means. It has a control means for performing an imaging process for acquiring a second image at a second timing after the timing, and the control means said the second when the detection means detects the unnecessary component. When the detection means does not detect the unnecessary component, a composite image of the images of the plurality of different viewpoints is acquired as the second image .

The image pickup device as another aspect of the present invention includes an image pickup element that photoelectrically converts an optical image formed via an image pickup optical system, and the control device.

The control method as another aspect of the present invention is after the first timing based on the step of detecting the unnecessary component of the first image acquired at the first timing and the detection result of the unnecessary component. When the unnecessary component is detected in the detection step in the step of performing the photographing process, which has a step of performing a photographing process for acquiring the second image at the second timing of the above-mentioned second. When a plurality of images of different viewpoints are acquired as images and the unnecessary component is not detected in the detection step, a composite image of the images of the plurality of different viewpoints is acquired as the second image .

The program as another aspect of the present invention causes a computer to execute the control method.

Other objects and features of the present invention will be described in the following examples.

According to the present invention, a control device, an image pickup device, a control method, and a control device capable of acquiring an image in which unnecessary components are reduced by performing a shooting process based on a detection result of unnecessary components detected before shooting, and A program can be provided.

It is explanatory drawing of the method of removing an unnecessary component in each Example. It is a schematic diagram which shows an example of the output image of the unnecessary component removal in each Example. It is a schematic diagram which shows the relationship between the light receiving part of the image pickup element and the pupil of an image pickup optical system in each Example. It is a block diagram of the image pickup apparatus in each Example. It is explanatory drawing of the structure of an image pickup optical system and unnecessary light in each Example. It is explanatory drawing which shows the operation of the image pickup apparatus in Example 1. FIG. It is a flowchart which shows the operation of the image pickup apparatus in Example 1. FIG. It is explanatory drawing which shows the operation of the image pickup apparatus in Example 2. FIG. It is a flowchart which shows the operation of the image pickup apparatus in Example 2. FIG. It is explanatory drawing which shows the operation of the image pickup apparatus in Example 3. FIG. It is a flowchart which shows the operation of the image pickup apparatus in Example 3. FIG. It is explanatory drawing which shows the operation of the image pickup apparatus in Example 4. FIG. It is a flowchart which shows the operation of the image pickup apparatus in Example 4. FIG.

Hereinafter, examples of the present invention will be described in detail with reference to the drawings.

The image pickup device capable of generating an image of a plurality of viewpoints used in each embodiment of the present invention transmits a plurality of light fluxes passing through different regions of the pupils of the image pickup optical system to different light receiving portions (pixels) of the image pickup element. It has an image pickup system that guides and performs photoelectric conversion.

First, with reference to FIG. 3, the relationship between the light receiving unit (photoelectric conversion unit) of the image sensor in the image pickup system and the pupil of the image pickup optical system will be described. FIG. 3A is an explanatory diagram showing the relationship between the light receiving portion of the image pickup device and the pupil of the image pickup optical system. In FIG. 3A, ML is a microlens and CF is a color filter. EXP is an exit pupil of an imaging optical system. G1 and G2 are light receiving units (G1 pixels, G2 pixels) as photoelectric conversion units, and one G1 pixel and one G2 pixel form a pair with each other. A plurality of pairs (pixel pairs or pixel sets) of G1 pixels and G2 pixels are arranged (arranged in two dimensions) in the image pickup device. The pair of G1 pixels and G2 pixels have a conjugate relationship with the exit pupil EXP via a common microlens ML (provided one for each pixel set). A plurality of G1 pixels arranged in the image sensor are collectively referred to as a G1 pixel group, and a plurality of G2 pixels arranged in the image sensor are collectively referred to as a G2 pixel group.

FIG. 3B schematically shows an imaging system assuming that there is a thin-walled lens at the position of the exit pupil EXP instead of the microlens ML shown in FIG. 3A. The G1 pixel receives the light flux that has passed through the region P1 of the exit pupil EXP, and the G2 pixel receives the light flux that has passed through the region P2 of the exit pupil EXP. The OSP is an imaged object point. It should be noted that the object does not need to exist in the object point OSP. The luminous flux that has passed through the object point OSP is incident on the G1 pixel or the G2 pixel depending on the region (position) in the pupil through which the light flux passes. The passage of the luminous flux through different regions in the pupil corresponds to the separation of the incident light from the object point OSP by the angle (parallax). That is, the image signal generated by using the output signal from the G1 pixel among the G1 and G2 pixels provided for each microlens ML and the image signal generated by using the output signal from the G2 pixel are different. It is a plurality of (here, a pair) images (viewpoint images) having a plurality of viewpoints. In the following description, receiving light flux passing through different regions in the pupil by different light receiving portions (pixels) is referred to as pupil division.

In FIGS. 3A and 3B, even when the above-mentioned conjugate relationship is not perfect due to the position of the exit pupil EXP being displaced, or when the region P1 and the region P2 partially overlap each other. In this embodiment, the obtained plurality of images are treated as viewpoint images.

Next, the configuration of the image pickup apparatus according to the present embodiment will be described with reference to FIG. FIG. 4 is a block diagram of the image pickup apparatus 200. The image pickup optical system 201 including the aperture 201a and the focus lens 201b forms an image of light from a subject (not shown) on the image pickup element 202. The image pickup device 202, which includes a photoelectric conversion element such as a CCD sensor or a CMOS sensor, transmits light flux passing through different regions in the pupil as described with reference to FIGS. 3 (a) and 3 (b). , The pupil is divided by receiving light at the pixel (light receiving unit) corresponding to each area.

The image pickup element 202 photoelectrically converts an optical image formed via the image pickup optical system 201 and outputs an analog signal (image signal). The A / D converter 203 converts the analog signal output from the image pickup device 202 into a digital signal. The digital signal output from the A / D converter 203 is input to the image processing unit 204. The image processing unit 204 generates an image (in this embodiment, an image having a plurality of different viewpoints) by performing image processing generally performed on a digital signal. The image processing unit 204 corresponds to an image processing device mounted on the image pickup device 200. The image processing unit 204 functions as an image acquisition unit that acquires images of a plurality of viewpoints (viewpoint images). Further, the image processing unit 204 includes an unnecessary component detecting unit (detecting means) 204a, an unnecessary component reducing unit (reducing means) 204b, and a light source direction estimation unit (light source direction calculating means) 211. The unnecessary component detection unit 204a detects an unnecessary image component (unnecessary component) generated by unnecessary light described later in each viewpoint image (determines whether or not there is an unnecessary component). The unnecessary component reduction unit 204b performs correction processing for reducing unnecessary components from each viewpoint image. The light source direction estimation unit 211 calculates the direction of the light source based on the information (shooting condition information) regarding the imaging optical system 201.

The image processed and output by the image processing unit 204 is temporarily held in the storage unit 208 and stored in an image recording medium 209 such as a semiconductor memory or an optical disk. The storage unit 208 also has a function of a work memory when the image processing unit 204 operates. The display unit 205 displays the image output from the image processing unit 204.

The system controller 210 controls the operation of the image sensor 202, the processing by the image processing unit 204, and the operation of the image pickup optical system 201 (aperture 201a and focus lens 201b), respectively. The image pickup optical system control unit 206 mechanically drives the aperture 201a and the focus lens 201b in the image pickup optical system 201 in response to a control instruction from the system controller 210. The aperture diameter of the aperture 201a is controlled according to the set aperture value (F number). Since the focus lens 201b adjusts the focus according to the subject distance, its position is controlled by an autofocus (AF) system (not shown) or a manual focus mechanism. The state detection unit 207 responds to an instruction from the system controller 210 to indicate the state of the imaging optical system at that time (current) (position of focus lens 201b, aperture value, focal length when zooming is possible, etc.). The image pickup condition information, which is the information indicating the above, is acquired. Although the image pickup optical system 201 is configured as a part of the image pickup apparatus 200 in FIG. 4, it may be an image pickup optical system having an interchangeable lens like a single-lens camera. The light source direction estimation unit 211 estimates the direction of the light source. Various methods have been conventionally proposed for estimating the direction of the light source using images from a plurality of viewpoints, but in this embodiment, any method may be used.

FIG. 5 is an explanatory diagram of the configuration of the imaging optical system 201 and unnecessary light. FIG. 5A shows a specific configuration example of the imaging optical system 201. STP is an aperture. The IMG is an image pickup surface, and the image pickup device 202 shown in FIG. 4 is arranged at the position of the image pickup surface IMG. In FIG. 5B, strong light from the sun SUN as an example of a high-intensity object is incident on the image pickup optical system 201, and the light reflected at the interface of the lens constituting the image pickup optical system 201 is unnecessary light (ghost or flare). ) Shows how it reaches the imaging surface IMG. FIG. 5C shows regions (pupil regions) P1 and P2 through which the light flux incident on the G1 pixel and the G2 pixel shown in FIG. 3 passes in the aperture STP (or the exit pupil of the imaging optical system 201). There is. The luminous flux from the high-luminance object passes through almost the entire area of the diaphragm STP, but the region through which the luminous flux incident on the G1 pixel and the G2 pixel passes is divided into pupil regions P1 and P2.

In a photographed image generated by such an image pickup apparatus, one method of detecting an unnecessary component appearing by photoelectric conversion of unnecessary light will be described with reference to FIGS. 1 and 2. FIG. 1 is an explanatory diagram of a method for removing unnecessary components. FIG. 2 is a schematic diagram showing an example of an output image (photographed image).

FIG. 2A shows a photographed image generated by a photograph in which pupil division is performed. In this photographed image, a building such as a building and trees existing around it are shown as subjects. Further, GST shown as a square portion in the captured image is an unnecessary component which is an image component of unnecessary light (ghost, flare, etc.). There is a light source on the outside of the shooting screen, and in this embodiment, the light source is arranged on an extension of the line connecting unnecessary light. There are various combinations of the relationship between the light source and the unnecessary light, and the relationship is not limited to the relationship described in this embodiment.

In FIG. 1, the unnecessary component GST is shown by thinly filling the inside of the circle with a black circle, but in reality, the subject may be transparent to some extent or crushed to pure white depending on the appearance intensity of the unnecessary light. Further, since the unnecessary component is a state in which the captured subject image is covered with unnecessary light, the unnecessary component is a portion having higher brightness than the subject image. This also applies to other examples described later.

1 (A-1) and 1 (B-1) are a pair of viewpoint images (a pair) obtained as a result of photoelectric conversion of the light flux passing through the pupil regions P1 and P2 by the G1 and G2 pixel groups, respectively. The viewpoint image) is shown. There is a clear difference (parallax component of the subject) corresponding to the viewpoint in the image components in the pair of viewpoint images obtained by imaging a short-distance subject. However, the parallax component in the pair of viewpoint images obtained by imaging a long-distance subject such as a landscape as shown in FIGS. 1 (A-1) and 1 (B-1) is very small. Further, the pair of viewpoint images also contain an unnecessary component GST schematically shown as a black circle, but the position thereof differs between the viewpoint images. Here, an example of a state in which unnecessary components GST are separated without overlapping with each other is shown, but a state in which they overlap and have a luminance difference may be used. That is, it suffices if the positions and brightnesses of unnecessary components of the black circles are different from each other.

FIG. 1 (A-2) uses the image of FIG. 1 (A-1) as a reference image among the pair of viewpoint images shown in FIGS. 1 (A-1) and 1 (B-1) as a reference image. An image obtained by subtracting the image (subtracted image) of FIG. 1 (B-1) from FIG. 1 is shown. The image showing the difference between the reference image and the subtracted image (relative difference image) includes the above-mentioned parallax component and unnecessary component as two-dimensional data (relative difference information) of the difference possessed by the pair of viewpoint images. .. However, as described above, since the parallax component between the pair of viewpoint images obtained by imaging a long-distance subject such as a landscape is very small, its influence can be almost ignored. Further, by performing the difference calculation by subtracting the subtracted image of FIG. 1 (B-1) from the reference image of FIG. 1 (A-1), the unnecessary component contained in FIG. 1 (B-1) is calculated as a negative value. To. However, in this embodiment, in order to simplify the unnecessary component reduction process performed in the subsequent stage, the negative value is rounded down in FIG. 1 (A-2) (that is, the negative value is treated as 0 value). Therefore, the relative difference image of FIG. 1 (A-2) shows only unnecessary components included in the reference image of FIG. 1 (A-1).

Similarly, FIG. 1 (B-2) is a two-dimensional image obtained by using the image shown in FIG. 1 (B-1) as a reference image and subtracting the subtraction image of FIG. 1 (A-1) from the reference image. An image showing relative difference information which is data is shown. Similar to the image of FIG. 1 (A-2), the unnecessary component included in FIG. 1 (A-1) is calculated as a negative value by the above-mentioned difference calculation, but in order to simplify the unnecessary component reduction process, In FIG. 1 (B-2), the negative value is rounded down. Therefore, the relative difference image of FIG. 1 (B-2) shows only unnecessary components contained in FIG. 1 (B-1). In this way, only unnecessary components remain in the relative difference image obtained by subtracting the other image (subtracted image) from one of the pair of viewpoint images (reference image) (in other words, separation or extraction). ) By performing the treatment, unnecessary components can be detected.

Then, the correction process for removing (reducing) the unnecessary components of FIGS. 1 (A-2) and 1 (B-2) detected as described above is performed in FIGS. 1 (A-1) and 1 (B-). This is performed on the original viewpoint image shown in 1). As a result, as shown in FIGS. 1 (A-3) and 1 (B-3), it is possible to obtain a viewpoint image as a pair of output images in which unnecessary components are reduced (preferably removed). .. Further, by synthesizing a pair of viewpoint images in which these unnecessary components are reduced, as shown in FIG. 2B, the images are equivalent to the captured image generated by the imaging without pupil division, and the unnecessary components are not included. It is possible to generate an image that contains almost no.

Next, the operation of the image pickup apparatus (reduction processing of unnecessary components) in the present embodiment will be described with reference to FIGS. 6 and 7. FIG. 6 is an explanatory diagram showing the operation of the image pickup apparatus 200. FIG. 7 is a flowchart showing the operation of the image pickup apparatus 200. Each step of FIG. 7 is executed by the system controller 210 and the image processing unit 204, which are computers, according to the image processing program as a computer program.

In FIG. 6, the second timing t2 is the time when the system controller 210 instructs the still image shooting, and shows the time calculated based on the shooting interval of the interval shooting preset by the user. The first timing t1 is the time for the system controller 210 to instruct the preparation for shooting a still image, and the shooting period and the detection period p required for detecting unnecessary components from the second timing t2 are set from the second timing t2. Shows the time calculated by subtracting. At the first timing t1, the system controller 210 starts video recording for detecting an unnecessary component. The moving images 1 to 5 indicate the shooting of frames, and the shooting of a plurality of viewpoint images is performed in sequence. Detection 1 indicates a process of detecting an unnecessary component using a plurality of viewpoint images taken by the moving image 1. Detection 2 to detection 5 correspond to moving images 2 to 5, respectively.

At the second timing t2, the system controller 210 starts the process of determination 1 for determining the presence / absence of an unnecessary component (whether or not an unnecessary component is detected) in detection 5, and takes a still image according to the determination result. .. FIG. 6A shows an example of still image shooting without pupil division as a result of determination of unnecessary components. FIG. 6B shows an example in which, as a result of the determination, pupil division imaging is performed and unnecessary component reduction processing is performed from a plurality of viewpoint images. In this embodiment, the moving image shooting is started at the first timing t1, but the metering and the focus detection process may be started at the same time with the first timing t1 as a signal. Further, here, in order to perform high-speed readout, moving image shooting is performed in which additive reading or thinning reading is performed from the image sensor 202, but if the reading time and detection time are sufficiently high, the reading is not limited to operation shooting, and all pixels are read out. Still image shooting may be performed.

First, in step S301 of FIG. 7, the system controller 210 reads the second timing t2 preset by the user from the storage unit 208, calculates and stores the first timing t1 for starting the first shooting. Set to unit 208. Assuming that the shooting time of the moving image 1 is f1 and the detection period of unnecessary components is p, the first timing t1 is obtained by the following equation (1).

t1 = t2- (f1 + p) ... (1)
Subsequently, in step S302, the system controller 210 compares the first timing t1 held in the storage unit 208 with the current time determined by the system controller 210, and whether or not the current time is the first timing. Is determined. If the current time is the first timing t1, the process proceeds to step S303. On the other hand, if the current time is not the first timing t1, step S302 is repeated.

In step S303, the system controller 210 starts the first shooting. At this time, the system controller 210 starts moving image shooting (moving image 1 to moving image 5) to acquire a plurality of viewpoint images, and sequentially displays a composite image of the plurality of viewpoint images as a through image on the display unit 205. Although the example of displaying the through image is used here, the through image may not be displayed, such as a shooting method of taking a picture by looking through the finder.

Subsequently, in step S304, the system controller 210 starts detecting unnecessary components (unnecessary light) from a plurality of viewpoint images acquired by moving image shooting, and holds the detection result in the storage unit 208. The system controller 210 performs detection 1, which is an unnecessary component detection process of the moving image 1, in parallel with shooting of the moving image 2. By detecting unnecessary components using a plurality of viewpoint images continuously acquired by the shooting method premised on high-speed readout in this way, unnecessary components are added to the image obtained in the still image shooting at the second timing t2. It can be determined whether or not it appears.

Subsequently, in step S305, the system controller 210 compares the second timing t2 held in the storage unit 208 with the current time determined by the system controller 210, and whether or not the current time is the second timing. Is determined. If the current time is the second timing t2, the process proceeds to step S306. On the other hand, if the current time is not the second timing t2, step S305 is repeated.

In step S306, the system controller 210 reads out the unnecessary component detection result obtained in step S304 from the storage unit 208. Then, when the system controller 210 determines that there is an unnecessary component, the system controller 210 proceeds to step S307. On the other hand, if it is determined that there is no unnecessary component, the process proceeds to step S311.

In step S307, the system controller 210 acquires the pupil division information which is the configuration information of the image pickup device 202. The pupil division information is the information of the pixel pair shown in FIG. 3, for example, in the image sensor having the pair of G1 pixel and G2 pixel in FIG. 3, it is acquired as the information divided into two in the horizontal direction or the vertical direction. .. Further, in an image pickup element having two pairs of horizontal and vertical pixels such as G11 pixel, G12 pixel, G13 pixel, and G14 pixel, the information is acquired as information divided into two in each of the horizontal direction and the vertical direction.

Subsequently, in step S308, the system controller 210 acquires information (optical information, shooting condition information) of the image pickup optical system 201 from the state detection unit 207. Here, the information of the imaging optical system 201 includes, for example, a lens ID peculiar to the lens, a position of the focus lens 201b, an aperture value (F value), a focal length when zooming is possible, and the like, but is limited thereto. It is not something that is done.

Subsequently, in step S309, the system controller 210 estimates the light source direction based on the information of the image pickup optical system 201 acquired in step S308 by using the light source direction estimation unit 211, and the light source generated on the image pickup surface is generated. Determine if it is an unnecessary component. Further, the system controller 210 determines the effect of reducing unnecessary components based on the pupil division information acquired in step S307. For example, when the pupil division information is divided into two horizontally and the direction of the light source is located in the center in the horizontal direction on the angle of view, it is determined that the reduction effect is low because parallax of unnecessary components does not occur in a plurality of viewpoint images. .. When the pupil division information is divided into two in each of the horizontal and vertical directions, the unnecessary component can be reduced by the parallax in the vertical direction, so that the reduction effect is judged to be high. Subsequently, in step S310, if the system controller 210 determines that the effect of reducing unnecessary components is high, the system controller 210 proceeds to step S311. On the other hand, if the system controller 210 determines that the effect of reducing unnecessary components is low, the system controller 210 proceeds to step S312.

In step S311, the system controller 210 takes a still image without performing the pupil division processing as the second shooting, and ends this flow. In step S312, the system controller 210 performs still image photographing in which pupil division processing is performed as a third imaging, reads out a plurality of viewpoint images from the image sensor 202, calculates the difference between the viewpoint images, and obtains unnecessary components. To detect. Then, the system controller 210 creates an unnecessary component reduction image by subtracting unnecessary components from a plurality of viewpoint images and then synthesizing them, and ends this flow.

As described above, in this embodiment, the control device has a detection means (unnecessary component detection unit 204a) and a control means (system controller 210). The detection means detects unnecessary components of the first image acquired at the first timing. The control means performs a shooting process for acquiring a second image at a second timing after the first timing based on the detection result of the unnecessary component by the detection means (acquiring the second image). Change the process for).

Preferably, the control means acquires a plurality of images from different viewpoints as a second image when the detection means detects an unnecessary component (S312). On the other hand, when the detection means does not detect an unnecessary component, a composite image of a plurality of images from different viewpoints is acquired as a second image (S311). Also preferably, the detection means detects unnecessary components contained in a plurality of images from different viewpoints as a first image. Further, preferably, the control means calculates the first timing based on the second timing, the shooting time of the first image, and the detection period of the unnecessary component (Equation (1)). Further, preferably, the control means determines the effect of reducing unnecessary components based on the pupil division information regarding the image pickup element 202 and the photographing condition information regarding the image pickup optical system 201 (S309, S310). Then, when the control means determines that there is an effect of reducing unnecessary components, it acquires images of a plurality of different viewpoints as a second image (S312). On the other hand, when the control means determines that there is no effect of reducing unnecessary components, the control means acquires a composite image of a plurality of images from different viewpoints as a second image (S311).

According to this embodiment, by detecting unnecessary components of a plurality of viewpoint images in advance before shooting a still image, it is not necessary to shoot a plurality of unnecessary viewpoint images in the still image shooting, so that it is unnecessary only when necessary. Component reduction processing can be performed.

Next, the operation of the image pickup apparatus (reduction processing of unnecessary components) in the second embodiment of the present invention will be described with reference to FIGS. 8 and 9. FIG. 8 is an explanatory diagram showing the operation of the image pickup apparatus 200. FIG. 9 is a flowchart showing the operation of the image pickup apparatus 200. Each step of FIG. 9 is executed by the system controller 210 and the image processing unit 204, which are computers, according to the image processing program as a computer program. The configuration of the image pickup apparatus in this embodiment is the same as that of the image pickup apparatus 200 described in the first embodiment with reference to FIG.

In FIG. 8, the first timing t1 and the second timing t2 are the same as in the first embodiment, respectively. At the first timing t1, the system controller 210 starts video recording for detecting an unnecessary component. The moving images 1 to 8 show the shooting of each frame of the moving image, and the shooting of a plurality of viewpoint images is sequentially performed in each frame. Detection 1 indicates a process of detecting an unnecessary component using a plurality of viewpoint images taken by the moving image 1. Detection 2 to detection 8 correspond to moving images 2 to 8, respectively. At the second timing t2, the system controller 210 starts the process of determination 1 for determining the presence / absence of detection of unnecessary components in detection 5. Judgment 2 to Judgment 4 correspond to detections 6 to 8, respectively. In FIG. 8A, when the unnecessary component is not detected as a result of the determination 4, still image shooting without pupil division is performed. The third timing t3 shown in FIG. 8B is an allowable time (predetermined period) for a shooting delay from the second timing t2 preset by the user or the image pickup apparatus 200. When the third timing t3 is reached before the unnecessary component is no longer detected, the system controller 210 takes a still image with the pupil divided and performs unnecessary component reduction processing from a plurality of viewpoint images.

First, in step S401 of FIG. 9, the system controller 210 reads the third timing t3 preset by the user from the storage unit 208 and sets it. By setting the third timing t3, as a result of the unnecessary light detection, it is possible to prevent the unnecessary light from being not extinguished and the shooting timing being significantly delayed and the shooting opportunity being lost. Since the following steps S402 to S406 are the same processes as steps S302 to S306 of the first embodiment, their description will be omitted.

Subsequently, in step S407, the system controller 210 compares the third timing t3 (held in the storage unit 208) set in step S401 with the current time determined by the system controller 210. Then, the system controller 210 determines whether or not the current time is the third timing t3. If the current time is the third timing t3, the process proceeds to step S409. On the other hand, if the current time is not the third timing t3, the process returns to step S406.

In step S408, the system controller 210 takes a still image without performing the pupil division processing as the second shooting, and ends this flow. In step S409, the system controller 210 performs still image shooting that performs pupil division processing as a third shooting. Then, the unnecessary component detection unit 204a of the image processing unit 204 reads out a plurality of viewpoint images from the image sensor 202, calculates the difference between the viewpoint images, and detects the unnecessary components. Then, the unnecessary component reduction unit 204b creates an unnecessary component reduction image by subtracting unnecessary components from a plurality of viewpoint images and synthesizing them, and ends this flow.

As described above, in the present embodiment, when the control means detects that the unnecessary component has disappeared in the first image that the detection means has started to acquire at the first timing, the control means has a plurality of different viewpoints as the second image. Acquires a composite image of the image of (S406). Preferably, the control means acquires a plurality of images from different viewpoints as a second image (S407, S409) when a predetermined allowable period has elapsed (at the third timing).

According to this embodiment, unnecessary components of a plurality of viewpoint images are detected in advance before shooting a still image, and the timing of still image shooting is shifted until the unnecessary components are no longer detected, thereby acquiring a still image with reduced unnecessary components. can do. Further, by providing an allowable time for delay in still image shooting, it is possible to acquire a still image with reduced unnecessary components while preventing loss of shooting opportunity.

Next, the operation of the image pickup apparatus (reduction processing of unnecessary components) in the third embodiment of the present invention will be described with reference to FIGS. 10 and 11. FIG. 10 is an explanatory diagram showing the operation of the image pickup apparatus 200. FIG. 11 is a flowchart showing the operation of the image pickup apparatus 200. Each step of FIG. 11 is executed by the system controller 210 and the image processing unit 204, which are computers, according to the image processing program as a computer program. The configuration of the image pickup apparatus in this embodiment is the same as that of the image pickup apparatus 200 described in the first embodiment with reference to FIG.

In FIG. 10, the first timing t1 and the second timing t2 are the same as in the first embodiment, respectively. The correlation calculation 1 calculates a new unnecessary component detection result using the result of the detection 1. The correlation calculation 2 calculates a new unnecessary component detection result based on the result of the correlation calculation 1 and the result of the detection 2. Similarly, the correlation calculation 3 to the correlation calculation 4 are obtained. The first timing tA and the second timing tB indicate the time for starting the preparation for still image shooting and the time for shooting the still image in the shooting of the next frame. Similarly, the moving image A to the moving image E, the detection A to the detection E, the correlation calculation A to the correlation calculation D, and the determination A indicate the respective processes in the shooting of the next frame. For example, the correlation calculation has a time constant of the unnecessary component detection result, so that the variation of the detection result can be moderated. Further, by calculating the cycle of unnecessary component detection using the continuous detection result and the correlation calculation result of the frames before and after, it becomes possible to acquire the timing avoiding the unnecessary light that may appear periodically. The result of the correlation calculation determines the time of the second timings t2 and tB, and is stored in the storage unit 208. As the second timings t2 and tB, a time during which unnecessary components are not detected is specified, so that still image shooting without pupil division is performed.

First, in step S501 of FIG. 11, the system controller 210 reads the preset second timing t2 from the storage unit 208, calculates the first timing t1 to start the first shooting, and stores the storage unit 208. Set to. Since the following steps S502 to S504 are the same processes as steps S302 to S304 of the first embodiment, their description will be omitted.

Subsequently, in step S505, the system controller 210 starts the correlation calculation based on the unnecessary component detection result in step S504. Subsequently, in step S506, the system controller 210 compares the second timing t2 held in the storage unit 208 with the current time determined by the system controller 210. Then, the system controller 210 determines whether or not the current time is the second timing t2. If the current time is the second timing t2, the process proceeds to step S507. On the other hand, if the current time is not the second timing t2, the process of step S506 is repeated.

In step S507, the system controller 210 predicts the second timing tB of the next frame, which is the timing at which the unnecessary component is not detected, based on the result of the correlation calculation in step S505. Then, the system controller 210 overwrites the second timing held in the storage unit 208 (changes the second timing t2 to the second timing tB). Subsequently, in step S508, the system controller 210 takes a still image without performing the pupil division processing as the second shooting. Subsequently, in step S509, the system controller 210 determines whether or not the shooting is completed. When the shooting is finished, this flow is finished. On the other hand, if the shooting is not completed, the process returns to step S501.

As described above, in the present embodiment, the control means performs the correlation calculation of the unnecessary components of the first image, acquires the cycle of the unnecessary components based on the result of the correlation calculation, and obtains the cycle of the unnecessary components, and the second is based on the cycle of the unnecessary components. The timing is set (S505 to S507).

According to this embodiment, unnecessary components of a plurality of viewpoint images are detected in advance before shooting a still image, and a correlation calculation is performed using the detection results of each unnecessary component. As a result, it is possible to calculate the generation cycle of unnecessary components, shift the timing of still image shooting, and acquire a still image with reduced unnecessary components.

Next, the operation of the image pickup apparatus (reduction processing of unnecessary components) in the fourth embodiment of the present invention will be described with reference to FIGS. 12 and 13. FIG. 12 is an explanatory diagram showing the operation of the image pickup apparatus 200. FIG. 13 is a flowchart showing the operation of the image pickup apparatus 200. Each step of FIG. 13 is executed by the system controller 210 and the image processing unit 204, which are computers, according to the image processing program as a computer program. The configuration of the image pickup apparatus in this embodiment is the same as that of the image pickup apparatus 200 described in the first embodiment with reference to FIG.

In FIG. 12, the second timing t2 is the same as that of the first embodiment. The second timing tB is the time for still image shooting in the shooting of the next frame. The second timing t (2-1) indicates the timing of re-shooting of the shooting at the second timing t2. In this embodiment, after the unnecessary light reduction processing, the system controller 210 determines whether or not the unnecessary light can be reduced by the unnecessary light reduction processing, and whether or not there is sufficient time until the next frame is shot. Make a judgment. For example, when the shooting interval is set in advance by the user, the system controller 210 calculates the time from the unnecessary light reduction processing to the next frame, and the shooting time of the pupil-divided still image and the processing time for reducing the unnecessary light. It is determined whether or not the time obtained by adding and is in time for this time. Then, the system controller 210 holds the determination result in the storage unit 208.

First, in step S601 of FIG. 13, the system controller 210 compares the second timing t2 held in the storage unit 208 with the current time determined by the system controller 210. Then, the system controller 210 determines whether or not the current time is the second timing t2. If the current time is the second timing t2, the process proceeds to step S602. On the other hand, when the current time is not the second timing t2, the process of step S601 is repeated.

In step S602, the system controller 210 performs still image shooting that performs pupil division processing as a third shooting. Then, the unnecessary component detection unit 204a of the image processing unit 204 reads out a plurality of viewpoint images from the image sensor 202, calculates the difference between the viewpoint images, and detects the unnecessary components. The unnecessary component reduction unit 204b creates an unnecessary component reduction image by subtracting unnecessary components from a plurality of viewpoint images and then synthesizing them.

Subsequently, in step S603, the system controller 210 determines whether or not the shooting is completed. When the shooting is finished, this flow is finished. On the other hand, if the shooting is not completed, the process proceeds to step S604. In step S604, the system controller 210 makes an unnecessary component reduction determination based on the unnecessary component reduction image acquired in step S602 (determines whether or not the unnecessary component has been reduced). Then, the system controller 210 holds the determination result in the storage unit 208.

Subsequently, in step S605, the system controller 210 reads out the determination result of step S604 from the storage unit 208. Then, if the system controller 210 determines that the unnecessary components have not been reduced, the system controller 210 proceeds to step S606. On the other hand, if it is determined that the unnecessary components are reduced, the process returns to step S601.

In step S606, the system controller 210 determines whether or not re-shooting is possible by the shooting time of the next frame (whether or not there is a margin by the time of the next frame). Then, if the system controller 210 determines that there is time to spare, the system controller 210 proceeds to step S607. On the other hand, if it is determined that there is no time to spare, the process returns to step S601.

In step S607, the system controller 210 calculates the second timing t (2-1) for reshooting, and holds the second timing t (2-1) in the storage unit 208. The second timing t (2-1) is set to a time close to the second timing t2 and a time during which it is possible to acquire a still image in which unnecessary light is reduced by reshooting.

Subsequently, in step S608, the system controller 210 compares the second timing t (2-1) held in the storage unit 208 with the current time in the system controller 210, and the current time is the second timing t. It is determined whether or not it is (2-1). If the current time is the second timing t (2-1), the process returns to step S602. On the other hand, when the current time is not the second timing t (2-1), the process of step S608 is repeated.

As described above, in the present embodiment, the control means acquires a plurality of images from different viewpoints as the second image. Then, when the detecting means detects an unnecessary component of the second image and there is a margin time longer than a predetermined time, the control means resets the second timing and acquires the second image (the second image is acquired). S605 to S608).

According to this embodiment, it is determined whether or not the unnecessary components are reduced from the unnecessary component reduction images created from the multiple viewpoint image shooting at the time of still image shooting, and if the reduction effect is low, reshooting is performed. As a result, it is possible to acquire a still image in which unnecessary components are reduced or removed.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

According to each embodiment, a control device, an image pickup device, a control method, which can acquire an image with reduced unnecessary components by performing a shooting process based on the detection result of unnecessary components detected before shooting. And the program can be provided.

Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these examples, and various modifications and modifications can be made within the scope of the gist thereof.

204a Unnecessary component detector (detection means)
210 System controller (control means)

Claims (13)

A detection means for detecting an unnecessary component of the first image acquired at the first timing, and
It has a control means for performing an imaging process for acquiring a second image at a second timing after the first timing based on the detection result of the unnecessary component by the detection means.
The control means is
When the detection means detects the unnecessary component, images of a plurality of different viewpoints are acquired as the second image, and images are obtained.
A control device comprising acquiring a composite image of a plurality of images from different viewpoints as the second image when the detection means does not detect the unnecessary component . The control device according to claim 1 , wherein the detection means detects the unnecessary component included in an image of a plurality of different viewpoints as the first image. Claim 1 or the control means is characterized in that the first timing is calculated based on the second timing, the shooting time of the first image, and the detection period of the unnecessary component. 2. The control device according to 2. The control means is
The effect of reducing unnecessary components is determined based on the pupil division information regarding the image sensor and the shooting condition information regarding the image pickup optical system.
When it is determined that there is an effect of reducing the unnecessary component, a plurality of images from different viewpoints are acquired as the second image.
6 . Control device. Further, it has a light source direction calculation means for calculating the direction of the light source based on the shooting condition information.
The control device according to claim 4 , wherein the control means determines the effect of reducing unnecessary components based on the direction of the light source and the pupil division information. When the control means detects that the unnecessary component has disappeared in the first image that the detection means has started to acquire at the first timing, the second image is an image of a plurality of different viewpoints. The control device according to any one of claims 1 to 5 , wherein the composite image of the above is acquired. The control device according to claim 6 , wherein the control means acquires images of a plurality of different viewpoints as the second image when a predetermined allowable period has elapsed. The control means is
Correlation calculation of the unnecessary component of the first image is performed.
Based on the result of the correlation operation, the period of the unnecessary component is acquired, and the period is obtained.
The control device according to claim 6 or 7 , wherein the second timing is set based on the cycle of the unnecessary component. The control means is
As the second image, images of a plurality of different viewpoints are acquired, and images are obtained.
When the detection means detects an unnecessary component of the second image and has a margin time longer than a predetermined time, the second timing is reset to acquire the second image. The control device according to any one of claims 6 to 8 . An image sensor that photoelectrically converts an optical image formed via an image pickup optical system,
A detection means for detecting an unnecessary component of the first image acquired by the image pickup device at the first timing, and a detection means.
Based on the detection result of the unnecessary component by the detection means, a control means for performing an imaging process for acquiring a second image by the image pickup device at a second timing after the first timing. Have and
The control means is
When the detection means detects the unnecessary component, images of a plurality of different viewpoints are acquired as the second image, and images are obtained.
An image pickup apparatus characterized in that when the detection means does not detect the unnecessary component, a composite image of a plurality of images from different viewpoints is acquired as the second image . The claim is characterized in that the image pickup element has a plurality of photoelectric conversion units for one microlens, the microlenses are arranged two-dimensionally, and images of a plurality of different viewpoints are acquired. 10. The image pickup apparatus according to 10. A step of detecting an unnecessary component of the first image acquired at the first timing, and
It has a step of performing a shooting process for acquiring a second image at a second timing after the first timing based on the detection result of the unnecessary component.
In the step of performing the shooting process
When the unnecessary component is detected in the detection step, a plurality of images from different viewpoints are acquired as the second image.
A control method comprising acquiring a composite image of a plurality of images from different viewpoints as the second image when the unnecessary component is not detected in the detection step . A step of detecting an unnecessary component of the first image acquired at the first timing, and
It has a step of performing a shooting process for acquiring a second image at a second timing after the first timing based on the detection result of the unnecessary component.
In the step of performing the shooting process
When the unnecessary component is detected in the detection step, a plurality of images from different viewpoints are acquired as the second image.
A program characterized in that , when the unnecessary component is not detected in the detection step, a computer is made to execute a control method for acquiring a composite image of images of a plurality of different viewpoints as the second image .

Images