JP4792929B2 - Digital camera - Google Patents

Description

  The present invention relates to an image processing apparatus.

  2. Description of the Related Art A so-called super-resolution image processing technique is known in which imaging is performed with a slight displacement of an image sensor or camera angle, and two captured images are combined to generate an image having a higher resolution than the original image. (For example, refer to Patent Document 1).

Prior art documents related to the invention of this application include the following.
JP 2002-300372 A

  By the way, with a general digital still camera, a through image is displayed on a monitor to determine the angle of view, and after adjusting the focus of the photographic lens for the main subject, the main shooting is performed when there is a shutter release operation. For example, when shooting a dark moving subject, a “blurred” image is obtained unless the shutter speed is increased in the main shooting. Here, even if the above-described super-resolution imaging is performed, the resolution is increased, but the image itself cannot be brightened. In addition, since the main shooting must be performed at least twice to superimpose the captured image, shooting takes time and effort.

According to a first aspect of the present invention, there is provided a digital camera that captures a subject image and outputs an image signal; a main photographing operation that generates a main photographed image based on the image signal of the image sensor; based on the image signal, and control means for selectively executing a through image capturing operation of generating a plurality of through image having a lower pixel resolution than the pixel resolution of the captured image, produced by the through image photographing operation a plurality of said plurality of through image as a pixel of another through-image is located between the adjacent pixels and the pixel of one of the through image of the through image by combining super-resolution, the captured image a super-resolution combining means for generating a through image combined image having a pixel resolution and the same pixel resolution, the generated the through image synthesis image and the main imaging operation by the super-resolution synthesis unit Comprising a normal combining means for combining the generated the captured image, and the present said normally the captured image to be synthesized by synthetic means, the through image combined image, which is performed in tandem with each other It is generated by the photographing operation and the through image photographing operation, respectively .

  According to the present invention, a bright image with high resolution can be obtained even when the subject is dark.

  An embodiment in which the image processing apparatus of the present invention is applied to an imaging apparatus will be described. The image processing apparatus according to the present invention is not limited to the imaging apparatus according to the embodiment, and can be applied to any apparatus.

  FIG. 1 shows a configuration of an imaging apparatus including an image processing apparatus according to an embodiment. The image pickup device 1 is an image pickup device capable of single-shot shooting and continuous shooting shooting of still images, and shooting of moving images. The shooting lens 2 shoots a subject image formed on the light receiving surface and outputs a video signal. To do. An image sensor such as a CCD or CMOS can be used as the image sensor 1, and an electronic shutter operation can be performed by controlling the charge accumulation time. The photographing lens 2 is composed of a plurality of lenses, and focusing and zooming are possible by the lens driving circuit 3. The photometry circuit 4 measures the subject and outputs photometric data. The operation member 5 includes various operation buttons and setting buttons such as a release button (shutter button), a zoom button, and a command dial (not shown).

  The image sensor driving circuit 6 drives the image sensor 1 based on the photometric data of the subject. The signal processing circuit 7 performs processing such as DC reproduction, A / D conversion, white balance, and gamma conversion on the video signal output from the image sensor 1, generates image data, and outputs the image data to the data processing circuit 8. The data processing circuit 8 performs resolution (number of pixels) conversion processing for displaying the image data on the monitor 9 as necessary, and outputs it to the compression / decoding (decompression) circuit 10 and the display control circuit 11. The display control circuit 11 performs predetermined signal processing on the image data and displays an image on the monitor 9. The display control circuit 11 also performs processing for superimposing overlay image data such as a shooting menu and a cursor on the image data. As a result, the overlay image is superimposed and displayed on the subject image of the monitor 9.

  The compression / decoding circuit 10 performs compression processing on the image data and records it on the recording medium 12. The monitor 9 displays an image corresponding to the image data compressed and recorded on the recording medium 12. The monitor 9 can also display an image corresponding to the image data recorded on the recording medium 12 as a reproduced image. The monitor 9 can be an LCD or any other type of display capable of displaying images and characters. As the recording medium 12, various types of recording devices capable of recording images and characters as well as various hard disks and various memory cards can be used.

  The control circuit 13 inputs an operation signal from the operation member 5, photometry data from the photometry circuit 4, and the like, and the lens drive circuit 3, the image sensor drive circuit 6, the signal processing circuit 7, the data processing circuit 8, the monitor 9, the compression / compression The decoding circuit 10, the display control circuit 11, and the recording medium 12 are controlled. The control circuit 13 reads out the image data recorded on the recording medium 12, performs a decoding process in the compression / decoding circuit 10, and outputs it to the monitor 9 through the data processing circuit 8 and the display control circuit 11.

  The control circuit 13 also extracts the image data of the focus detection area set on the imaging screen based on the operation signal of the operation member 5, and calculates the contrast value of the area or the high spatial frequency component amount of the area. A so-called contrast AF operation is performed to adjust the focus state of the subject image on the light receiving surface of the image sensor based on the calculation result. That is, the photographic lens 2 is driven by the lens driving circuit 3, and the photographic lens 2 is focused on the subject in the focus detection area.

  The control circuit 13 further drives the photographing lens 2 by the lens driving circuit 3, analyzes the sequentially obtained video signal or image data for each subject in the screen, and the lens when the contrast value in the area becomes maximum. Based on the position, relative shooting distance information for each subject is acquired. Instead of such contrast AF, the lens may be driven by phase difference AF of a known pupil division method. Also in this case, the shooting distance information of each area can be obtained by the automatic focusing operation.

  The control circuit 13 drives the photographic lens 2 by the lens driving circuit 3 based on the operation of the zoom button of the operation member 5 and enlarges or reduces the subject image formed on the light receiving surface of the image sensor. I do. The control circuit 13 controls the resolution conversion function of the data processing circuit 8 according to the operation of the zoom button of the operation member 5, and the captured image data captured by the image sensor 1 or the recorded image recorded on the recording medium 12. Enlarge or reduce data and perform electrical zooming.

  Next, the main photographing operation and the through image photographing operation in the imaging apparatus according to the embodiment will be described. Note that the through image shooting is a preliminary shooting performed prior to the main shooting. For example, in a digital still camera, the through image is displayed on the monitor to determine the angle of view, and the focus of the shooting lens is adjusted for the main subject. When the shutter release operation is performed, a high-resolution still image is obtained.

  The through image photographing operation is executed together with the automatic focusing operation of the photographing lens 1 while the release button of the operation member 5 is half-pressed (pressing operation up to the first stroke). When the release button is half-pressed, the control circuit 13 controls the image sensor driving circuit 6 to execute a through image photographing operation by the image sensor 1. When the image pickup device 1 receives a drive signal for a through image shooting operation, the image pickup device 1 adds accumulated charges of pixels of the same color in the vicinity of the light receiving surface (in the case of a single-plate color image pickup device) in order to realize high sensitivity. Compared with high-resolution main shooting, a low-resolution video signal is continuously output at a high frame rate of, for example, 30 frames / second.

  The video signal output from the image sensor 1 is subjected to predetermined processing by the signal processing circuit 7 and the data processing circuit 8, and then supplied to the monitor 9 via the display control circuit 11 and displayed on the monitor 9. . Here, an example is shown in which the accumulated charges of pixels of the same color in the vicinity on the image sensor 1 are added and read out, but pixel thinning out reading may be performed according to a predetermined rule. The photographer can visually recognize the state of the object scene to be photographed in the actual photographing from the image displayed on the monitor 9.

  The automatic focusing operation for the main subject is executed by half-pressing the release button so that the contrast of the video signal in a predetermined area (focus detection area) in the video signal output from the image sensor 1 is maximized. The focus of the taking lens 2 is adjusted. When focusing is achieved as a result of the focus adjustment, the AF indicator lamp of the monitor 9 is switched from red to green to notify the photographer that the taking lens 2 has focused on the main subject.

  If the release button is fully pressed during the through image shooting operation (pressing operation up to the second stroke larger than the first stroke), the main shooting operation is executed. The control circuit 13 controls the image sensor driving circuit 6 to execute the main photographing operation by the image sensor 1. When the image sensor 1 receives a drive signal for the main photographing operation, the image sensor 1 outputs the accumulated charge of each pixel on the light receiving surface as it is. Therefore, in this photographing operation, a video signal with a higher resolution is output from the image sensor 1 as compared to the through image photographing operation described above.

  The video signal output from the imaging device 1 is subjected to predetermined processing by the signal processing circuit 7 and the data processing circuit 8, and then supplied to the monitor 9 via the display control circuit 11 and displayed on the monitor 9. At the same time, it is recorded on the recording medium 12 via the compression / decoding circuit 10.

  The imaging apparatus according to the embodiment performs a super-resolution composition operation in addition to the normal through image photographing operation and the still image main photographing operation described above. In the super-resolution composition operation, alignment processing of a plurality of through images acquired after focusing of the photographing lens 1 is performed with an accuracy of less than one pixel, and these through images are synthesized and a through image having the same pixel resolution as that of the main photographing. A composite image is generated, and the through image composite image and the actual captured image acquired by the actual capturing are combined.

  In this embodiment, when the super-resolution mode for executing the super-resolution operation is set, the shutter priority mode is automatically set as the photographing mode, and a long shutter speed is set according to the photometric result of the photometric circuit 4. Set to prevent camera shake and camera shake due to subject movement.

  The super-resolution composition operation will be described taking a single-plate image sensor as an example. In recent years, there is a tendency to increase the pixel aperture ratio of the image sensor in order to increase sensitivity in parallel with the increase in resolution of the image sensor. As a result, the gap between the pixel openings is becoming narrower. The photographing operation by the solid-state image sensor corresponds to real-world spatial sampling, and the pixel aperture of the image sensor indicates that this spatial sampling is performed by integrating the light quantity in a certain region. In other words, the frequency characteristics of the spatial sampling in this case are degraded by the integration.

  Here, shooting is performed by shifting the position of the image sensor so that the pixel at the time of the second shooting is positioned between the pixels of the image sensor at the time of the first shooting, and a plurality of acquired images are synthesized. An imaging apparatus that performs super-resolution synthesis processing is known. However, even if such super-resolution synthesis is performed, the improvement in resolution cannot be expected so much simply by increasing the number of pixels. This is because if the pixel opening is enlarged and the gap between the pixel opening and the pixel opening is narrowed, the pixel opening at the first shooting and the pixel opening at the second shooting overlap, that is, the spatial sampling position is This is because they overlap. When such an overlap occurs, it becomes impossible to accurately separate the pixel opening at the first photographing and the pixel opening at the second photographing, and the resolution cannot be improved.

  Therefore, in the imaging apparatus according to the embodiment, a plurality of through images are super-resolution synthesized to generate a through image synthesized image having the same number of pixels as the number of pixels obtained by the actual imaging, and the through image synthesized image is captured. Composite with the image.

  For example, when shooting a moving subject with low brightness, a blurred image is obtained unless the shutter speed is increased in the main shooting. However, as described above, in the through image shooting operation, a plurality of through images are super-resolved and combined to generate a through image combined image, and the main image is added to reproduce the high frequency components of the through image combined image. If combined, a bright image without blurring can be generated. In particular, as described above, in the through image shooting operation, if the pixel thinning-out readout is performed according to a predetermined rule, the number of pixels of the through image is smaller than the number of pixels of the actual image and the through image is captured. When an image having the same number of pixels as the image is developed, a gap between the pixel opening and the adjacent pixel opening is widened. Therefore, even if a plurality of through images are super-resolved and synthesized, the pixel openings do not overlap each other, and the resolution can be further increased.

  Note that even if the live view image is combined with the actual shot image without super-resolution combining, the image quality may be deteriorated due to the difference in the number of pixels between the two images (a brightness step can be created on the screen). Therefore, an appropriate image cannot be generated.

  FIG. 2 is a flowchart illustrating super-resolution mode processing of the imaging apparatus according to the embodiment. The operation of the embodiment will be described with reference to this flowchart. When the super-resolution shooting mode is set by the operation member 5, the control circuit 13 controls each part so as to start the through image shooting operation and starts the processing shown in FIG.

  In step 1, the shutter speed priority mode is set as the shooting mode. When shooting in super-resolution shooting mode, set the shutter speed priority shooting mode to high electronic shutter speed forcibly, and the through image used for super-resolution composition and the actual shooting image in the main shooting will be subject to camera shake or subject. Avoid blurring by moving. In the next step 2, it is confirmed whether or not the release button is half-pressed. When the release button is half-pressed, the process proceeds to step 3. In step 3, the lens driving circuit 3 is controlled to perform contrast AF, and the focus of the photographing lens 2 is adjusted. While the release button is pressed halfway, the focus of the photographic lens 2 is adjusted, and when an in-focus state is detected in a predetermined focus detection area, the AF indicator lamp (not shown) of the monitor 9 is lit in green. To inform.

  In step 4, it is confirmed whether or not the half-press of the release button is continued. If the half-press is interrupted, the process returns to step 2 to repeat the above-described processing. When the half-pressing is continued, the process proceeds to step 5 and the image pickup device 1 is driven by the image pickup device drive circuit 6 while the photographing lens 2 is held in focus by the lens drive circuit 3 to repeat through image shooting. The captured through image is recorded on a memory (not shown) built in the data processing circuit 8 while being displayed on the monitor 9.

  In step 6, subject recognition processing is performed using the through image, and for example, it is determined whether or not there is a specific “Mr. A” in the object scene. The subject of the subject recognition process is not limited to a person, and may be an object other than a person or a color region. It may also be a specific subject registered in advance as an image. Details of the subject recognition processing will be described later.

  FIG. 3A is a timing chart showing the operation of super-resolution mode processing. Automatic focus adjustment (AF) is performed by half-pressing the release button, and through image shooting is performed while the photographing lens 2 is kept in focus. In this embodiment, it is assumed that ten through images are photographed in order to perform super-resolution composition of the through images. If the release button is fully pressed while taking a through image while the focusing lens 2 is kept in focus, the camera waits until 10 (frame) through images have been taken, and 10 through images are taken. After shooting is complete, take a picture.

  In this embodiment, the through image that is used as a reference for alignment when generating a through image composite image by super-resolution combining a plurality of through images, that is, a through image captured immediately before the main photographing, that is, FIG. In the example shown in (a), the tenth through image is used. By using the through image taken immediately before the main shooting as the reference through image for alignment, the difference in the acquisition time between the reference through image and the main shooting image can be minimized. The position of the through image composite image and the actual captured image can be adjusted with a minimum amount of motion compensation.

  When the through image is stored in the built-in memory of the data processing circuit 8, the 10 through images are written in the order in which they were photographed, but the first through image is erased before the 11th through image is written. To do. Thereafter, the oldest through image is erased before writing the latest through image. That is, with the photographing lens 2 focused on a predetermined subject, the latest 10 through images immediately before the main photographing are always stored in the built-in memory of the data processing circuit 8.

  In this embodiment, ten through images are stored and used for super-resolution composition. However, the number of through images used for super-resolution composition is not limited to ten. The greater the difference between the resolution of the through image (number of pixels) and the resolution of the actual captured image (number of pixels), the greater the number of through images used for super-resolution composition.

  Returning to FIG. 2, the description of the super-resolution mode processing will be continued. In step 7, it is confirmed whether or not the half-press operation of the release button is continued. If the half-press operation is interrupted, the process proceeds to step 8, where the through image shooting is finished and the built-in memory of the data processing circuit 8 is stored. After deleting the stored through image, the process returns to step 2 to repeat the above-described processing.

  On the other hand, when the half-press operation of the release button is continued, the process proceeds to step 9 to check whether or not the release button is fully pressed. When the release button is not fully pressed, the process returns to step 5 to repeat the above-described processing. When the release button is fully pressed, the process proceeds to step 10, and it is confirmed whether or not a predetermined number of through images have been shot, in this embodiment, ten. If the number of through images stored in the memory of the data processing device 8 is less than 10, the process returns to step 5 to repeat the above-described processing.

  If ten through images are stored in the memory of the data processing device 8, the process proceeds to step 11, the image pickup device driving circuit 6 is controlled to perform the main photographing with the image pickup device 1, and the built-in memory of the data processing circuit 8 is used. Remember. In the subsequent step 12, super-resolution synthesis is performed using the 10 through images stored in the memory of the data processing circuit 8.

  FIG. 4 is a diagram illustrating a through image selection method when super-resolution synthesis is performed using a plurality of through images. In the figure, a cross indicates a through image taken immediately before the main photographing, that is, the pixel position of the tenth through image in this embodiment. As described above, in this embodiment, the tenth through image taken immediately before the main photographing is used as a reference through image when super-resolution synthesis is performed by aligning a plurality of through images. To do. If a plurality of through images are combined so that the pixels of other through images come to the positions of “□”, “Δ”, and “O” between the pixels “×” of the reference through image, the original through image A through image composite image having a resolution (number of pixels) that is four times the resolution (number of pixels) can be generated. In this specification, the pixel positions of “□”, “Δ”, and “◯” are referred to as “ideal pixel positions” when super-resolution synthesis is performed.

  Through-images with pixels at the ideal pixel positions “□”, “△”, and “◯” for super-resolution synthesis from 9 images excluding the reference through-images out of 10 through-images acquired in through-image shooting If three images are selected and combined with the reference through image, a through image combined image with higher resolution than the original through image can be obtained. However, among the nine through images excluding the reference through image, there is rarely a through image in which a pixel comes to an ideal pixel position. Therefore, from the nine through images excluding the reference image, three through images of the pixels closest to the ideal pixel position are selected, and the through image data at the ideal position is calculated by interpolation processing, and then super-resolution synthesis is performed. Do.

  Of course, in order to shorten the synthesis time of the super-resolution image, it is possible to select the through image of the pixel closest to the ideal pixel position and perform the super-resolution synthesis assuming that it is in the ideal pixel position without performing the interpolation calculation process. Good.

  When compositing a through image close to the ideal pixel position and a reference through image, the through images cannot be completely overlapped due to camera shake or subject movement during continuous shooting of the through images. It is necessary to shift and overlap. If this shift amount is large, the common image area when the images are combined is narrowed, and a super-resolution image is generated in the common area, but an image is not combined in the non-common area and a super-resolution image is not generated. Therefore, when the through images are shifted and overlapped in super-resolution synthesis, if the shift amount is large, a region in which the pixel resolution does not increase in the through-image synthesized image after super-resolution synthesis increases. For this reason, when selecting a through image at a pixel position closest to the ideal pixel position described above, the selection condition is that the area of the synthesis area in super-resolution synthesis is wide, that is, the image area common to the reference through image is wide. May be. Further, it may be added to the selection condition that the shift amount with respect to the reference through image is small.

  In FIG. 4, for example, the through image closest to the ideal pixel position “□” has a large shift amount with respect to the reference through image (pixel position “×”), and “ When the pixel at the “□” position is missing, the image area may be calculated from the remaining pixel data of “×”, “Δ”, and “◯” by interpolation or the like. Such a phenomenon occurs particularly at the edge of the image.

  Also, if the shooting conditions at the time of shooting, that is, the shutter speed, aperture value, focal length, image signal amplification factor, white balance, gamma value, and the like are different, there is a possibility that a through image that is not suitable for super-resolution synthesis will be formed. Therefore, when there is a possibility that these shooting conditions are changed, a through image to be combined with the reference through image may be selected on condition that the shooting conditions are the same. Alternatively, in the super-resolution image processing, after the in-focus state is achieved by the automatic focus adjustment, the change of the shooting condition of the through image may be prohibited.

  In step 12 of FIG. 2, a through image composite image is generated by super-resolution combining using a plurality of through images, and then the through image composite image and the actual captured image are combined in step 13 and recorded on the recording medium 12. At the same time, it is displayed on the monitor 9.

  Here, a dialog for confirming to the photographer whether or not to record the composite image of the through image composite image and the actual captured image on the recording medium 12 is displayed on the monitor 9 and is recorded only when recording is selected. It may be recorded on the medium 12. When the composite image of the through image composite image and the actual captured image is not recorded, the photographer confirms whether the actual captured image is automatically recorded on the recording medium 12 or whether the actual captured image is recorded. A dialog for displaying the image may be displayed on the monitor 9 and the actual captured image may be recorded on the recording medium 12 only when recording is selected.

  Until the synthesis process of the live view composite image and the actual captured image is completed, the image captured by the actual capture is displayed on the monitor 9, and an icon indicating that super-resolution composition is being performed is displayed on the actual captured image. Superimposed on the display. When the composition processing of the through image composite image and the actual captured image is completed, the composite processing result image is displayed on the monitor 9 instead of displaying the actual captured image and the icon.

  When recording the composite image of the through image composite image and the actual captured image on the recording medium 12, the through image data used for the super-resolution composition, the through image composite image data after the super-resolution composite, and the actual captured image data The amount of compensation for misalignment in super-resolution synthesis, the shooting time and shooting time of each image used for synthesis, the number of through images used for super-resolution synthesis, etc. may be recorded in association with each other as necessary. Good. For example, record them in the same folder, or record them with the same file name but different extensions. By adopting such a recording method, it is possible to simplify the image editing work after photographing.

<Monitor display during live view shooting>
The through images captured in the through image capturing operation after achieving the focus by the automatic focus adjustment (AF) may be sequentially superimposed and displayed on the monitor 9 without motion compensation. Thereby, it is possible to grasp an image area that is not common with the above-described reference through image at the screen edge during through image shooting. If the photographer determines that there is a large area where the live view composite image at the edge of the screen cannot be obtained by looking at the monitor display, immediately release the half-press of the release button to shoot the image so far and store it in memory. The stored through image can be deleted.

<Modification of selection method of through image used for super-resolution synthesis>
In the above-described embodiment, an example in which the through image taken immediately before the main shooting after the release button is fully pressed is used as a reference through image for super-resolution synthesis is shown. The through image obtained first may be used as the reference through image. In this case, every time a through image is taken after taking a reference through image, whether or not the through image of the pixel closest to the ideal pixel position of the reference through image is a through image having a wide image area in common with the reference through image, In other words, it is possible to determine whether or not the image is an optimal through image that can be used for super-resolution synthesis. For a through image that is determined to be usable for super-resolution synthesis, the registration processing and the synthesis processing are immediately performed. Can start.

  Specifically, the first through image obtained after focusing by automatic focus adjustment (AF) is set as a reference through image, and the pixel position of the through image taken after the reference through image is the position of the above-described reference through image. For example, it is determined whether or not the pixel position is less than 1/8 pixel pitch from the ideal pixel position. If the deviation of the pixel position is less than 1/8 pixel pitch, the through image is used for super-resolution synthesis. At this time, by displaying the acquisition status of the through image used for super-resolution synthesis on the monitor 9 by a progress bar or the like, the photographer can easily grasp the progress status of the imaging.

  Further, as shown in FIG. 3B, when the release button is fully pressed during such a through image shooting, the through image is obtained when the required number of through images that can be used for super-resolution composition are obtained. It is also possible to stop shooting and immediately execute the actual shooting operation. That is, when a through image composite image as a super-resolution composite result is obtained, a bright image with high resolution can be reliably captured by permitting the main photographing operation.

  Furthermore, after the focus is achieved by automatic focus adjustment (AF), if the release button is pressed all the way down, the main image is taken after a predetermined number of through images have been captured, and the super-resolution synthesis of the through image and the through image synthesis are performed. You may make it perform automatically to the synthesis | combination of an image and a real picked-up image. In this case, the first through image after achieving the focus may be used as the reference image, or the through image immediately before the main photographing may be used as the reference image.

  Furthermore, when the release button is pressed halfway, the first through image after the completion of the composition is used as the reference through image, automatic focusing of the photographic lens and through image photographing operation are performed, and super-resolution composition is continuously performed. If the release button is released, the super-resolution composition may be stopped.

<< Other Modifications of Through Image Selection Method Used for Super-Resolution Synthesis >>
In the above-described embodiment, an example in which a through image used for super-resolution synthesis is acquired before the main photographing is shown. However, the through image may be acquired after the main photographing. Specifically, as shown in FIG. 3C, after achieving focus by automatic focus adjustment (AF), the main shooting operation is executed by pressing the release button fully, and the main shooting image is displayed on the data processing device 8. Temporarily store in memory. Immediately after the main photographing operation, photographing of a through image used for super-resolution synthesis is started. Then, the first through image taken after the main shooting is set as a reference through image, and the pixel position of the through image taken after the reference through image is, for example, 1/8 pixel pitch from the ideal pixel position of the reference through image. If the pixel position shift is less than ８ pixel pitch, the through image is used for super-resolution synthesis.

  At the time of obtaining a through image having pixels less than 1/8 pixel pitch at all ideal image positions around the pixel position of the reference through image, shooting of the through image is terminated, and the reference through image and the selected through image are used. Then, a through image synthesized image is generated by super-resolution synthesis, and image processing for synthesizing the through image synthesized image and the actual captured image is executed.

  In addition, when performing a through image shooting after such a main shooting, the through image shooting following the main shooting may be performed only while the release button is fully pressed. When the full release of the release button is released during the shooting of the through image, a series of image processing including super-resolution synthesis is not executed, and the book that has been taken and temporarily stored in the memory of the data processing circuit 8 The captured image and the through image may be deleted. Alternatively, when the super-resolution mode is set, the through image shooting may be automatically executed after the main shooting by pressing the release button fully even if the release button is not fully pressed. .

  In this way, by shooting the through image used for super-resolution composition after the main shooting, the through image taken at the time closest to the main shooting becomes the reference through image in the super-resolution combining. A super-resolution composite image close to the time can be obtained. In addition, since the selection of the through image and the super-resolution composition are executed in parallel with the through-image shooting operation, it is sufficient if there is a memory for storing the reference through-image data and the composite image data in the super-resolution composition process. The capacity can be reduced.

  In the above-described embodiment, when selecting a through image to be used for super-resolution synthesis, an example has been shown in which whether or not the through image of the pixel closest to the ideal pixel position is confirmed with respect to the ideal pixel position of the entire screen range. However, the above-described confirmation process may be executed in a partial area of the preset shooting screen. Note that, for example, a human face area may be extracted from an image using a known face recognition technique, and it may be confirmed whether or not the human face area is a through image of a pixel closest to the ideal pixel position. In general image processing, since processing is performed in order from the top to the bottom of the image, it may be confirmed whether or not it is a through image of the pixel closest to the ideal pixel position in the upper region of the image. By these methods, the through image selection processing time can be shortened.

《Subject recognition processing》
Next, subject recognition in a super-resolution composite image will be described. It is known to pay attention to a subject included in a captured image, extract the subject from the captured image, and perform image processing according to the subject. For example, a red-eye state associated with a light emission operation of an illumination is detected from a captured image, and the red-eye state is reduced by post-processing for that region.

  By the way, if subject recognition processing is performed on the basis of a super-resolution composite image, a series of processing from shooting to recording takes an enormous amount of time, which may hinder subsequent shooting operations. Therefore, in this embodiment, subject recognition processing is not performed using the super-resolution synthesized image, but subject recognition processing is performed using one image used for super-resolution synthesis.

  Here, for example, a process of detecting a human face in a through-image for super-resolution synthesis captured before and after the above-described main imaging, and reducing the red-eye phenomenon by post-processing when a red-eye phenomenon occurs Will be described. Note that the subject recognition process can detect not only the face area of a person but also many other parts, such as the upper body, hand, and whole body of the person. Further, as described above, the subject recognition process target is not limited to a person, and may be an object other than a person or a specific color region. It may also be a specific subject registered in advance as an image.

  Detection of a human face area is performed by using a through image after an automatic focusing operation and extracting skin color area, contour extraction (template matching), or, if necessary, temperature distribution data in a captured image. When the human face area is detected, the human eye area can be identified from the through image using the color level and contour information of the image data in the face area. Position data in the super-resolution synthesized image, that is, the through-image synthesized image based on the through-image used for detection and the alignment information (motion compensation information) at the time of super-resolution synthesis. And is recorded on the recording medium 12 together with live view composite image (super-resolution composite image) data as necessary.

  Note that if subject recognition is performed using a reference through image used for super-resolution synthesis, an accurate subject recognition result can be obtained in a through-image synthesized image as a super-resolution synthesis result.

  The control circuit 13 determines whether or not a red component is included in the image data of the eye region based on the position data of the eye region in the through image composite image (super-resolution composite image). If it is determined that a red component of a predetermined level or higher is included, red level reduction processing is performed to remove the red-eye phenomenon.

  In this way, a subject is recognized in a through image with a low pixel resolution used for super-resolution composition, and a through image composite image (super-resolution composite image) with a high pixel resolution is processed based on the recognition result. The time required for subject recognition in the super-resolution composite image can be shortened. In addition, the position information related to the subject in the through-image composite image (super-resolution composite image) is obtained by correcting (converting) the alignment information (motion compensation information) at the time of super-resolution composition. Even when the area is small, the subject area can be quickly detected in an image with a high pixel resolution.

  Note that the red-eye reduction process may be performed before recording the through-image synthesized image (super-resolution synthesized image) data, or may be performed by reproducing the super-resolution synthesized image recorded on the recording medium 12. . Alternatively, the red eye phenomenon may be reduced by an external image processing apparatus such as a personal computer. The red-eye phenomenon reduction process may be performed on a combined image of the through image combined image and the actual captured image.

  According to one embodiment, an image sensor that captures a subject image and outputs an image signal, a main shooting operation that performs shooting at the pixel resolution of the image sensor, and a pixel resolution that is higher than that of the main shooting operation using the image sensor. In an image processing apparatus that includes a control circuit that executes a through image shooting operation for performing low shooting, a plurality of through images shot by the through image shooting operation are super-resolved and combined with a pixel that has the same pixel resolution as the actual shooting operation. Since a live view composite image with resolution is generated and the live view composite image and the actual captured image by the actual shooting operation are combined, a bright image with high resolution can be obtained even when the subject is dark.

  According to one embodiment, a through image acquired at a time closest to the acquisition time of the main captured image by the main shooting operation from among a plurality of through images captured by the through image shooting operation is set as a reference through image, and the reference through image is acquired. Since the other through images are super-resolved and combined with the reference through image so that the other through image pixels come between the pixels of the image, the acquisition time difference between the reference through image and the actual captured image Can be minimized. Therefore, the effects of camera shake and subject movement during this period can be suppressed to the minimum, and the alignment of the through image composite image and the actual captured image can be executed with the minimum amount of motion compensation.

  According to an embodiment, when selecting a through image to be super-resolution synthesized with a reference through image from a plurality of through images, a through image having a wide image area in common with the reference through image is selected. Therefore, the region of the super-resolution image having a high pixel resolution is widened in the through image composite image after the super-resolution synthesis, and a bright image having a high resolution can be finally obtained.

  According to an embodiment, when selecting a through image to be super-resolved and synthesized with a reference through image from a plurality of through images, pixels and pixels of the reference through image are set in a predetermined region of the shooting screen. Since the process to confirm whether or not other through-image pixels come between is done, it is possible to shorten the selection processing time of the through-image to be super-resolved with respect to the reference through-image, and the shutter Bright images with high resolution can be taken without missing the opportunity.

  According to an embodiment, a human face area is detected from a captured image by a subject recognition method, and another through image is detected between pixels of the reference through image in the human face area of the captured image. Since the process of checking whether or not a pixel comes is performed, it is possible to shorten the selection processing time of a through image to be super-resolved with respect to the reference through image, and the resolution is high without missing a photo opportunity. In addition to being able to capture a bright image, it is possible to capture a person's face brightly and clearly in portrait shooting.

  According to one embodiment, a process for checking whether or not another through-image pixel comes between the pixels of the reference through-image in the upper region of the shooting screen is performed. In general image processing, processing is performed in order from the top to the bottom of the image, so that it is possible to reduce the processing time for selecting a through image to be super-resolved with respect to the reference through image, and resolution without missing a photo opportunity. High brightness images can be taken.

  According to an embodiment, a through image taken immediately before the main photographing operation is set as a reference through image, and another through image taken before the main photographing operation is super-resolution combined with the reference through image. As a result, the difference in acquisition time between the reference live image and the actual captured image is minimized, and the effects of camera shake and subject movement during this period can be minimized. Alignment can be performed.

  According to one embodiment, since the actual photographing operation is permitted when a predetermined number of through images are photographed in the through image photographing operation, the pixel periphery of the reference through image is used when performing super-resolution composition. The probability that a through image with a pixel at the ideal pixel position can be selected is increased, and a through image synthesized image with a high resolution can be generated by super-resolution synthesis, and finally a bright image with a high resolution can be obtained.

  According to one embodiment, a through image taken immediately after the main photographing operation is set as a reference through image, and another through image taken after the main photographing operation is super-resolution combined with the reference through image. . Since the through image taken at the time closest to the main photographing is used as the reference through image in the super-resolution composition, a super-resolution composite image close to the time when the release button is fully pressed can be obtained. Further, since the selection of the through image and the super-resolution composition are executed in parallel with the through image shooting operation, it is sufficient to have a memory for storing the reference through image, and the memory capacity can be reduced.

  According to one embodiment, a photographing lens for forming a subject image on an image sensor and a lens driving circuit that performs automatic focusing by driving the photographing lens, and performs automatic focusing of the photographing lens, Since the live view shooting operation is executed when the taking lens is in focus, super-resolution synthesis can be performed using a live view image that is in focus on the subject, and finally the resolution that is in focus on the subject. High bright images can be obtained.

  According to one embodiment, a through image that is first taken after the in-focus state of the taking lens is achieved is set as a reference through image, and pixels of other through images are placed between pixels of the reference through image. In addition, another through image taken in the focused state of the taking lens with respect to the reference through image is super-resolved and synthesized. Thus, every time a through image is taken after taking a reference through image, whether or not the through image of the pixel closest to the ideal pixel position of the reference through image is a through image having a wide image area in common with the reference through image, In other words, it can be determined whether or not it is an optimal through image that can be used for super-resolution synthesis. Can start. Therefore, it is possible to quickly obtain a through image composite image as a super-resolution composite result, and to capture a bright image with high resolution.

  According to the embodiment, since the main photographing operation is permitted when a through image composite image as a super-resolution composition result is obtained, a bright image with high resolution can be reliably captured.

  According to one embodiment, when the release button is half-pressed, automatic focusing of the photographic lens and through-image shooting operation are performed and super-resolution synthesis is performed. When the release button is released, super-resolution is performed. Since the synthesis is stopped, the super-resolution synthesis of the through image can be operated by half-pressing the release button, and the operability can be improved.

  According to one embodiment, subject recognition is performed from a through image used for super-resolution composition, and subject recognition processing of a through-image composite image of the super-resolution composition result is performed using the recognition result. The time required for subject recognition in the resolution composite image can be shortened. As a result, it is possible to further reduce the selection processing time of a through image to be super-resolution combined with the reference through image, and it is possible to capture a bright image with high resolution without missing a photo opportunity.

  According to the embodiment, since the subject recognition is performed from the reference through image used for super-resolution composition, an accurate subject recognition result in the through-image composition image of the super-resolution composition result can be obtained.

  According to the embodiment, the position data of the subject recognition result is converted into the position data in the through image composite image based on the alignment information at the time of super-resolution composition. However, it is possible to quickly detect a subject area in an image with a high pixel resolution.

It is a figure which shows the structure of the imaging device provided with the image processing apparatus of one embodiment. It is a flowchart which shows the super-resolution mode process of the imaging device of one embodiment. 6 is a timing chart illustrating super-resolution mode processing of the imaging apparatus according to the embodiment. It is explanatory drawing of the process which performs super-resolution composition using a through image.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image pick-up element 2 Shooting lens 3 Lens drive circuit 4 Photometry circuit 5 Operation member 6 Image pick-up element drive circuit 7 Signal processing circuit 8 Data processing circuit 9 Monitor 10 Compression / decoding circuit 11 Display control circuit 12 Recording medium 13 Control circuit

Claims (16)

An image sensor that captures a subject image and outputs an image signal;
Based on the image signal of the image pickup device, a main shooting operation for generating a main shot image, and based on the image signal of the image pickup device, a plurality of through images having a pixel resolution lower than the pixel resolution of the main shot image. A control means for alternatively executing a through image shooting operation to be generated ;
Super-resolution synthesis of the plurality of through images so that pixels of other through images are located between adjacent pixels of one through image among the plurality of through images generated by the through image shooting operation. to the super-resolution combining means for generating a through image combined image having a pixel resolution and same pixel resolution of the captured image,
And a normal combining means for combining said captured image more generated in the main photographing operation and generated the through image combined image by the super-resolution synthesis means,
The main photographed image and the through image composite image synthesized by the normal synthesizing unit are respectively generated by the main photographing operation and the through image photographing operation executed in succession to each other. Digital camera . The digital camera according to claim 1, wherein
The super-resolution composition means uses, as a reference through image, a through image generated at a time closest to a generation time of the main captured image by the main shooting operation from among a plurality of through images captured by the through image shooting operation. A digital camera characterized in that another through image is super-resolved with the reference through image so that a pixel of another through image is located between adjacent pixels of the reference through image. The digital camera according to claim 2,
The super-resolution composition means has through-image selection means for selecting a predetermined number of through-images for super-resolution composition with the reference through-image from a plurality of through-images generated by the through-image shooting operation. And
The digital camera, wherein the through image selection means selects a through image having a wide image area in common with the reference through image when selecting the predetermined number of through images . The digital camera according to claim 2,
The super-resolution composition means has through-image selection means for selecting a predetermined number of through-images for super-resolution composition with the reference through-image from a plurality of through-images generated by the through-image shooting operation. And
When the through image selection unit selects the predetermined number of through images, the through image selection unit has another region within a predetermined range between adjacent pixels of the reference through image in a predetermined region of the shooting screen. A determination unit configured to determine whether or not a pixel of the through image is located; when the determination unit determines that the pixel of the other through image is located within the predetermined range, the other through image is selected; A digital camera characterized by The digital camera according to claim 4, wherein
Subject recognition means for detecting a human face area from the through image ,
The super-resolution composition unit selects the other through image when the determination unit determines that the pixel of the other through image is located within the predetermined range in the face area detected by the subject recognition unit. A digital camera characterized by The digital camera according to claim 4, wherein
The super-resolution combining means, when the pixels of the other of the through image in the upper region of said determining means shooting screen is determined to be located within the predetermined range, and wherein selecting the other of the through image Digital camera. The digital camera according to any one of claims 2 to 6,
The super-resolution composition unit uses a through image generated immediately before the main photographing operation as the reference through image, and superimposes another through image generated before the main photographing operation with respect to the reference through image. A digital camera characterized by resolution synthesis. The digital camera according to claim 7, wherein
An operation member for instructing execution of the main photographing operation;
Wherein, when the operating member during the through image photographing operation is operated, after the through-image of a preset number of sheets is generated, digital, characterized by performing the main photographing operation Camera . The digital camera according to any one of claims 2 to 6,
The super-resolution composition unit sets a through image generated immediately after the main photographing operation as the reference through image, and super-resolutions another through image generated after the main photographing operation with respect to the reference through image. A digital camera characterized by combining images. The digital camera according to claim 1, wherein
A photographic lens for forming a subject image on the image sensor;
Focusing means for driving the photographing lens to perform automatic focusing, and
The digital camera characterized in that the super-resolution composition means performs the super-resolution composition on the plurality of through images generated after completion of automatic focus adjustment by the focus adjustment means . The digital camera according to claim 10, wherein
The super-resolution composition means sets a through image generated first after the focus adjustment means has achieved focusing adjustment of the photographing lens as a reference through image, and the reference through image is compared with the reference through image. A digital camera characterized by super-resolution synthesis of another through image generated after generation . The digital camera according to claim 11, wherein
An operation member for instructing execution of the main photographing operation;
The digital camera according to claim 1 , wherein when the operation member is operated during execution of the through image shooting operation , the control unit executes the main shooting operation after generating the through image composite image . The digital camera according to claim 11, wherein
A release operation member capable of half-pressing operation and full-pressing operation is further provided.
The focus adjustment unit starts the automatic focus adjustment operation in response to a half-press operation of the release operation member,
The control means executes the main photographing operation in response to a full pressing operation of the release operation member,
The super-resolution composition unit sequentially performs the super-resolution composition every time a plurality of through images are generated after the generation of the reference through image, and the half-press operation of the release operation member is released. The digital camera is characterized in that the super-resolution synthesis is stopped . The digital camera according to claim 1 , wherein
Subject recognition means for detecting a human face area from the through image and acquiring position information of the face area;
Red-eye reduction means for performing red-eye reduction processing on one of the through image composite image and the composite image by the normal synthesis means,
The digital camera according to claim 1, wherein the red-eye reduction unit performs a red-eye reduction process on the one image based on the position information . The digital camera according to claim 14, wherein
The digital camera characterized in that the subject recognition means detects the face area from the reference through image . The digital camera according to claim 15, wherein
The digital camera according to claim 1, wherein the red-eye reduction unit performs a red-eye reduction process on the through image composite image based on the position information .

Images