JP4562182B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

The present invention relates to an image processing apparatus , an image processing method, and a program for correcting image blur due to camera shake or the like by combining a plurality of image data obtained by a plurality of imaging operations.

  With conventional cameras, many of the important tasks for shooting such as determining exposure and focusing are automated, and the possibility of making a shooting mistake is very low even for a person who is unskilled in the operation of the camera. Recently, a system (anti-vibration system) that suppresses image blur caused by camera shake applied to the camera has been studied, and factors that cause a photographer to take a picture are further reduced.

  Here, the vibration isolation system will be briefly described. It is said that the hand shake applied to the camera at the time of shooting is a vibration having a frequency of usually 1 Hz to several tens Hz. In order to obtain an image (photographed image) with no image shake even when the above-mentioned camera shake occurs during exposure, it is necessary to detect camera vibration due to camera shake and correct the change in the optical axis of the imaged light beam according to this detection result. There is.

  When detecting camera vibration, a vibration detection unit is used that detects acceleration, angular acceleration, angular velocity, angular displacement, etc. by an accelerometer, vibration gyroscope, laser gyroscope, etc., and appropriately calculates the detection result. .

  Further, as a means for correcting the change in the optical axis, there is one using a correction lens that is arranged in a photographing optical system and is movable in a plane substantially orthogonal to the optical axis (optical vibration isolation system).

  On the other hand, by repeatedly performing the imaging operation multiple times with an exposure time that does not cause image blurring, and performing composition processing while correcting the shift between images for the image data obtained by these imaging operations, There is a method (an electronic image stabilization system) for generating an exposure-corrected image (photographed image) (see, for example, Patent Document 1).

This electronic image stabilization system eliminates the need for a member for correcting image blur (a correction lens and a drive mechanism for driving the correction lens), and therefore has an advantage of being easily miniaturized compared to the optical image stabilization system. .
JP-A-9-261526 (paragraph numbers 0013, 0014, FIG. 1, etc.)

  In the electronic image stabilization system disclosed in Patent Document 1, the exposure time determined based on the subject brightness and the exposure time set by the photographer are divided into a plurality of times, and the imaging operation is performed in the divided time. Therefore, it is possible to suppress the occurrence of image blur as much as each exposure time is shortened.

  However, if the imaging operation is repeated a plurality of times, depending on the subject and the state of camera shake, the obtained plurality of images may include an image having a composition that is significantly different from that of the other images. A good image cannot be obtained even if images having greatly different compositions are combined. That is, the synthesized image is blurred by the amount of composition shift.

The image processing apparatus according to the first aspect of the present invention generates a composite image that has been subjected to exposure correction by performing image composition on a plurality of images, and includes a plurality of images obtained by a plurality of imaging operations. A selection unit that selects an image to be used for the image composition based on information about the image including at least information about the contrast of the image, and the number of images selected by the selection unit is the number of images necessary for the image composition. when insufficient for is to have a gain adjustment means for number of sheets up missing the gain of the selected image by the selection unit, the selection unit, the reference image and another image of the plurality of images When the contrast difference is equal to or greater than a predetermined value, the other image is not used for the image composition .

The image processing method according to the second invention of the present application is to generate a composite image that has been exposure-corrected by performing image composition on a plurality of images, and among the plurality of images obtained by a plurality of imaging operations. A selection step of selecting an image to be used for the image composition based on information on the image including at least information on the contrast of the image, and the selected image is insufficient with respect to the number of images necessary for the image composition. A gain adjustment step of increasing the gain of the selected image by an insufficient number , and when the difference in contrast between the reference image and other images among the plurality of images is greater than or equal to a predetermined value. The other image is not used for the image composition .

According to the first and second inventions of the present application, since an image to be used for image composition is selected based on information relating to an image including at least information relating to contrast obtained by an imaging operation, images that are not suitable for image composition are excluded. Thus, the image quality of the composite image can be improved.

  Examples of the present invention will be described below.

  FIG. 1 is a diagram illustrating a configuration of an imaging apparatus according to the present embodiment.

  A light beam (photographing light beam) incident on the photographing lens 11 from the subject reaches the image pickup surface of the image pickup device 19a through the shutter 12a after the light amount is adjusted by the diaphragm 13a. The image sensor 19a is configured by a semiconductor image sensor such as a CMOS sensor or a CCD sensor.

  In FIG. 1, the photographing lens 11 is represented as one lens, but the imaging apparatus of the present embodiment actually has a plurality of lens units. Here, the lens unit is composed of one or a plurality of lenses. At least one lens unit among the plurality of lens units is a lens unit (focus lens) for performing focus adjustment, and a lens unit (zoom) for changing the focal length of the photographing optical system is used for the other lens units. Lens).

  The focus lens receives the driving force from the AF drive motor 14a and moves in the direction of the optical axis 10 to adjust the focus in the photographing optical system. The AF drive motor 14a is driven by receiving a drive signal from the focus drive unit 14b.

  Further, the zoom lens stops in a predetermined zoom position by moving in the direction of the optical axis 10 in response to the driving force from the zoom drive motor 15a. Thereby, the focal length (shooting angle of view) of the taking optical system can be changed. The zoom drive motor 15a is driven by receiving a drive signal from the zoom drive unit 15b.

  The diaphragm 13a has a plurality of diaphragm blades, and these diaphragm blades operate by receiving a driving force from the diaphragm drive unit 13b, thereby changing an opening area (diaphragm aperture) serving as a light passage port. As a result, the amount of light incident on the image plane can be adjusted.

  The shutter 12a has a plurality of shutter blades, and these shutter blades receive an opening (a fixed opening provided in the shutter 12a) serving as a light passage when receiving a driving force from the shutter driver 12b. Open and close. Thereby, the light beam incident on the image plane is controlled.

  In addition, timing control such as charge accumulation time and reading in the image sensor 19a is performed by the sensor driving unit 19b.

  Further, the illumination unit 16a is driven (emitted) in response to a drive signal from the illumination drive unit 16b in accordance with shooting conditions (subject brightness, etc.) and settings by the photographer. The speaker 17a for informing the photographer of the photographing operation is driven (sounded) in response to a driving signal from the sounding driving unit 17b.

  Driving of the focus driving unit 14b, zoom driving unit 15b, aperture driving unit 13b, shutter driving unit 12b, illumination driving unit 16b, sound generation driving unit 17b, and sensor driving unit 19b described above is controlled by the imaging control unit 18.

  Operation signals from the release operation unit 12c, the aperture operation unit 13c, the zoom operation unit 15c, the illumination operation unit 16c, and the image stabilization operation unit 120 are input to the imaging control unit 18, and according to the input signal Control operation. That is, based on the operation signal from each operation part 12c-16c, the drive of the focus drive part 14b, the zoom drive part 15b, the aperture drive part 13b, the shutter drive part 12b, the illumination drive part 16b, and the sensor drive part 19b is controlled. .

  The aperture diameter of the aperture 13a and the light emission amount of the illumination unit 16a are automatically set in the camera (shooting control unit 18) based on shooting conditions (subject brightness, etc.). In order to set (aperture diameter and light emission amount), an aperture operation unit 13c and an illumination operation unit 16c are provided.

  The imaging control unit 18 measures the subject brightness (photometry) based on an image signal generated by the signal processing unit 111 described later, and based on the photometric result, the aperture diameter of the aperture 13a and the closing timing (exposure) of the shutter 12a. Time). Further, the imaging control unit 18 obtains the focus position of the focus lens based on the output from the signal processing unit 111 while moving the focus lens (imaging lens 11) in the optical axis direction via the focus driving unit 14b. Yes. That is, focus adjustment is performed by a so-called contrast detection method.

  A signal (analog signal) output from the image sensor 19 a is converted into a digital signal by the A / D converter 110 and input to the signal processing unit 111. The signal processing unit 111 generates a color video signal (image data) by performing signal processing for generating a luminance signal, a color signal, and the like on the input signal.

  The video signal generated by the signal processing unit 111 is input to the second image correction unit 117b via the signal switching unit 112. That is, when the image stabilization mode described later is not set, the video signal is input to the second image correction unit 117b.

  The second image correction unit 117b performs gamma correction processing and nonlinear level compression processing on the input video signal. The output signal of the second image correction unit 117b is displayed on the display unit 118 as a captured image, and is recorded on the recording unit 119 or is recorded on a recording medium (not shown) such as a memory card via the recording unit 119. Or

  In the image stabilization mode according to the present exemplary embodiment, a plurality of image data is generated by performing an imaging operation a plurality of times with an exposure time that is unlikely to cause image blur, and an image subjected to exposure correction by combining these image data. Data (photographed image) is generated. Here, by operating the image stabilization operation unit 120, it is possible to set the image stabilization mode or cancel the setting of the image stabilization mode.

  The operation when the image stabilization mode is set will be described below.

  A plurality of signals output from the image sensor 19 a by a plurality of imaging operations are converted into digital signals by the A / D converter 110 and then subjected to signal processing by the signal processing unit 111. Here, when a signal for instructing the image stabilization control unit 18 to set the image stabilization mode is output from the image stabilization operation unit 120, the image capture control unit 18 uses the image data generated by the signal processing unit 111 as a signal. The image data is input to the image storage unit 113 via the switching unit 112. In this case, image data is not output to the second image correction unit 117b.

  The image storage unit 113 stores image data that satisfies a predetermined condition among a plurality of image data generated by a plurality of imaging operations, as will be described later. Specifically, the imaging control unit 18 specifies image data suitable for image composition from among a plurality of image data obtained by a plurality of imaging operations, and stores only this image data in the image storage unit 113. Other image data is not output to the image storage unit 113 and the second image correction unit 117b.

  The deviation detection unit 114 detects a shift (motion vector) between the images in the horizontal direction and the vertical direction. This motion vector corresponds to image blur caused by camera shake, and represents the direction and amount of image blur. Specifically, a feature point in each image data stored in the image storage unit 113 is extracted, and a position coordinate of the feature point in the shooting screen is determined.

  For example, as shown in the frame 121a of FIG. 2, the operation of the deviation detection unit 114 will be described using a case where a person 122a photographs a subject standing against a building 123a.

  When performing the imaging operation a plurality of times, as shown in the frame 121b, an image whose composition is shifted from the frame 121a due to camera shake may be obtained.

  In the frame 121a, the shift detection unit 114 extracts an edge 125a of the window 124a, which is a point having a high luminance, from the building 123a located around the screen as a feature point by edge detection. Similarly, also in the frame 121b, the edge 125b of the window 124b is extracted as a feature point.

  Then, the feature point 125a of the frame 121a and the feature point 125b of the frame 121b are compared, and the frame 121b is corrected (coordinate conversion) by this difference (corresponding to the motion vector). That is, the frame 121a is used as a reference image, and coordinate conversion is performed on other images (such as the frame 121b) so that the composition of the reference image is aligned.

  In FIG. 2, the frame 121b is coordinate-converted by the amount of the arrow 126 so that the feature point 125b of the frame 121b overlaps the feature point 125a of the frame 121a. Here, an arrow 126 indicates a motion vector.

  In this embodiment, feature points are extracted from a peripheral area (an area close to the outer frame of the shooting screen) in the shooting screen. Hereinafter, this reason will be described.

  When shooting, the main subject is often located near the center of the shooting screen, and the main subject is often a person. In this case, extracting feature points from the area of the main subject may cause inconvenience due to subject shake.

  During the multiple imaging operations, not only the camera shake of the photographer but also the shake of the subject is superimposed, so using the feature points in the main subject area is based on the shake of the subject. Coordinate conversion processing may be performed.

  In this case, since coordinate conversion is performed based on the shake of the main subject, the composition of the main subject seems to be appropriate (a state in which there is almost no deviation). However, generally, the movement of a person is complicated, and the accuracy of detecting a shift between images is greatly influenced by the location where a feature point is selected.

  For example, when the eye of the main subject (person) is selected as a feature point, the actual shake of the entire subject differs from the effect of blinking. In addition, when the tip of the hand is selected as the feature point, the hand is easy to move, which is different from the actual shake of the entire subject.

  As described above, even if the coordinate conversion of the image data is performed based on the shift regarding one point (characteristic point) of the person, the whole person is not properly coordinate-converted. Even if the image data is synthesized after performing coordinate conversion based on a shake (deviation) different from the shake of the whole person, the person remains shifted in the synthesized image data.

  On the other hand, since a non-moving subject (stationary subject) such as a building is generally located in the peripheral area in the shooting screen, if a stationary subject is selected as a feature point extraction region, coordinate conversion processing is performed. There is no influence of subject shake. Therefore, in this embodiment, as described above, feature points are extracted from the peripheral area in the shooting screen.

  Note that the method for detecting the shift between images is not limited to the method of extracting one point in the peripheral region as a feature point and detecting the shift based on the feature point of each image as described above. For example, the entire region of each image is divided into a plurality of regions, and a motion vector between images in each region is detected. Then, a plurality of motion vectors can be grouped based on a predetermined condition, and the motion vector having the highest frequency in each group can be used as a gap between images.

  In the above description, the motion vector in the horizontal direction and the vertical direction of the image is detected, and the image data is shifted in the horizontal direction and the vertical direction by coordinate conversion based on the motion vector. Here, if image rotation information is also detected, image blur in the rotation direction can be corrected based on this detection information.

  The coordinate conversion unit 115 performs a coordinate conversion process on image data other than the reference image data based on the shift (motion vector) between images obtained by the shift detection unit 114. The coordinate conversion unit 115 also performs the following operations.

  First, a reference image serving as a reference when performing coordinate conversion processing is specified. This reference image data is any one of a plurality of compositing image data. Then, among the images other than the reference image, an image whose composition is shifted by a predetermined amount or more with respect to the reference image (an image having a motion vector greater than a predetermined value) is deleted. That is, the image to be deleted is not subjected to coordinate conversion processing and is not used for image composition processing.

  Each image data coordinate-converted by the coordinate conversion unit 115 is output to the image composition unit 116, and one image data (hereinafter also referred to as composite image data) is generated by the image composition processing of the image composition unit 116.

  In the case of a digital image, it is possible to correct exposure by increasing the gain even for an underexposed image. However, increasing the gain increases the noise and results in a low-quality image.

  However, when a plurality of image data is combined to increase the gain of the entire image as in the present embodiment, if there is a sufficient number of image data for combining, the entire signal level is compressed during the image combining process. By processing, the noise of each image data can be relatively reduced. As a result, image data with a good S / N ratio can be obtained, and as a result, noise can be suppressed and exposure can be optimized.

  In another way of thinking, for example, noise is allowed and the image pickup device 19a is set to high sensitivity, and then a plurality of image pickup operations are performed. Random noise is reduced.

  The composite image data is input to the first image correction unit 117a, and the first image correction unit 117a performs gamma correction processing and nonlinear level compression processing. In addition, when the coordinate-converted image data is combined, a region 129 where a plurality of images do not overlap each other is generated as shown in FIG. The size of this area 129 corresponds to the shift between images caused by camera shake. The first image correction unit 117a cuts the image data in the area 129.

  When the region 129 is cut, the region where a plurality of images overlap is smaller than the original image size. Therefore, the image data in this region is subjected to enlargement interpolation processing as necessary, so that the original The image size (aspect) is adjusted to (or a predetermined).

  Here, when it is not necessary to make the total number of pixels of the composite image equal to a predetermined number, it is not necessary to perform the enlargement interpolation process in order to prevent unnecessarily increasing the data amount. Further, when reproducing the composite image data, an enlargement interpolation process may be performed.

  The image data generated by the first image correction unit 117a is displayed as a captured image on the display unit 118 or recorded in the recording unit 119.

  In general, in the case of a digital camera, a photographed image is displayed on the liquid crystal monitor (display unit 118) after the photographing operation.

  However, when generating composite image data that has been subjected to exposure correction by combining a plurality of image data as in the present embodiment, it takes time to generate composite image data and display it on the display unit 118. .

  Therefore, in this embodiment, the image data for synthesis before image synthesis is displayed on the display unit 118. Thereby, it is possible to shorten the time from the completion of the photographing operation (multiple imaging operations) until the display of the image data on the display unit 118. Here, the composition image data to be displayed on the display unit 118 is determined in advance. For example, reference image data is displayed, or image data obtained by the first imaging operation among a plurality of imaging operations is displayed. It can be displayed.

  Since the display unit 118 is generally small, deterioration in image quality is not noticeable even when the image data for synthesis that is underexposed is displayed with the gain increased. The reason why the photographed image is observed on the display unit 118 is mainly to check the photographed composition, so that it is sufficient to display the image data for synthesis that shows the rough composition.

  On the other hand, if the image data recorded in the recording unit 119 is input to the display unit 118, the image data can be reproduced and displayed.

  As described above, when a plurality of image data is acquired by a plurality of imaging operations and these image data are combined, immediately after the shooting operation (corresponding to a plurality of imaging operations), a predetermined value is displayed on the display unit 118. Image data (compositing image data) is displayed. On the other hand, when the composite image data recorded by the recording unit 119 is reproduced and displayed on the display unit 118, the display prohibition unit 118a prohibits the display of the composite image data on the display unit 118 and displays the composite image data. This is displayed on the unit 118.

  Note that, when the reproduction mode for reproducing and displaying a photographed image is set, generally, all image data recorded in the recording unit 119 can be displayed on the display unit 118. That is, each image data can be displayed on the display unit 118, or a predetermined number of image data can be displayed together on the display unit 118.

  On the other hand, in this embodiment, when the playback mode is set, only the composite image data is displayed on the display unit 118. That is, the display prohibition unit 118a prohibits the display of the composition image data recorded in the recording unit 119. Note that the image data for synthesis may be displayed on the display unit 118 by a predetermined operation.

  If a plurality of compositing image data is displayed when setting the playback mode, it takes time to observe each image data, and the photographer is confused by image data different from the captured image data (composite image data). It may end up. For this reason, in this embodiment, only the composite image data as the captured image is displayed when the playback mode is set.

  Further, as described above, the image storage unit 113 stores only image data satisfying a predetermined condition among a plurality of image data obtained by a plurality of imaging operations. Here, all the image data may be stored in the image storage unit 113. However, as in the present embodiment, the storage capacity in the image storage unit 113 is ensured, or the time required for storing the image data. In addition, it is possible to shorten the time until generating the composite image data.

  Next, the characteristic part in a present Example is demonstrated using FIG. FIG. 4 shows a scene for photographing sunset, and shows images (1) to (7) generated by performing a plurality of imaging operations on the scenery 130 as a subject.

  A plurality of frames (for example, a frame 130a) shown in the scenery 130 indicate the outer frames of the images (1) to (7), and the frames and the images (1) to (7) are indicated by arrows. It corresponds.

  Here, it is assumed that the shutter speed (exposure time) determined based on the result of the photometric operation on the scene 130 is 1/10 msec. When the focal length of the photographing optical system is 50 mm (135 film equivalent), image blur due to camera shake may occur unless the shutter speed is set to 1/50 msec or less.

  Therefore, in the case described above, the shutter speed is set to 1/50 msec, and the imaging operation at the shutter speed is performed five times so that the total exposure time is 1/10 msec. Then, the image data generated by the five imaging operations is combined to generate one image data.

  In this embodiment, as shown in FIG. 4, seven image data are generated by seven imaging operations. Then, as described above, it is determined whether or not each image data is inappropriate for image composition, and only image data that can be used for image composition is stored in the image storage unit 113. Here, inappropriate image data that cannot be used for image composition will be described with reference to FIG.

  Obtained when another subject gets in between the main subject (main subject at the time of shooting) and the camera, or when image blur more than a predetermined amount occurs during multiple imaging operations. The image data is determined as inappropriate image data.

  In such inappropriate image data, there is a high possibility that various parameters are greatly different from the image data obtained first. Specifically, the contrast of the image data decreases due to subject shake during exposure, or the contrast decreases due to the other subject entering the nearer distance than the main subject being out of focus.

  Therefore, the contrast of the image data is evaluated, and when the evaluation value is outside the predetermined range, it is determined that the image data is inappropriate image data and is not stored in the image storage unit 113. Details of this processing will be described later. In this way, it is possible to determine whether or not the image data is inappropriate based on the contrast of the image data.

  On the other hand, it is possible to easily determine whether or not the image data is inappropriate based on the in-focus state of the photographing optical system. As described above, the shooting situation in which inappropriate image data is obtained is different from the focused state when a plurality of imaging operations are started. For this reason, it is possible to detect a change in the focus position of the photographing lens (focus lens) 11 using an encoder and determine whether or not the image data is inappropriate based on the detection result.

  In the image (3) of FIG. 4, a subject (vehicle) that moves at high speed during the photographing operation enters the photographing screen, and has a composition different from that of the other image data (1). That is, in the image (3), the subject shake occurs due to the vehicle moving at high speed. The contrast in the region where the subject shake occurs is likely to be lower than the contrast in the same region in other image data.

  Depending on the shape of the moving subject and the moving direction, the contrast may increase. For example, when a train colored in different colors on the top and bottom is running almost horizontally against a background with uniform brightness and the like, the contrast becomes high.

  Image (6) shows a case where a person enters the shooting screen at a close distance after the shooting operation is started with the focus set to infinity in order to shoot a distant landscape. A person outside the focus range (depth of field) is a so-called out-of-focus image, and the contrast of the image area corresponding to this person is the contrast in the same area in other images (1) and the like. Compared to the low possibility.

  Note that the contrast may increase depending on the brightness of the background and the person. For example, when the brightness is uniform and the background is dark, and illumination light with sufficient illuminance is applied to the person, the brightness of the person increases and the contrast also increases.

  As described above, when the images (3) and (6) are compared with another reference image (for example, the image (1)), the contrast may differ by a predetermined amount or more in some regions. Therefore, in this embodiment, the images (3) and (6) having a contrast difference of a predetermined amount or more with respect to the reference image are specified, and these images are not used for image synthesis. Images (3) and (6) are inappropriate images in image composition.

  The process of specifying image data not used for image composition based on the contrast difference is performed by the imaging control unit 18 as described above.

  In this embodiment, the images (1), (2), (4), (5), and (7) are stored in the image storage unit 113 among the images (1) to (7) shown in FIG.

  Here, when a predetermined number (five in the case shown in FIG. 4) of image data is stored in the image storage unit 113, the average value of contrast in the image data obtained by the second and subsequent imaging operations is calculated. Then, it can be confirmed that the contrast of the image data obtained by the first imaging operation is not less than a predetermined amount with respect to the average value. Thereby, the precision at the time of synthesize | combining the image data memorize | stored in the image memory | storage part 113 can be improved.

  Further, when the contrast value of the first image data is equal to or less than a predetermined amount with respect to the average value, the first image data can be deleted from the image storage unit 113. Thereby, the image quality of the composite image data can be further improved.

  On the other hand, all the image data (1) to (7) are stored in the image storage unit 113, an average value of contrast in all the image data is obtained, and an image having a contrast different from the average value by a predetermined amount or more. Data may not be used for image composition processing. Further, image data that is not used for the image composition process may be deleted from the image storage unit 113. In this case, the data capacity stored in the image storage unit 113 can be reduced.

  As described above, in the present embodiment, each time one imaging operation is completed, whether or not the image data is image data suitable for image composition processing based on the contrast of the image generated by this imaging operation. Is determined. It is not determined whether the first image data is suitable for image composition. If the image data is inappropriate for the image composition process, the image capturing operation is repeated until the necessary number of image data is obtained without storing the image data in the image storage unit 113.

  However, even if the imaging operation is repeated, depending on the imaging environment, the necessary number of image data suitable for the image composition process may not be acquired, and the imaging operation may not be completed.

  In this case, an upper limit can be set for the number of imaging operations and the time for performing the imaging operation (multiple imaging operations). For example, if only image data that is not suitable for image composition processing can be obtained even after repeated imaging operations, the imaging operation can be terminated when two minutes have elapsed since the start of the first imaging operation. . In addition, when the imaging operation is performed 20 times, the imaging operation can be terminated regardless of the number of image data stored in the image storage unit 113.

  On the other hand, instead of fixing the upper limit of the number of imaging operations, for example, the number necessary for image composition is set so that the number is twice the number necessary for image composition (depending on the shooting conditions). Depending on the total exposure time, the upper limit of the number of imaging operations may be changed.

  Here, when the necessary number of image data cannot be acquired due to the above-described limitation, the same effect as that of combining the necessary number of image data can be obtained by performing the image processing described below. it can. The number of image data (minimum necessary number) necessary for image composition can be obtained based on the luminance information of the image obtained by a plurality of imaging operations.

  For example, when the number of image data required for image composition is five and only four image data are stored in the image storage unit 113, each image data in the image storage unit 113 is The gain is increased so that the values of the luminance data and the color data become 5/4 times. Then, after performing coordinate conversion processing on each image data, image composition processing is performed.

  In the present embodiment, image data obtained by a plurality of imaging operations are combined after being subjected to coordinate conversion, so that fixed pattern noise unique to the sensor (imaging device) is randomized by image blur. For this reason, noise is less conspicuous than in the case where gain increase is performed on image data obtained by a photographing operation on a low-luminance subject.

  Here, the user may be able to selectively set a mode in which one of the reduction of the time required for a plurality of imaging operations and the improvement of the image data image quality (S / N ratio, etc.) is prioritized. That is, when priority is given to shortening the time, the mode is set to limit the number of times of imaging operations and the time for performing the imaging time, and when priority is given to image quality improvement, the mode for performing the gain increase described above can be set. it can.

  Next, an operation for specifying image data suitable for image composition based on the above-described contrast of image data and an operation for specifying image data suitable for image composition based on a motion vector will be specifically described.

  As shown in FIG. 1, the coordinate conversion unit 115 includes a shift unit 115a, a reference image setting unit 115b, and an added image pass / fail determination unit 115c. The signal processing unit 111 includes an image processing unit 111a and a contrast detection unit 111b.

  When a plurality of imaging operations are started, the image data generated by the first imaging operation is stored in the image storage unit 113.

  The contrast detection unit 111b detects the contrast of the image data generated by the image processing unit 111a.

  Here, the first imaging operation is performed after the photographer observes the composition and the focus state and examines the timing of the imaging start, so the image data generated by this imaging operation shows a good contrast. There is a high possibility. On the other hand, as described with reference to FIG. 4, the image data generated by the second and subsequent imaging operations may have a reduced contrast due to subject shake or defocusing.

  The low-contrast discriminating unit 18a, which is a part of the imaging control unit 18, calculates the contrast of the image data and the contrast of the first obtained image data each time image data is generated by the second and subsequent imaging operations. Compare. If the contrast difference is equal to or greater than a predetermined value, the image data compared with the first image data is not output to the image storage unit 113. Then, image data that is not output to the image storage unit 113 is deleted.

  Each image obtained by a plurality of imaging operations in the image stabilization mode has a lower contrast than an image obtained by a single imaging operation in which proper exposure is obtained. In this embodiment, since contrast comparison (comparison between low-contrast images) is performed between a plurality of images obtained by a plurality of imaging operations, it is determined whether the image is suitable for image synthesis. The discrimination can be performed accurately.

  The image data stored in the image storage unit 113 is sequentially sent to the deviation detection unit 114, and as described above, the deviation detection unit 114 obtains a motion vector of a feature point in another image with respect to a feature point in the reference image. . Here, the reference image is the first acquired image. Further, when the image data stored in the image storage unit 113 is output to the deviation detection unit 114, these image data are not input to the shift unit 115a.

  The reference image setting unit 115b determines image data serving as a reference in the image composition process from among a plurality of image data stored in the image storage unit 113, based on the detection result of the deviation detection unit 114. Specifically, any one of the plurality of images stored in the image storage unit 113 is set as a temporary reference image, and a deviation amount from other images is obtained. In this process, each image in the image storage unit 113 is used as a temporary reference image. Then, the temporary reference image when the amount of deviation between the temporary reference image and all other images is the smallest is set as the true reference image.

  The addition image acceptance / rejection determination unit 115c identifies another image having a motion vector equal to or greater than a predetermined value with respect to the reference image determined by the reference image setting unit 115b.

  When the above processing is completed, a plurality of image data in the image storage unit 113 are sequentially output to the shift unit 115a.

  The reference image setting unit 115b outputs information regarding the motion vector of another image with respect to the reference image determined by the reference image setting unit 115b to the shift unit 115a. Further, the addition image pass / fail determination unit 115c stores image data that is not suitable for image composition, that is, other image data (for example, the image (7) in FIG. 4) in which the motion vector with respect to the reference image is a predetermined value or more. From being input to the shift unit 115a.

  The shift unit 115a performs coordinate conversion on other image data so that other images are aligned with the reference image set by the reference image setting unit 115b. As described above, the image output to the shift unit 115a is a reference image and another image whose motion vector is equal to or less than a predetermined value with respect to the reference image, so that the coordinate conversion in the shift unit 115a can be accurately performed. it can.

  The image data coordinate-converted by the shift unit 115a and the reference image data are combined by the image combining unit 116.

  In addition, it is not based on the motion vector alone, but directly accesses the image data in the image storage unit 113 to perform further image analysis (analysis of color information and the like), and uses it for image composition processing based on the analysis result. It is also possible to specify image data to be recorded. For example, it is possible to detect changes in the subject by analyzing the color information of each image, and use this detection result to specify image data used for image composition. That is, it is possible not to use an image with greatly different color information for image composition.

  FIG. 5 is a flowchart showing a shooting operation in the image pickup apparatus of the present embodiment, and this flow starts when the power supply of the image pickup apparatus is turned on.

  In step S1001, the photographer waits until the SW1 is turned on by half-pressing the release button, and when SW1 is turned on, the process proceeds to step S1002.

  In step S1002, the imaging control unit 18 detects the contrast of the image data generated based on the output of the image sensor 19a while driving the AF motor 14a to move the imaging lens (focus lens) 11 in the optical axis direction. To do. Then, when the contrast becomes the highest, the driving of the photographing lens 11 is stopped. As a result, the photographing optical system is brought into focus. This focus adjustment method is a focus adjustment method based on a so-called contrast detection method.

  Since the imaging apparatus capable of performing focus adjustment by the contrast detection method has a function of determining the low contrast state, the operation of the low contrast determination unit 18a described above can be performed by using this function. It can be carried out.

  In step S1002, the luminance of the subject is measured (photometric) based on the image data. Based on the photometric value obtained by the photometric operation, the closing timing (exposure time) of the shutter 12a and the aperture diameter of the aperture 13a are set. Then, the diaphragm 13a is driven via the diaphragm driver 13b so that the set diaphragm aperture is obtained.

  In step S1003, it is determined whether or not the image stabilization mode is set by operating the image stabilization operation unit 120. If the image stabilization mode is set, the process proceeds to step S1004. If the image stabilization mode is not set, the process proceeds to step S1019.

  First, the operation when the image stabilization mode is set will be described.

  In step S1004, the number of imaging operations and the exposure time in each imaging operation are obtained based on the imaging conditions.

The imaging conditions here include the following four conditions.
1) The brightness of the subject. This is obtained by the photometric operation in step S1002.
2) Focal length of the photographic optical system. This is obtained based on the output of a detector that detects the position of the photographing lens 11 (zoom lens).
3) Brightness (aperture value) of the photographing optical system. This is preset for each imaging device (imaging optical system).
4) Sensitivity of the image sensor 19a. This is a preset value, and is set by the photographer's operation or automatically set.

  Here, for example, it is assumed that the sensitivity of the image sensor 19a is set to ISO200. Further, in order to perform appropriate exposure from the photometric result in step S1002, it is necessary to set the aperture 13a to fully open (for example, f2.8) and to set the exposure time to 1/8 msec.

  Under the above-described conditions, when the focal length of the photographing optical system is 30 mm in terms of 35 mm film, image blur due to camera shake may occur at an exposure time of 1/8 msec. Therefore, in this embodiment, the exposure time is set to 1/32 msec, and the imaging operation with this exposure time is set to be performed six times.

  Here, if the imaging operation with the exposure time of 1/32 msec is performed four times, the total exposure time becomes 1/8 msec, and the exposure is complemented by combining the four image data obtained by the four imaging operations. be able to. However, in the present embodiment, as described above, it is set so that the image pickup operation is performed twice as many times as possible in view of the generation of image data that is not suitable for image composition.

  On the other hand, when the focal length of the photographing optical system is 300 mm (35 mm film equivalent), the exposure time at which there is no risk of image blurring is set to 1/320 msec, and the imaging operation with this exposure time is performed 48 times. Set as follows.

  In this case, exposure can be complemented by performing 40 imaging operations and synthesizing 40 image data generated thereby. However, in this embodiment, it is assumed that image data not suitable for image composition is generated, and the imaging operation is performed by 20% (8 times) more than the number of imaging operations (40). Yes.

  As described above, as the number of imaging operations increases, the number of extra imaging operations is increased. This is because, in general, the frequency of image data that is not suitable for image synthesis increases as the number of imaging operations increases.

  As described above, in the present embodiment, when a plurality of imaging operations are performed (when the image stabilization mode is set), the exposure time and the number of imaging operations in each imaging operation are determined based on the imaging conditions described above. ing.

  Here, even when an imaging operation is performed a plurality of times on the same subject, more accurate image information can be obtained when the exposure condition in each imaging operation is closer to the condition for obtaining proper exposure. For this reason, when the subject is dark, the diaphragm 13a is narrowed down, or the imaging element 19a is set to low sensitivity, the exposure time in each imaging operation is set to be as long as possible. However, if the exposure time is too long, the effect of image blur due to camera shake occurs.

  Therefore, it is preferable to set the exposure time in each imaging operation to an exposure time that can suppress the above-described problems. Specifically, when the focal length of the photographing optical system is 30 mm (35 mm film equivalent), the exposure time can be set to 1/32 that is approximately one focal length. Further, when the focal length of the photographing optical system is 300 mm, the exposure time can be set to 1/320 msec.

  The imaging operation is performed a plurality of times as much as the exposure time in each imaging operation is insufficient.

  Here, when the focal length of the photographing optical system is longer than 300 mm, the exposure time is further shortened and the number of imaging operations is increased by the amount corresponding to the shortened exposure time in order to suppress the occurrence of image blur.

  That is, the exposure time in each imaging operation becomes longer as the subject or the photographing lens becomes darker and the sensitivity of the imaging element 19a becomes lower. Further, the longer the focal length of the photographing optical system, the shorter the exposure time. In this way, by changing the exposure time in each imaging operation according to the focal length of the photographing optical system, it is possible to obtain preferable image data at any focal length.

  On the other hand, the number of imaging operations decreases as the subject or the photographing lens becomes darker and the sensitivity of the imaging device 19a decreases. Further, the longer the focal length of the photographic optical system, the greater the number of imaging operations.

  As described above, when the number of imaging operations and the exposure time in each imaging operation are determined, the fact that the image stabilization mode is set in the optical viewfinder of the imaging device or the display unit 118 and the number of imaging operations are set. indicate. Here, the exposure time in each imaging operation may be displayed.

  In step S1005, steps SW1001 to S1005 are circulated and waited until SW2 is turned on by full depression of the release button. If SW2 is turned on, the process proceeds to step S1006.

  In step S1006, the first imaging operation is started. First, the charge accumulated in the image sensor 19a is reset, and charge accumulation is started again. Then, after waiting for the exposure time set in step S1004, the shutter 12a is closed to transfer charges.

  Further, when starting the first imaging operation, the speaker 17a is driven via the sound generation drive unit 17b to notify the photographer that the imaging operation (first imaging operation) has started. Here, the sound emitted from the speaker 17a may be, for example, an electronic sound “beep” or a voice. Further, the start of the photographing operation may be notified using the sound of the shutter 12a opening operation. Further, in the case of a single-lens reflex type camera, the start of the photographing operation may be notified using a sound that is raised by the mirror.

  In step S1007, the image data obtained by the first imaging operation is temporarily stored in the image storage unit 113.

  In step S1008, it is determined whether the imaging operation has been performed the number of times set in step S1004. Here, if all the imaging operations are not completed, the process returns to step S1006. When the first imaging operation is completed, the process returns to step S1006 to perform the second and subsequent imaging operations.

  In step S1007, image data having a contrast substantially equal to the contrast of the image data obtained by the first imaging operation among the image data obtained by the second and subsequent imaging operations is stored in the image storage unit 113. That is, the second and subsequent image data having a contrast that differs (lower) by a predetermined value or more compared to the contrast of the first image data is not stored in the image storage unit 113.

  For example, when seven image data (1) to (7) are acquired as shown in FIG. 4, the images (3) and (6) are not stored in the image storage unit 113, and the other five images are stored. It is stored in the image storage unit 113.

  If all the imaging operations are completed in step S1008, the process proceeds to step S1009.

  In step S1009, the speaker 17a is driven via the sound generation drive unit 17b to notify the photographer that the shooting operation (last image pickup operation) has been completed. Here, the sound emitted from the speaker 17a may be, for example, an electronic sound such as “beep” or a voice. Further, the completion of the photographing operation may be notified using the sound of the closing operation of the shutter 12a. Further, in the case of a single-lens reflex type camera, the completion of the photographing operation may be notified using a sound of the mirror going down.

  In the present embodiment, the number of times of sound generation when performing a plurality of imaging operations is made equal to the number of times of sound generation when performing one shooting operation. That is, when shooting in the image stabilization mode, an electronic sound is emitted from the speaker 17a once each when the first imaging operation is performed and after the last imaging operation is performed. In addition, when shooting in a mode other than the image stabilization mode, an electronic sound is emitted from the speaker 17a at the start and completion of the shooting operation. By making the number of electronic sounds emitted in this way equal, it is possible to prevent the photographer from feeling uncomfortable.

  In step S1010, the deviation detection unit 114 extracts a characteristic image (feature point) from the peripheral region of the image (for example, the building 123a in FIG. 2), and obtains coordinates of the image on the photographing screen. This coordinate indicates a relative shift coordinate of each image obtained by the second and subsequent imaging operations when the image obtained by the first imaging operation is used as a reference. Here, the above-described processing is performed on all the image data stored in the image storage unit 113 in step S1007.

  In step S1011, the coordinate conversion unit 115 performs coordinate conversion on each image data. Here, before performing coordinate conversion, an image serving as a reference for image composition is determined as described above, and an image suitable for image composition is determined for the reference image.

  In step S1012, a reference image is determined, and it is determined whether or not coordinate conversion processing has been completed for all image data suitable for image composition. If the coordinate conversion process has not been completed, the coordinate conversion process in S1011 is repeated.

  That is, in step S1011, after determining the reference image, coordinate conversion processing is performed on the other image data. Then, by rotating through steps S1012, S1011, coordinate conversion processing is sequentially performed on the other image data.

  On the other hand, if the coordinate conversion processing has been completed for all image data suitable for image composition, the process proceeds to step S1013, where the image data is synthesized.

  In step S1014, a region where the images do not overlap (region 129 in FIG. 3) due to image blur is cut, and the image data is subjected to diffusion interpolation so that the original (or predetermined) image size is obtained. In step S1015, the composite image data is subjected to gamma correction processing and nonlinear level compression processing.

  In step S1016, the composite image data obtained by the process of step S1015 is displayed on the display unit 118. In step S1017, the composite image data obtained by the processing in step S1015 is recorded in the recording unit 119 or recorded on a recording medium via the recording unit 119. Here, when recording on a recording medium, the image data is compressed by the JPEG method or the like as necessary.

  In step S1018, the process returns to the start.

  In step S1018, when SW1 remains in the ON state, each process is performed in the order of steps S1001, S1002, S1003, and S1004. If SW2 remains in the ON state in step S1018, the process stands by in step S1018 without returning to the start.

  Next, an operation when the image stabilization mode is not set will be described. In this case, the process proceeds from step S1003 to step S1019.

  In step S1019, it is determined whether there is a possibility of image blur due to camera shake based on the shooting conditions. The imaging conditions here are the same as the imaging conditions described above, and include the brightness of the subject, the brightness of the imaging lens, the sensitivity of the image sensor 19a, and the focal length of the imaging optical system.

  That is, the exposure time is obtained based on the brightness of the subject, the brightness of the photographing lens, and the sensitivity of the image sensor 19a, and there is a possibility that image blur may occur based on the relationship between the exposure time and the focal length of the photographing optical system. It is determined whether or not.

  If it is determined in step S1019 that image blur may occur, the process proceeds to step S1020. If not, the process proceeds to step S1021.

  In step S1020, a display indicating that setting of the image stabilization mode is recommended is displayed in the optical viewfinder of the imaging apparatus or on the display unit 118. Even when the image stabilization mode is not set in this way, if there is a shooting condition that may cause image blur, the photographer is urged to set the image stabilization mode so that image degradation due to image blur occurs. It can be prevented in advance.

  In step S1021, the process waits by circulating from step S1001 to step S1021 until SW2 is turned on. If SW2 is turned on, the process proceeds to step S1022.

  In step S1022, one imaging operation is performed. Here, the imaging operation is performed only for the exposure time obtained based on the photometry result in step S1002, that is, for the exposure time for obtaining a proper exposure by one exposure. If one imaging operation is completed, the process proceeds to step S1015.

  Although omitted in the flowchart of FIG. 5, when one imaging operation is performed, an electronic sound is emitted by driving the speaker 17 a when the imaging operation is started and when the imaging operation is completed. When performing one imaging operation or when performing a plurality of imaging operations, the number of times the electronic sound is emitted is the same, and it is possible to prevent the photographer from feeling uncomfortable. Note that the time from when the first electronic sound is emitted to when the second electronic sound is emitted differs between the imaging in the image stabilization mode and the imaging other than the image stabilization mode.

  The operations from step S1015 to step S1018 are the same as described above. However, these processes are performed not on the composite image data but on the image data obtained by one imaging operation.

In this embodiment, out of a plurality of images obtained by a plurality of imaging operations, images inappropriate for image composition are excluded based on the contrast of each image and a motion vector between images, and the remaining images are processed. Image composition is performed. Thus, it is possible to suppress deterioration of the image quality of the composite image by inappropriate images used for image synthesis, it is possible to improve the quality of the composite image.

  An imaging apparatus that is Embodiment 2 of the present invention will be described. Here, the same reference numerals are used for the same components as those described in the first embodiment.

  In this embodiment, when a plurality of imaging operations are performed, all of a plurality of image data obtained by these imaging operations are stored in the image storage unit 113. The contrast detector 111b detects the contrast in each image data and obtains the average value of the contrast in all the image data. Information regarding the average value of contrast is output to the low contrast determination unit 18a.

  The low contrast discriminating unit 18a compares the average value of contrast with the contrast of each image data, and specifies image data whose contrast difference is equal to or greater than a predetermined value. Then, information regarding the specified image data is output to the coordinate conversion unit 115. Image data whose difference with respect to the average value is a predetermined value or more is not used for image composition.

  Similar to the first embodiment, the addition image acceptance / rejection determination unit 115c performs control so that image data that is not suitable for image synthesis specified based on the motion vector is not input from the image storage unit 113 to the shift unit 115a. At this time, the image data specified by the low contrast determination unit 18a is not input from the image storage unit 113 to the shift unit 115a.

  Next, a shooting operation in the image pickup apparatus according to the present embodiment will be described with reference to a flowchart shown in FIG. In FIG. 6, the same reference numerals are used for the same processes as those described in the first embodiment (FIG. 5). Hereinafter, differences from the first embodiment will be described.

  In this embodiment, in step S1006 and step S1008, the imaging operation is repeated for the number of times determined in step S1004, and a plurality of image data obtained by these imaging operations is stored in the image storage unit 113.

  In step S2001, the contrast of all image data stored in the image storage unit 113 is detected, and an average value of these is obtained. Further, the average value is compared with the contrast of each image data, and image data having a contrast lower than the average value by a predetermined amount or more is specified.

Next , as described above, image data that is not suitable for image synthesis is specified based on the motion vector of each image data. Then, among the plurality of image data stored in the image storage unit 113, other than the image data not suitable for the image synthesizing identified based on the motion vector, and specific image data other than the image having a low contrast The data is image data for synthesis. In addition, a reference image is specified from the image data for synthesis (a plurality of image data), and coordinate conversion is performed on the other image data so that images other than the reference image are aligned with the reference image.

  Also in this embodiment, the same effect as that of Embodiment 1 can be obtained. In the first and second embodiments described above, the lens-integrated imaging device has been described. However, the present invention is also applied to an imaging system having an imaging device and a lens device that is detachably attached to the imaging device. be able to.

  A third embodiment of the present invention will be described. In the first embodiment, coordinate conversion processing and image composition processing for image data are performed in the imaging apparatus.

  In this embodiment, the imaging apparatus acquires a plurality of image data and determines image data suitable for image composition (first determination processing) among these image data. In the first determination process, as described in the first embodiment, it is determined whether the image data is suitable for image composition based on the contrast of each image data.

  On the other hand, discrimination of image data suitable for image synthesis (second discrimination processing), determination of reference image data, coordinate conversion processing of other image data with respect to the reference image data, and image synthesis processing are performed by an external device such as a personal computer. I am doing so. Here, as described in the first embodiment, the second determination process is image data suitable for image composition among the image data (image data suitable for image composition) specified in the first determination process. Whether or not there is is determined based on the motion vector.

  Note that at least one of the processes other than the process of acquiring a plurality of image data, which is performed by the imaging apparatus according to the first embodiment, may be performed by an external device. For example, the first determination process can be performed by an external device. In addition, the first and second determination processes can be performed by the imaging apparatus, and the coordinate conversion process for the image data specified by the first and second determination processes can be performed by an external device. Further, only one of the first and second determination processes may be performed.

  Image data obtained by the imaging operation (image data suitable for image composition processing) is recorded on a recording medium via the recording unit 119. Then, by connecting the recording medium to the external device, the image data in the recording medium can be stored in the memory in the external device. When image data is stored in the recording unit 119, the image data can be stored in a memory in the external device via wired or wireless.

  Here, in the recording medium, in addition to a plurality of image data obtained by a plurality of imaging operations, image data (appropriate exposure image data) obtained by one imaging operation may be recorded. is there. That is, in the recording medium, a plurality of image data used for image composition processing and image data that does not need to be subjected to image composition processing are mixed.

  In the state described above, a large number of image data is recorded on the recording medium, and when selecting the image data for synthesis from these image data, for example, the user selects the image data in the recording medium one by one. It must be confirmed, and the work of selecting is extremely troublesome.

  Therefore, in this embodiment, a plurality of compositing image data and image data that do not require image composition are recorded in the recording medium in a state where they can be identified.

  For example, when using the DCF (Design Rule For Camera File System) standard used for digital cameras, etc., it is synthesized in the header of the image data, and when compatibility is important, it is synthesized in the recording area of the vendor unique information. The identification information of the image data can be expressed and recorded with attributes and serial numbers.

  As a result, the image data for synthesis and the image data that does not require image synthesis can be easily identified, and the troublesome work for selecting the image data for synthesis can be saved.

  The application software attached to the camera recognizes the identification information described above, extracts only the composition image data, performs coordinate transformation on the extracted composition image data, and then performs image composition processing.

  The synthesized image data obtained by the image synthesizing process is stored in the memory in the external device, and each synthesized image data is deleted. Here, the composition image data becomes unnecessary when the image composition processing is completed, and if the composition image data is stored in the memory of the external device, the memory capacity of the memory may be insufficient.

  Therefore, by deleting the image data for synthesis as in the present embodiment, it is possible to suppress a shortage of storage capacity in the memory of the external device.

  FIG. 7 is a diagram showing a personal computer and a camera connected via a cable. That is, the notebook computer 33 is connected to the camera 31 via the USB cable 32. Thereby, the image data obtained by the camera 31 can be transmitted to the notebook computer 33 via the USB cable 32.

  Here, when image data is recorded in a recording medium that can be attached to and detached from the camera 31, the image data in the recording medium is removed by removing the recording medium from the camera 31 and inserting it into the slot 34 provided in the notebook computer 33. Can be stored in the notebook computer 33. The image data in the camera 31 can be transmitted to the notebook computer 33 by wireless communication.

  The notebook computer 33 is preinstalled with application software for performing the operation of the present embodiment. The operation of this application software will be briefly described below. Here, a case where the camera 31 is connected to the notebook computer 33 via the USB cable 32 will be described.

  First, compositing image data is identified from a plurality of image data recorded in the recording unit 119 in the camera 31, and the compositing image data is taken into the notebook computer 33 (memory). When all the image data for composition is transferred to the notebook computer 33, the image data for composition recorded in the recording unit 119 of the camera 31 is deleted.

  Next, reference image data is specified from among a plurality of image data for synthesis, and image data suitable for image synthesis is specified by the second determination process described above. Then, coordinate conversion processing is performed on the image data (image data other than the reference image data) specified in the second determination processing, and the image data subjected to the coordinate conversion processing and the reference image data are combined.

  Next, a gamma correction process or the like is performed on the composite image data, and thereafter, a non-overlapping area of the image is cut, and an enlargement interpolation process is performed for the amount of image size reduction. Then, the composite image data that has undergone the enlargement interpolation process is subjected to a compression process using the JPEG method or the like as necessary, and then stored in the memory. The composite image data stored in the memory can be transmitted to the recording unit 119 in the camera 31 via the USB cable 32.

  The above-described operation will be specifically described with reference to FIG.

As shown in FIG. 8, in the folder 35 created in the recording unit 119, the normal image folder 36 for storing image data that does not require image synthesis and the image data for synthesis specified by the first determination processing are stored. A composition image folder 37 to be stored is created.

  In the normal image folder 36, image data 39, 310, 311a, 312, 313a, and 314 are stored in the order of photographing. Here, one image data among a plurality of image data used for the image composition processing, that is, image data 311 a obtained by the first imaging operation is stored in the normal image folder 36. The file number of the image data 312 is not a sequential number with respect to the file number of the previous image data 311a, and image data corresponding to the file number 004 is not stored.

  The image data 313a is one image data (image data obtained by the first imaging operation) among a plurality of image data used for the image composition process. The file numbers of the image data 313a and the next image data 314 are not serial numbers, and the image data corresponding to the file number 0007 is not stored.

  The composition image folder 37 stores all image data used for the image composition process, that is, image data suitable for the image composition process specified in the first determination process. Here, the image data 311a to 311e are image data generated by the imaging operation in the first image stabilization mode, and the image data 313a to 313d are generated by the imaging operation in the second image stabilization mode. Image data.

  In the camera 31 of the present embodiment, as in the first and second embodiments, the contrast in specific image data or the average value of the contrast in all image data is used among image data generated by a plurality of imaging operations. Thus, image data that is not suitable for image composition processing is specified. The image data is not recorded in the recording unit 119.

  By not recording image data that is not suitable for image composition processing in the recording unit 119 in this way, it is possible to prevent useless image data from being recorded, to secure a recording capacity in the recording unit 119, and to record image data. The time required can be shortened.

  The camera 31 transfers the composition image folder 37 to the notebook computer 33. In the notebook computer 33, one composite image data is generated using the image data 311a to 311e, and one composite image data is generated using the image data 313a to 313d. Here, when generating the composite image data, a reference image is specified, and an image that is not suitable for the image composition processing with respect to the reference image is not used in the image composition processing.

  The two composite image data generated by the image composition processing described above is transmitted from the notebook computer 33 to the camera 31 and stored in the normal image folder 36 provided in the recording unit 119 in the order of photographing. Here, the composite image data is not stored in the memory of the notebook computer 33.

  In the normal image folder 36, composite image data 315 and 316 are stored in addition to the image data stored in advance. The respective composite image data 315 and 316 are stored so as to be positioned next to the composite image data 311a and 313a corresponding to the composite image data. The file number of the composite image data 315 is “004”, and the file number of the composite image data 316 is “007”.

  Here, by storing the composite image data and one composite image data corresponding to the composite image data in the normal image folder 36, the photographer can compare the composite image data and the composite image data. . If at least one of the synthesized image data and the synthesized image data is unnecessary, it may be deleted.

  Next, the photographing operation in the image pickup apparatus of the present embodiment will be described with reference to the flowchart shown in FIG. In the flowchart of FIG. 9, each process from step S1010 to S1014 in the flowchart of FIG. 5 is omitted, and a process of step S3001 is provided instead. In FIG. 9, the same processes as the processes in each step shown in FIG. Hereinafter, differences from the operation shown in FIG. 5 will be mainly described.

  In step S1007, among the image data obtained by the imaging operation, image data having a contrast close to that of the reference image (for example, image data obtained by the first imaging operation) (the difference is equal to or less than a predetermined value). Only in the image storage unit 113. Here, as described in the second embodiment, all the image data may be stored in the image storage unit 113, and only the image data suitable for image composition among these image data may be recorded in the recording unit 119. .

  In step S3001, it is determined whether or not the image data is obtained by the first imaging operation among a plurality of imaging operations in the image stabilization mode. If the image data is the first image data, the process proceeds to step S1015. If not, the process skips steps S1015 and S1016 and proceeds to step S1017.

  The image data obtained by the second and subsequent imaging operations is recorded in the recording unit 119 in step S1017 without performing the processing in step S1015 and the like because the image composition processing and the like are performed in the external device.

  The first image data is displayed on the display unit 118 in step S1016 after the processing in step S1015, and is recorded in the recording unit 119 in step S1017. Here, by displaying the first image data on the display unit 118, the photographer can check the composition and the like. Further, by recording the first image data in the recording unit 119, it can be compared with the composite image data.

  The first image data is stored in the normal image folder 36 shown in FIG. 8, and this image data corresponds to the image data 311a and 313a in FIG. The first image data is also stored in the composition image folder.

  In this embodiment, the first image data is recorded in the recording unit 119 after performing the processes in steps S1015 and S1016, and the first image data is recorded in the recording unit 119 without performing the processes in steps S1015 and S1016. There is.

  When image data obtained by a plurality of imaging operations is recorded in the recording unit 119, these image data are stored in the composition image folder 37 shown in FIG.

  FIG. 10 is a flowchart showing image processing in the notebook computer 33 (see FIG. 7). This processing is performed by confirming that the composition image folder 37 exists in the recording unit 119 of the digital camera 31 connected to the notebook computer 33 or in the recording medium used by the digital camera 31 and connected to the notebook computer 33. The process starts when the notebook computer 33 recognizes.

  In step S3002, the composition image folder 37 in the recording unit 119 (or recording medium) is moved to the memory of the notebook computer 33. At this time, the composition image folder 37 in the recording unit 119 is deleted.

  Note that when a plurality of image data is generated by a plurality of imaging operations in the image stabilization mode, a composition image folder is automatically created in the recording unit 119 again. Further, instead of moving the composition image folder 37 in the recording unit 119 to the memory of the notebook computer 33, the composition image folder 37 may be copied and stored in the memory.

  In step S3003, a plurality of pieces of image data are sequentially read out from the composition image folder 37 stored in the memory of the notebook computer 33. The processing from step S3004 to step S3010 described below is processing for generating composite image data with appropriate exposure by combining a plurality of image data in a state in which a shift between images is corrected (coordinate conversion).

  In step S3004, feature points in each image data are extracted, and position coordinates of the feature points in the shooting screen are specified. It is also possible to divide the entire area of each image into a plurality of areas, obtain a motion vector in each corresponding area, and specify a motion vector having the highest frequency among these motion vectors as an image shift amount. .

  In step S3005, the image data with the smallest file number is selected as the reference image data from among the plurality of image data. For example, when there is image data obtained by the first imaging operation among a plurality of imaging operations, this image data becomes reference image data.

  In step S3005, the image data for synthesis is selected, and coordinate conversion is performed on the selected image data for synthesis. Specifically, coordinate conversion processing is performed on each image data for synthesis based on the amount of deviation of each image data for synthesis from the reference image data.

  In step S3006, the coordinate-converted image data for synthesis and the reference image data are synthesized. When combining a plurality of image data to increase the gain of the entire image, if there is a sufficient number of image data for synthesis, the signal level of the entire image is compressed in the course of the image compositing process. In addition, noise of each image data can be reduced. Thereby, it is possible to obtain composite image data with a good S / N ratio, and as a result, it is possible to suppress the noise and optimize the exposure. Further, the exposure can be further optimized by increasing the gain as necessary for the composite image data with reduced noise.

  For example, in the case where five pieces of image data for synthesis are required to generate composite image data, and there are only four pieces of image data suitable for image composition, 5/4 of the composite image data is obtained. The exposure can be corrected by increasing the gain twice.

  In step S3007, whether the processing from step S3003 to step S3006 has been performed on a plurality of image data (for example, image data 311a to 311e shown in FIG. 8) obtained by one imaging operation in the image stabilization mode. Determine whether or not. If processing for all image data is completed, the process advances to step S3008.

  In step S3008, the final generated composite image data is subjected to processing such as gamma correction, and a non-overlapping area of each image is cut, and two-dimensional processing such as enlargement interpolation processing is performed by the amount of image size reduction. Processing on the pixel plane is performed. In addition, compression processing based on the JPEG method or the like is performed as necessary to reduce the amount of digital data.

  In step S3009, the composite image data generated in step S3008 is stored in the normal image folder 36 (see FIG. 8) provided in the memory.

  In step S3010, it is determined whether or not the processing from steps S3003 to S3009 has been completed for all the image data stored in the memory of the notebook computer 33. For example, as shown in FIG. 8, in the case where the composition image data 311a to 311e and the composition image data 313a to 313d are stored in the composition image folder 37, only the composition image data 311a to 311e is stored. If the image composition process has been completed, the process returns to step S3003 to perform the composition image process on the remaining composition image data 313a to 313d. If image composition processing has been completed for the composition image data 311a to 311e and 313a to 313d, the process proceeds to step S3011.

  In step S3011, a display indicating that the image composition processing has been completed for all the composition image data stored in the composition image folder 37 is performed. Simultaneously with this display processing, all the image data for synthesis stored in the memory of the notebook personal computer 33 is deleted.

  In the present embodiment, the case where the digital camera 31 and the notebook computer 33 are used has been described. However, the present invention is not limited to the notebook computer 33, and a desktop type personal computer, a dedicated storage or a dedicated image processing apparatus is used even if it is not a personal computer. The same processing as in this embodiment can also be performed.

  Also in this embodiment, the same effect as that of the above-described embodiment can be obtained. That is, the image quality of a composite image can be improved by eliminating image data inappropriate for image composition and performing image composition.

  In the above-described embodiments, the case where a digital camera is used has been described. In addition, since it is configured to electronically correct image shake, it can also be applied to small portable electronic devices (for example, mobile phones).

  The operations in the above-described embodiments can be executed by a computer based on a program. This program can be stored in a computer-readable recording medium. Examples of the recording medium include a hard disk, CD-R, CD-ROM, flexible disk, DVD disk, optical disk, magneto-optical disk, magnetic tape, nonvolatile memory card, IC card, ROM, and RAM.

  In addition, when the computer executes the program code read from the storage medium, not only the operation of the above-described embodiment is realized, but also based on an instruction of the program code, it is executed by an OS running on the computer. Some or all of the example operations can be performed.

1 is a diagram illustrating a configuration of a camera that is Embodiment 1. FIG. The figure explaining the synthetic | combination method of a some image. The figure explaining the area | region where each image for a synthesis | combination does not overlap in a synthesized image. FIG. 3 is a diagram illustrating a method for selecting a composition image in the first embodiment. 3 is a flowchart illustrating a photographing operation of the camera that is Embodiment 1. 9 is a flowchart illustrating a shooting operation of a camera that is Embodiment 2. Explanatory drawing at the time of using the camera and personal computer which are Example 3 of this invention. FIG. 10 is a diagram illustrating a file state before image composition processing in the third embodiment. 9 is a flowchart illustrating a shooting operation of a camera according to a third embodiment. 9 is a flowchart showing image processing on a personal computer in Embodiment 3.

Explanation of symbols

18: Imaging control unit 18a: Low contrast discrimination unit 111: Signal processing unit 111a: Image processing unit 111b: Contrast detection unit 114: Deviation detection unit 115: Coordinate conversion unit 115a: Shift unit 115b: Reference image setting unit 115c: Addition image Pass / fail judgment unit 116: image composition unit

Claims (9)

In the case of generating an exposure-corrected composite image by performing image composition on a plurality of images, the present invention relates to an image including at least information related to the contrast of the images among a plurality of images obtained by a plurality of imaging operations. A selection unit for selecting an image to be used for the image composition based on information;
Gain adjustment means for increasing the number of images selected by the selection unit by an insufficient number when the number of images selected by the selection unit is insufficient with respect to the number of images required for the image composition; Yes, and
The selection unit is configured not to use the other image for the image composition when the difference in contrast between the reference image and the other image of the plurality of images is equal to or greater than a predetermined value. apparatus. 2. The image processing according to claim 1 , wherein the selection unit does not use, in the image synthesis, an image having a contrast having a difference equal to or greater than a predetermined value with respect to an average contrast value in the plurality of images. apparatus. The image processing according to claim 1, wherein the information about the image includes at least one of color information, focusing information at the time of image acquisition, and displacement information between images in addition to the contrast of the image. apparatus. The image processing apparatus according to claim 3 , wherein the selection unit does not use other images having a displacement amount greater than or equal to a predetermined value with respect to a reference image among the plurality of images for the image composition. . The image processing apparatus according to claim 4 , wherein the selection unit uses, as the reference image, an image that minimizes a displacement amount with respect to another image. The image processing apparatus, an image processing apparatus according to any one of claims 1 to 5, characterized in that an imaging apparatus including an imaging unit that performs imaging operation. The image processing apparatus according to claim 6 , wherein the imaging unit acquires more images than the number of images necessary for the image synthesis by an imaging operation. In the case of generating an exposure-corrected composite image by performing image composition on a plurality of images, the present invention relates to an image including at least information related to the contrast of the images among a plurality of images obtained by a plurality of imaging operations. A selection step of selecting an image used for the image synthesis based on information;
A gain adjustment step of increasing the selected image by an insufficient number of gains when the selected image is insufficient with respect to the number of images necessary for the image composition;
An image processing method , wherein, when the contrast difference between a reference image and another image among the plurality of images is equal to or greater than a predetermined value, the other image is not used for the image synthesis . A program for causing a computer to execute image processing,
The image processing is
In the case of generating an exposure-corrected composite image by performing image composition on a plurality of images, the present invention relates to an image including at least information related to the contrast of the images among a plurality of images obtained by a plurality of imaging operations. Select an image to be used for the image composition based on the information,
When the selected image is insufficient with respect to the number of images necessary for the image composition, the gain of the selected image is increased by an insufficient number ,
A program characterized in that , when the contrast difference between a reference image and another image among the plurality of images is equal to or greater than a predetermined value, the other image is not used for the image composition .

Images