JP5476900B2 - Image composition apparatus, image composition method, and program - Google Patents

Description

  The present invention relates to an image composition device, an image composition method, and a program.

Conventionally, a technique is known in which a composite image is generated by combining a subject image, a background image, and a frame, and the composite image is printed (see, for example, Patent Document 1).
Further, in recent years, a plurality of images are obtained by continuous shooting, main subject images are cut out from these images, and the plurality of cut out subject images are sequentially switched and displayed for reproduction and display as a moving image. Technology has appeared.

JP 2004-159158 A

  However, since the main purpose of the technique of the above-mentioned Patent Document 1 is printing, there is no idea of combining a plurality of clipped subject images with a background image or a frame image, and even if they are combined, the background of a still image In addition, there is a problem that only a subject image moves, resulting in a synthesized moving image with little change.

  Therefore, an object of the present invention is to provide an image composition device, an image composition method, and a program capable of generating a synthetic moving image rich in interesting and varied.

In order to solve the above problem, an image composition device according to claim 1 is provided.
Given consecutively a predetermined number of the imaging frame rate is captured, and the object obtaining means for obtaining a plurality of object image object is present in the background, the object from the acquired plurality of object images by the object obtaining means A subject cropping unit that extracts a subject region including a plurality of subject cropped images, a number calculation unit that calculates the number of the plurality of subject cropped images generated by the subject cropping unit, and the subject acquisition unit A background acquisition unit that acquires the same number of background images as the number of the plurality of subject cutout images calculated by the number calculation unit, and a background image that is generated by the subject cutout unit. synthesizing the plurality of background images obtained by the plurality of subject cut-out image and the background acquisition means Obtain a plurality of composite images Te, and synthesizing means for generating a synthesized moving image frame a plurality of synthetic images, comprising the.

The image composition device according to claim 2 is the image device according to claim 1,
And the object recording means for plural record composed subject cut-out image group from the subject cut-out means said plurality of subject cut-out image generated in, any one from among a plurality of subject cut-out image group recorded in the object recording unit It is characterized in designating means for designating subject cut-out image group, further comprising a.

The image composition device according to claim 3 is the image composition device according to claim 1 or 2,
The image acquisition apparatus further includes an imaging frame rate acquisition unit that acquires an imaging frame rate when the subject acquisition unit acquires a plurality of subject images, and the background acquisition unit is the same as the imaging frame rate acquired by the imaging frame rate acquisition unit It is characterized by acquiring a background image at a frame rate.

The image composition device according to claim 4 is the image composition device according to any one of claims 1 to 3,
The image acquisition means includes the background acquisition means for acquiring at least the same number of background images as the number of the subject cut-out images generated by the subject cut-out means by the image pickup by the image pickup means.

The image composition method of the invention described in claim 5 is:
Given consecutively a predetermined number of the imaging frame rate is captured, and the object acquisition step of acquiring a plurality of object image object is present in the background, the object from the acquired plurality of object images by the object acquisition step A subject cropping step for generating a plurality of subject cropped images, a number calculation step for calculating the number of the plurality of subject cropped images generated in the subject cropping step, and a subject acquisition step. and background acquiring many as the number of the plurality of subject cut-out image calculated by the number calculation step different background image more acquired background plurality of subject images were present was in, generated in the subject cutting-out step the background acquisition step and the plurality of subject cut-out image Thus to obtain a plurality of combined images by combining a plurality of background images acquired, is characterized in that it comprises a synthetic step of generating a synthesized moving image as a frame the plurality of composite images, the.

The program of the invention described in claim 6 is:
The computer of the image synthesizer, is continuously captured predetermined number at a predetermined imaging frame rate, the subject obtaining means for obtaining a plurality of object image object is present in the background, the plurality obtained by the object obtaining means A subject region including the subject from the subject image and generating a plurality of subject cropped images; a number calculating unit for calculating the number of the plurality of subject cropped images generated by the subject cropping unit; number as many acquisition background acquisition means of the object acquisition means is acquired Ri by the a plurality of subject cut-out image calculated by the number calculator means different background image as a background in which the plurality of subject images is present, the subject obtained by the plurality of subject cut-out image and the background acquisition unit generated by the cutout section Obtain a plurality of combined images by combining the number of the background image, it is characterized in that to function the plurality of composite images synthesizing means for generating a synthesized moving image as a frame as.

  ADVANTAGE OF THE INVENTION According to this invention, the synthetic | combination moving image rich in the change with high interest property can be produced | generated.

It is a block diagram which shows schematic structure of the imaging device of one Embodiment to which this invention is applied. It is a flowchart which shows an example of the operation | movement which concerns on a subject clipping process. FIG. 3 is a flowchart showing a continuation of the subject clipping process in FIG. 2. FIG. It is a figure which shows typically an example of the image which concerns on a subject clipping process. It is a figure which shows an example of a to-be-photographed object image. It is a flowchart which shows an example of the operation | movement which concerns on a synthesized image generation process. It is a figure which shows an example of the background image which concerns on a synthetic image generation process. It is a flowchart which shows an example of the operation | movement which concerns on the image composition process in a composite image generation process. It is a figure which shows an example of the image for demonstrating an image composition process. It is a figure which shows an example of the synthesized image which concerns on an image synthesis process.

Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. However, the scope of the invention is not limited to the illustrated examples.
FIG. 1 is a block diagram illustrating a schematic configuration of an imaging apparatus 100 according to an embodiment to which the present invention is applied.
The imaging apparatus 100 according to the present embodiment captures the same number of background images D1 to Dn (see FIG. 7) as the number of subject cutout images (subject images) C1 to Cn (see FIG. 5), and background images D1 to D1. Each of Dn and each of the subject cutout images C1 to Cn are combined (see FIG. 9) to generate a combined moving image M (see FIG. 10).
Specifically, as illustrated in FIG. 1, the imaging apparatus 100 includes a lens unit 1, an electronic imaging unit 2, an imaging control unit 3, an image data generation unit 4, an image memory 5, and a feature amount calculation unit. 6, a block matching unit 7, an image processing unit 8, a recording medium 9, a display control unit 10, a display unit 11, an operation input unit 12, and a CPU 13.
In addition, the imaging control unit 3, the feature amount calculation unit 6, the block matching unit 7, the image processing unit 8, and the CPU 13 are designed as, for example, a custom LSI 1A.

The lens unit 1 includes a plurality of lenses and includes a zoom lens, a focus lens, and the like.
Although not shown, the lens unit 1 includes a zoom drive unit that moves the zoom lens in the optical axis direction and a focus drive unit that moves the focus lens in the optical axis direction when imaging a subject. May be.

  The electronic imaging unit 2 is composed of an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-oxide Semiconductor), for example, and converts an optical image that has passed through various lenses of the lens unit 1 into a two-dimensional image signal. To do.

Although not shown, the imaging control unit 3 includes a timing generator, a driver, and the like. Then, the imaging control unit 3 scans and drives the electronic imaging unit 2 with a timing generator and a driver, converts the optical image into a two-dimensional image signal with the electronic imaging unit 2 every predetermined period, and the electronic imaging unit 2 Image frames are read out from the imaging area for each screen and output to the image data generation unit 4.
When live view image display is performed at the time of shooting, the imaging control unit 3 causes the electronic imaging unit 2 to continuously capture the subject S at a predetermined imaging frame rate, and each image frame from the electronic imaging unit 2. The image data generation unit 4 sequentially outputs the data.

  In addition, in the composite image generation process (described later), when capturing the background images D1 to Dn (see FIG. 7) combined with each of the subject cutout images C1 to Cn, the imaging control unit 3 performs the subject cutout image. Data related to the imaging frame rate and the number of imaging is read out from the Exif information attached to the C1 to Cn image files. Then, the imaging control unit 3 causes the electronic imaging unit 2 to capture the background images D1 to Dn under the same imaging conditions as the imaging frame rate and the number of imaging of the read subject clipped images C1 to Cn. For example, when the imaging frame rate data read from the Exif information attached to the image files of the subject cut-out images C1 to Cn is 10 fps and the number of captured images is 20, the imaging control unit 3 uses the imaging frame rate of 10 fps as the background. Twenty images D1 to Dn are captured by the electronic imaging unit 2.

  In addition, the imaging control unit 3 performs adjustment control of conditions when imaging a subject such as AF (automatic focusing process), AE (automatic exposure process), and AWB (automatic white balance).

The image data generation unit 4 appropriately adjusts the gain for each RGB color component with respect to the analog value signal of the image frame transferred from the electronic imaging unit 2, and then performs sample holding by a sample hold circuit (not shown). The digital signal is converted into digital data by a / D converter (not shown), color processing including pixel interpolation processing and γ correction processing is performed by a color process circuit (not shown), and then a digital luminance signal Y and color difference signal Cb , Cr (YUV data).
The luminance signal Y and the color difference signals Cb and Cr output from the color process circuit are DMA-transferred to an image memory 5 used as a buffer memory via a DMA controller (not shown).

  The image memory 5 is composed of, for example, a DRAM or the like, and temporarily stores data processed by the feature amount calculation unit 6, the block matching unit 7, the image processing unit 8, the CPU 13, and the like.

The feature amount calculation unit 6 performs feature extraction processing for extracting feature points from the subject non-existing image B on the basis of the subject non-existing image B where the subject S does not exist in the background (see FIG. 4B). Specifically, the feature amount calculation unit 6 selects a predetermined number (or a predetermined number or more) of highly featured block regions (feature points) based on the YUV data of the subject non-existing image B, and Is extracted as a template (for example, a square of 16 × 16 pixels).
Here, the feature extraction process is a process of selecting a feature having a high characteristic convenient for tracking from a large number of candidate blocks.

  The block matching unit 7 performs block matching processing for aligning each of the subject presence images A1 to An (see FIG. 4A) and the subject nonexistence image B. Specifically, the block matching unit 7 determines where in the subject existence images A1 to An the template extracted in the feature extraction process corresponds, that is, the template pixels in the subject existence images A1 to An. A position (corresponding region) where the values are optimally matched is searched for in each of the subject existing images A1 to An. Then, the optimum offset between the subject non-existing image B and the subject existing images A1 to An having the best evaluation values (for example, the sum of squared differences (SSD) and the sum of absolute differences (SAD)) of the pixel values. Is calculated as a motion vector of the template.

  The image processing unit 8 includes a cut-out image generation unit 8a, a subject acquisition unit 8b, a background acquisition unit 8c, an image synthesis unit 8d, a calculation unit 8e, and a synthesis control unit 8f.

Specifically, the cut-out image generation unit 8a includes an alignment unit, a subject area extraction unit, a position information generation unit, and the like (all not shown).
Based on the feature points extracted from the subject non-existing image B, the alignment unit calculates a coordinate conversion formula (projection transformation matrix) of each pixel of the subject existing images A1 to An with respect to the subject non-existing image B. Each subject existing image A1 to An is subjected to coordinate conversion according to the coordinate conversion formula, and aligned with the subject non-existing image B.
The subject region extraction unit generates difference information of each corresponding pixel between each of the subject presence images A1 to An and the subject non-existence image B aligned by the alignment unit, and uses the difference information as a reference for the subject A subject area including the subject S is extracted from each of the existing images A1 to An. Specifically, the subject region extraction unit applies a low-pass filter to each of the YUV data of the subject existing images A1 to An and each of the YUV data of the subject nonexistent image B to remove high frequency components of each image. Thereafter, the subject region extraction unit calculates a difference for each pixel corresponding to each of the subject existing images A1 to An and the subject non-existing image B to which the low-pass filter is applied, and generates a difference map. Subsequently, the subject region extraction unit binarizes the dissimilarity map relating to each pixel with a predetermined threshold, and then performs a contraction process to remove a region in which the dissimilarity is caused by fine noise or camera shake from the dissimilarity map. . After that, the subject region extraction unit performs a labeling process, removes a region below a predetermined value or a region other than the maximum region, identifies the largest island pattern as the subject region, and corrects the contraction Perform expansion treatment.
The position information generation unit specifies the position of the subject area extracted in each of the subject presence images A1 to An, and generates position information indicating the position of the subject area in each of the subject presence images A1 to An.
Here, the position information includes, for example, an alpha map. The alpha map is a weight for alpha blending an image of a subject area with respect to a predetermined background for each pixel of each subject existing image A1 to An. Is expressed as an alpha value (0 ≦ α ≦ 1).

  Then, based on the alpha map generated by the position information generation unit, the cut-out image generation unit 8a selects a pixel having an alpha value of 1 among each pixel of the subject existing images A1 to An as a predetermined single color image (illustrated). The image of the subject S is synthesized with a predetermined single color image so that the pixel data of each of the subject cut-out images C1 to Cn is generated so that the pixels having the alpha value of 0 are not transmitted. To do.

The subject acquisition unit 8b acquires a plurality of subject cutout images C1 to Cn in which the moving subject S is continuously captured.
That is, the subject acquisition unit 8b acquires a plurality of subject cut-out images C1 to Cn constituting one subject cut-out image group C desired by the user in the composite image generation process. Specifically, in the composite image generation process, any one subject cut out from a plurality of subject cut image groups displayed on the display screen of the display unit 11 based on a predetermined operation of the selection determination button 12b by the user. When an image group (for example, a subject cut-out image group C) is designated, the subject acquisition unit 8b reads out image data of the subject cut-out images C1 to Cn constituting the designated subject cut-out image group C from the recording medium 9. get. Here, the subject acquisition unit 8b acquires a plurality of subject cutout images C1 to Cn from which subject regions have been extracted from subject presence images A1 to An in which moving subjects S are continuously captured.
In addition, the subject acquisition unit 8b, when acquiring the image data of the subject clipped images C1 to Cn, captures the imaging frame rate and the number of images when continuously capturing the subject existing images A1 to An related to the subject clipped images C1 to Cn. Data related to (imaging conditions) is read and acquired from the Exif information attached to the image data of the subject cutout images C1 to Cn.

The background acquisition unit 8c acquires the same number of background images D1 to Dn (see FIG. 7) as the number of the subject cutout images C1 to Cn acquired by the subject acquisition unit 8b.
That is, in the composite image generation process, when an imaging instruction input for continuously capturing the background images D1 to Dn is made based on a predetermined operation of the shutter button 12a by the user, the CPU 13 causes the imaging control unit 3 to perform a subject acquisition unit. The background image D1 to Dn is continuously captured by the electronic imaging unit 2 under the same imaging conditions as the imaging frame rate and the number of imaging of the subject S acquired by 8b, and the background acquisition unit 8c performs the continuous imaging. Background images D1 to Dn are acquired.
Thereby, the number of the background images D1 to Dn acquired by the background acquisition unit 8c and the subject cut-out images C1 to Cn acquired by the subject acquisition unit 8b are the same. The imaging frame rates of the background images D1 to Dn and the subject cutout images C1 to Cn are the same rate.

The image combining unit 8d combines the plurality of subject cutout images C1 to Cn and the plurality of background images D1 to Dn to generate combined images M1 to Mn. That is, the image compositing unit 8d has a plurality of subject cut-out images C1 to Cn and a plurality of background images D1 to Dn, the first one (see FIG. 9A) and the second one (see FIG. 9B). ) And the third image (see FIG. 9C),... The Nth image (see FIG. 9D), and the images are combined in the order of image capturing.
Specifically, the image composition unit 8d transmits pixels having an alpha value of 0 among the pixels of the first background image D1, and pixels having an alpha value of 1 are those of the first subject cutout image C1. Overwrite with the pixel value of the corresponding pixel, and among the pixels of the background image D1, an image with the alpha value 0 <α <1 cut out from the subject area using 1's complement (1-α) ( After generating the background image × (1-α)), a value blended with a single background color is calculated when the subject cutout image C1 is generated using the one's complement (1-α) in the alpha map, The value is subtracted from the subject cutout image C1 and is combined with an image obtained by cutting out the subject area (background image × (1−α)).
The image composition unit 8d also performs composition processing similar to that described above for the second and subsequent subject clipped images C2 and background images D2. As a result, the image composition unit 8d performs composition processing on all the subject cut-out images C1 to Cn and the background images D1 to Dn, and generates a composite moving image M (see FIG. 10) composed of a plurality of composite images M1 to Mn. Generate.

  The calculation unit 8e calculates and acquires the number of subject regions that have been successfully extracted by the cutout image generation unit 8a among the subject presence images A1 to An. That is, the calculation unit 8e calculates the number of subject cutout images C1 to Cn.

The synthesis control unit 8f causes the image synthesis unit 8d to synthesize the background images D1 to Dn and the subject cutout images C1 to Cn to generate a synthesized moving image M.
Specifically, the composition control unit 8f uses the calculation unit 8e, for example, of the acquired number “20” of the subject presence images A1 to An to be “18” as the number of subject cut-out images that have successfully extracted the subject region. Is calculated, the same number of background images D1 to Dn as the number of subject cutout images “18” are acquired, and the background image D1 to Dn and the subject cutout images C1 to Cn are combined by the image composition unit 8d. A synthesized moving image M is generated.

The recording medium 9 is composed of, for example, a non-volatile memory (flash memory) or the like, and a plurality of subject cut images (subject cut images C1 to Cn and the like) encoded by an encoding unit (not shown) of the image processing unit 8. A plurality of subject cutout image groups each consisting of the above are recorded as one image file.
The recording medium 9 records image data of a plurality of background images D1 to Dn acquired by the background acquisition unit 8c.
Each image data of the subject cutout images C1 to Cn is associated with an alpha map generated by the position information generation unit of the cutout image generation unit 8a, and the extension of the image data of the subject cutout images C1 to Cn is “ .jpg ". Similarly, each image data of the background images D1 to Dn is stored with the extension of the image data of the background images D1 to Dn as “.jpg”.
Further, the image data of the subject cut-out images C1 to Cn is composed of an Exif format image file in which the imaging frame rate and the number of captured images when the subject S is continuously captured are attached as Exif information.
Here, the recording medium 9 records a plurality of subject cutout image groups including a plurality of subject cutout images C1 to Cn. The recording medium 9 records background images D1 to Dn.

The display control unit 10 performs control for reading display image data temporarily stored in the image memory 5 and displaying the read image data on the display unit 11.
Specifically, the display control unit 10 includes a VRAM, a VRAM controller, a digital video encoder, and the like. The digital video encoder periodically reads the luminance signal Y and the color difference signals Cb and Cr read from the image memory 5 and stored in the VRAM (not shown) under the control of the CPU 13 from the VRAM via the VRAM controller. Are read out, a video signal is generated based on these data, and is output to the display unit 11.

  The display unit 11 is, for example, a liquid crystal display device, and displays an image captured by the electronic imaging unit 2 based on a video signal from the display control unit 10 on a display screen. Specifically, the display unit 11 displays a live view image based on a plurality of image frames generated by imaging the subject by the lens unit 1, the electronic imaging unit 2, and the imaging control unit 3 in the imaging mode. The REC view image captured as the actual captured image is displayed.

  The operation input unit 12 is for performing a predetermined operation of the imaging apparatus 100. Specifically, the operation input unit 12 includes a shutter button 12a related to a subject shooting instruction, a selection instruction for an imaging mode, a function, and the like, a setting instruction for a reference composite position of a subject area in each of the subject cut-out images C1 to Cn, and display A selection determination button 12b related to a designation instruction for designating one subject cutout image group from among a plurality of subject cutout image groups displayed on the display screen of the unit 11, and a zoom button (not shown) related to a zoom amount adjustment instruction. ) And the like, and a predetermined operation signal is output to the CPU 13 in accordance with the operation of these buttons.

  The CPU 13 controls each part of the imaging device 100. Specifically, the CPU 13 performs various control operations in accordance with various processing programs (not shown) for the imaging apparatus 100.

Next, the subject clipping process performed by the imaging apparatus 100 will be described with reference to FIGS.
The subject clipping process is a process executed when a subject clipping mode is selected from a plurality of imaging modes displayed on the menu screen based on a predetermined operation of the selection determination button 12b by the user.
In the subject clipping process of the present embodiment, a continuous shot image (for example, imaging frame rate: 10 fps, number of images: 20 images) is selected as the type of the subject clipping process based on a predetermined operation of the selection determination button 12b by the user. The instruction will be described below.

  As shown in FIG. 2, first, the CPU 13 displays a live view image on the display control unit 10 based on a plurality of image frames generated by imaging a subject by the lens unit 1, the electronic imaging unit 2, and the imaging control unit 3. While displaying on the display screen of the display part 11, it superimposes on the said live view image, and displays the imaging instruction message of a to-be-photographed object on the display screen of the display part 11 (step S1).

Next, the CPU 13 determines whether or not an imaging instruction has been input based on a predetermined operation of the shutter button 12a by the user (step S2). If it is determined that an imaging instruction has been input (step S2; YES), the CPU 13 causes the imaging control unit 3 to adjust conditions such as the focus lens in-focus position, exposure conditions, and white balance, and the subject. Optical images of the presence images A1 to An are continuously captured by the electronic imaging unit 2 at an imaging frame rate of 10 fps, and each of the subject existing images A1 to An transferred from the electronic imaging unit 2 to the image data generation unit 4 is captured. YUV data of the image frame is generated (step S3). The YUV data of each image frame of the subject existing images A1 to An is temporarily stored in the image memory 5.
Further, the CPU 13 controls the imaging control unit 3 to maintain a state in which conditions such as a focus position, an exposure condition, and a white balance at the time of imaging the subject presence images A1 to An are fixed.

Then, the CPU 13 causes the display control unit 10 to display a live view image on the display screen of the display unit 11 based on a plurality of image frames generated by imaging the subject by the lens unit 1, the electronic imaging unit 2, and the imaging control unit 3. In addition to the display, the image of the semi-transparent display mode of the first subject presence image A1 and the imaging instruction message of the subject non-existence image B are superimposed on the live view image and displayed on the display screen of the display unit 11 ( Step S4).
Thereafter, the CPU 13 determines whether or not an imaging instruction has been input based on a predetermined operation of the shutter button 12a by the user (step S5). Then, after the user moves the subject S out of the angle of view or waits for the subject S to move, the camera position is set so that the subject non-existing image B overlaps the semi-transparent image of the subject existing image A1 by the user. Is determined, and it is determined that the shutter button 12a is operated for a predetermined time and an imaging instruction is input (step S5; YES), the CPU 13 causes the imaging control unit 3 to display the optical image of the subject non-existing image B as the subject exists. Based on the image frame of the subject non-existing image B that is captured by the electronic imaging unit 2 under the fixed conditions after the images A1 to An are captured and transferred from the electronic imaging unit 2 to the image data generation unit 4, YUV data of the existing image B is generated (step S6). Note that the YUV data of the subject non-existing image B is temporarily stored in the image memory 5.

Next, the CPU 13 uses the subject existence images A1 to An on the basis of the YUV data of the subject non-existence image B temporarily stored in the image memory 5 in the feature amount calculation unit 6, the block matching unit 7, and the image processing unit 8. A projection transformation matrix for projectively transforming each of the YUV data is calculated with a predetermined image transformation model (for example, a similarity transformation model or a joint transformation model) (step S7).
Specifically, the feature amount calculation unit 6 selects a predetermined number (or a predetermined number or more) of highly featured block regions (feature points) based on the YUV data of the subject non-existing image B, and Is extracted as a template. Then, the block matching unit 7 searches the image frames of the subject existing images A1 to An for positions where the pixel values of the template extracted by the feature extraction process are optimally matched, and determines the difference between the pixel values. An optimal offset between the subject non-existing image B and the subject existing images A1 to An having the best evaluation value is calculated as a motion vector of the template. Then, the alignment unit of the cut-out image generation unit 8a statistically calculates the entire motion vector based on the motion vectors of the plurality of templates calculated by the block matching unit 7, and performs feature point correspondence related to the motion vector. The projection transformation matrix of each of the subject existing images A1 to An is calculated.

Next, the CPU 13 causes the alignment unit of the cut-out image generation unit 8a to perform projective transformation of the subject existing images A1 to An based on the calculated projective conversion row examples, so that the YUV data of the subject existing images A1 to An is obtained. And the YUV data of the subject nonexistent image B are aligned (step S8).
Then, as shown in FIG. 3, the CPU 13 causes the subject region extraction unit of the cut-out image generation unit 8a to perform processing for extracting the subject region including the subject S from the subject existing images A1 to An (step S9).
Next, the CPU 13 causes the position information generation unit of the cutout image generation unit 8a to generate an alpha map indicating the position of the extracted subject region in each of the subject existing images A1 to An (step S10).

Thereafter, the CPU 13 causes the cutout image generation unit 8a to perform processing for generating image data of the subject cutout images C1 to Cn obtained by combining the image of the subject S with a predetermined single color image (step S11).
In other words, the cut-out image generation unit 8a includes the first subject existing image A1, the second subject existing image A2, the third subject existing image A3,... The Nth subject existing image An. Image data of the subject cut-out images C1 to Cn respectively synthesized with a predetermined single color image is generated in the order of image capturing.
Specifically, the cut-out image generation unit 8a reads the first subject existing image A1, the single color image, and the alpha map corresponding to the subject existing image A1, develops them in the image memory 5, and then extracts the subject existing image A1. For all pixels, a pixel with an alpha value of 0 (α = 0) is transmitted, and a pixel with an alpha value of 0 <α <1 (0 <α <1) is blended with a predetermined single color. For a pixel having an alpha value of 1 (α = 1), nothing is performed so as not to transmit a predetermined single color.
In addition, the cut-out image generation unit 8a performs the same processing for the second and subsequent subject existing images A2. As a result, cut-out image generation processing is performed for all the subject presence images A1 to An, and a plurality of subject cut-out images C1 to Cn are generated.

  Next, the CPU 13 causes the calculation unit 8e to calculate the number of subject cut-out images C1 to Cn, that is, the number of subject-existing images from which the subject area is extracted among the plurality of subject-existing images A1 to An. (Step S12).

Then, the CPU 13 associates each alpha map generated by the position information generation unit of the cutout image generation unit 8a with the image data of the subject cutout images C1 to Cn in a predetermined storage area of the recording medium 9 in one image file. And the imaging frame rate when the subject existing images A1 to An are captured and the number of extracted subject areas are stored as Exif information (step S13).
Thereby, the subject clipping process is completed. As a result, image data of the subject cutout images C1 to Cn in which the subject S (for example, a person) (see FIG. 4A) is extracted from the background is generated.

  Next, the composite image generation process by the imaging device 100 will be described with reference to FIGS.

  As shown in FIG. 6, first, the CPU 13 selects any one subject from among a plurality of subject cutout image groups displayed on the reproduction display screen of the display unit 11 based on a predetermined operation of the selection determination button 12b by the user. It is determined whether a cutout image group (for example, a subject cutout image group C) has been designated (step S20). When it is determined that the subject cutout image group C has been designated (step S20; YES), the CPU 13 reads out the image data of the subject cutout image group C from the recording medium 9, and obtains the subject cutout image group C as a subject. The part 8b is made to acquire (step S21).

  Next, the CPU 13 reads the imaging conditions (imaging frame rate and number of images) stored in association with the plurality of subject clipped images C1 to Cn constituting the acquired subject clipped image group C. Then, the CPU 13 causes the imaging control unit 3 to perform processing for setting the imaging conditions (step S22).

  Next, the CPU 13 causes the display control unit 10 to display a live view image on the display unit 11 based on a plurality of image frames generated by imaging the background by the lens unit 1, the electronic imaging unit 2, and the imaging control unit 3. Each of the images in the semi-transparent display mode of the subject cut-out images C1 to Cn is displayed on the display screen of the display unit 11 so as to be superimposed on the live view image (step S23). At this time, when the CPU 13 detects a predetermined operation of the selection determination button 12b by the user, the display position and size of each image in the semi-transparent display mode of the subject cut-out images C1 to Cn to be displayed superimposed on the live view image. The angle may be changed.

  Next, the CPU 13 updates the position on the live view image of the subject S included in the subject cutout images C1 to Cn displayed superimposed on the live view image, that is, the display position until an imaging instruction is input. The image data is stored in the image memory 5 (step S24). It should be noted that the subject S included in the subject cutout images C1 to Cn displayed at this time may display only the subject S in any one of the subject cutout images C1 to Cn, or the subject cutout. The subject S included in the images C1 to Cn may be displayed while being switched in order.

Next, the CPU 13 determines whether or not an imaging instruction has been input based on the full pressing operation of the shutter button 12a by the user (step S25). If it is determined that an imaging instruction has been input (step S25; YES), the CPU 13 sets the optical images of the background images D1 to Dn (see FIG. 7) in the imaging control unit 3 in step S22. An image is captured by the electronic imaging unit 2 under predetermined imaging conditions. Then, the CPU 13 causes the image data generation unit 4 to generate the YUV data of the background images D1 to Dn transferred from the electronic imaging unit 2 (Step S26). Thereafter, the CPU 13 temporarily stores the YUV data of the background images D1 to Dn in the image memory 5.
Thereby, for example, image data of the background images D1 to Dn in which the truck travels from the left side to the right side is generated.

  Subsequently, the CPU 13 synthesizes each of the background images D1 to Dn and each of the subject cutout images C1 to Cn, and performs an image synthesis process for generating a synthesized moving image M using the synthesized images M1 to Mn as a frame. 8d (step S27), the synthesized moving image M generated in step S27 is recorded on the recording medium 9 (step S28).

Here, the image composition processing will be described in detail with reference to FIGS.
In the image composition process, for each of the plurality of subject cutout images C1 to Cn and the plurality of background images D1 to Dn, the first image, the second image, the third image,... Composite processing is performed in the order of imaging. Further, the composite position at this time is the position stored in the image memory 5 in step S24, that is, the display position of the subject S included in the subject cut-out images C1 to Cn on the live view image immediately before the imaging instruction is input. To be determined.
For example, when the display position of the subject S included in the subject cutout image C1 is stored in the image memory 5, the display position is set as the combined position of the subject S included in the subject cutout image C1. Then, the subject S included in the subject cut-out images C2 to Cn is synthesized with the background images D2 to Dn using the synthesis position as a reference.
That is, as illustrated in FIG. 8, the image composition unit 8d includes the subject cutout image C1 and the first background image in which the display positions are stored from the subject cutout images C1 to Cn acquired by the subject acquisition unit 8b. The DUV YUV data is designated (step S30).

  Next, the image composition unit 8d reads the alpha map stored in association with the first subject cutout image C1 and develops it in the image memory 5 (step S31).

  Next, the image composition unit 8d designates any one pixel (for example, the pixel at the upper left corner) of the first background image D1 (step S32), and sets the alpha value of the alpha map for the pixel. Based on this, the process is branched (step S33). Specifically, the image composition unit 8d, for any one pixel of the background image D1, has a alpha value of 1 (step S33; α = 1), and the first subject cut-out image C1. The pixel value of the corresponding pixel is overwritten (step S34), and for the pixel with an alpha value of 0 <α <1 (step S35; 0 <α <1), the subject area is calculated using 1's complement (1-α). When the first subject cutout image C1 is generated using the 1's complement (1-α) in the alpha map after the image is cut out (background image × (1-α)) Is calculated by subtracting the value from the first subject cutout image C1 and combining it with the image obtained by cutting out the subject area (background image × (1-α)) (step S35). For pixels with a value of 0 (step 33; α = 0), the first sheet of background image D1 so as to transmit without doing anything.

Subsequently, the image composition unit 8d determines whether or not all the pixels of the first background image D1 have been processed (step S36).
If it is determined that all the pixels have not been processed (step S36; NO), the image composition unit 8d designates the next pixel as the processing target and moves the processing target to the pixel (step S36). S37), the process proceeds to step S33.
By repeating the above processing until it is determined in step S36 that all pixels have been processed (step S36; YES), the image composition unit d allows the first subject cutout image C1 and the first background to be displayed. A first synthesized image M1 constituting a synthesized moving image M synthesized with the image D1 is generated.

Next, the image composition unit 8d determines whether or not all of the plurality of subject cut-out images C1 to Cn have been subjected to composition processing (step S38).
Here, if it is determined that the composition processing has not been performed for all the images (step S38; NO), the image composition unit 8d determines that the next image (for example, the second subject cutout image C2 and two images) are to be processed. The eye background image D2) is designated, the processing target is moved to the image (step S39), and the process proceeds to step S31.
By repeating the above processing until it is determined in step S38 that all the images have been processed (step S38; YES), the image composition unit d allows each of the subject cutout images C1 to Cn and the background images D1 to Dn. Are combined with each other to generate composite images M1 to Mn.
Thereby, the image composition process is terminated. The composite images M1 to Mn generated in this way are displayed by sequentially switching from M1 → M2 → M3 →... → Mn, for example, a composition in which a person follows the track traveling from the left side to the right side. A moving image M is obtained.

Returning to FIG. 6, the CPU 13 stores a group of a plurality of image files in which the extensions of the image data of the composite images M1 to Mn are “.jpg” in a predetermined storage area of the recording medium 9 (step S1). S28).
The synthesized moving image M does not exist as a file itself, but exists as an aggregate of the synthesized images M1 to Mn. However, a moving image file composed of Motion JPEG using the synthesized images M1 to Mn as frames (frames) ( The extension is “.avi”) or a video file that is compression-encoded in an MPEG-compliant file format (extensions are “.mpg”, “.mov”, “.3g2”, “.3gp”, etc.) May be stored.
Thus, the composite image generation process is finished.

As described above, according to the imaging apparatus 100 of the present embodiment, the same number of background images D1 to Dn as the plurality of subject cutout images C1 to Cn are acquired, and each of the plurality of background images D1 to Dn and the plurality of subjects are acquired. Since each of the cut-out images C1 to Cn is combined with each other in the order of image pickup such as the first image, the second image, and the N-th image, the combined moving image M is generated. Movement occurs not only in the subject S related to Cn but also in the background related to the background images D1 to Dn, and it is possible to reduce a sense of incongruity that the background does not move as in the past. A synthesized moving image M can be generated.
In addition, since a plurality of background images D1 to Dn are acquired so as to have the same number as the plurality of subject cutout images C1 to Cn acquired by the subject acquisition unit 8b, the background images D1 to Dn and the subject cutout image C1 related to the image composition processing are acquired. The number of .about.Cn can be made uniform, and the image composition process can be performed appropriately. That is, for example, when any one of the background images D1 to Dn and the subject cut-out images C1 to Cn is insufficient, an image with a smaller composite number than the user-desired number according to the smaller image. It is possible to avoid compositing processing and complicated processing such as additional generation of images for the shortage later, and image compositing processing can be performed appropriately.

  Further, the background acquisition unit 8c acquires background images D1 to Dn captured under the same conditions as the imaging conditions of the plurality of subject cutout images C1 to Cn, and the recording medium 9 records the background images D1 to Dn. The imaging conditions of the background images D1 to Dn and the subject cutout images C1 to Cn related to the image composition process can be made uniform, and the image composition process can be performed appropriately. In particular, the background acquisition unit 8c acquires the same number of background images D1 to Dn as the plurality of subject cutout images C1 to Cn. And image composition processing can be performed appropriately.

  In addition, the background acquisition unit 8c can acquire the same number of background images D1 to Dn as the number of the subject cutout images C1 to Cn constituting one designated subject image group C, and is related to the image composition processing. Image composition processing can be appropriately performed by aligning the number of background images D1 to Dn and subject cutout images C1 to Cn.

  Further, the background acquisition unit 8c can acquire the background images D1 to Dn captured at the same imaging frame rate as the imaging frame rates of the plurality of subject cutout images C1 to Cn acquired by the subject acquisition unit 8b. The shooting intervals of the subject cut-out images C1 to Cn and the background images D1 to Dn can be matched, and a sense of incongruity can be further reduced to generate a synthetic moving image M rich in changes with high interest.

  Further, since the same number of background images D1 to Dn as the number of subject cutout images C1 to Cn can be acquired and the background image D1 to Dn and the subject cutout images C1 to Cn can be synthesized by the image composition unit 8d. Even when the extraction of the subject area fails, it is possible to avoid a situation where the number of subject cut-out images is insufficient with respect to the number of the plurality of background images D1 to Dn, and appropriate image composition processing can be performed. .

The present invention is not limited to the above-described embodiment, and various improvements and design changes may be made without departing from the spirit of the present invention.
For example, in the above embodiment, the background image is continuously captured so that the number of the subject images C1 to Cn acquired by the subject acquisition unit 8b is the same. However, the number of the subject images C1 to Cn can be arbitrarily changed as appropriate.
That is, for example, with respect to 20 subject cutout images C1 to C20, 10 background images captured continuously and 10 background images captured continuously separately from the 10 background images May be acquired, or may be performed 20 times by separate imaging processes that are not continuous.
According to the background image acquired in this manner, the background can be changed to a completely different background, and the synthesized moving image M can be generated as if the subject S related to the subject cutout images C1 to Cn moved instantaneously. More interesting.
Further, the two background images (for example, a red background image and a white background image) are displayed alternately with respect to the 20 subject cutout images C1 to C20. You may make it acquire 10 sets of images.
In the above embodiment, the same number of background images D1 to Dn as the number of subject existing images from which subject areas have been extracted are acquired. However, the present invention is not limited to this, and the same number as the number of acquired subject existing images. The background image may be acquired.

Further, in the above embodiment, in the first step of the composite image generation process, one subject cutout image group C is designated from among a plurality of subject cutout image groups stored in the recording medium 9. However, the present invention is not limited to this, and in the first step of the composite image generation process, a plurality of subject cut-out images C1 to Cn are acquired by continuously capturing the subject presence images A1 to An and the subject absence image B. You may do it.
According to this, imaging of the subject existing images A1 to An and imaging of the background images D1 to Dn are performed in a series of flows, and the subject cutout images C1 to Cn and the background images D1 to Dn can be combined. Therefore, it becomes possible to fix the imaging conditions when imaging the subject existing images A1 to An, and to perform quick image composition.
In addition, the subject presence images A1 to An may be captured by other imaging devices. In this case, if data related to the imaging frame rate and the number of imaging when the subject S is continuously captured is recorded in the Exif information attached to the image file, the calculation unit 8e reads the information to detect the presence of the subject. The number of images can be acquired.
The subject cut-out images C1 to Cn are not limited to those extracted from the subject presence images A1 to An imaged by the imaging device, but may be those recorded in the recording medium 9 in advance.

In the above embodiment, the functions of the subject acquisition unit, the background acquisition unit, and the synthesis unit are realized by driving the image processing unit 8 under the control of the CPU 13. However, the present invention is not limited to this, and a configuration realized by executing a predetermined program or the like by the CPU 13 may be adopted.
That is, a program including a subject acquisition processing routine, a background acquisition processing routine, and a synthesis processing routine is stored in a program memory (not shown) that stores the program. Then, the subject acquisition processing routine may cause the CPU 13 to function as subject acquisition means for acquiring a plurality of subject images in which the subject is continuously captured. In addition, the background acquisition processing routine may cause the CPU 13 to function as background acquisition means for acquiring the same number of background images as the plurality of subject images acquired by the subject acquisition processing routine. Further, the CPU 13 causes the synthesis processing routine to synthesize the plurality of subject images acquired by the subject acquisition processing routine and the plurality of background images acquired by the background acquisition processing routine, thereby obtaining a plurality of synthesized images. You may make it function as a synthetic | combination means which produces | generates the synthetic | combination moving image which used the image as the frame.

DESCRIPTION OF SYMBOLS 100 Imaging device 1 Lens part 2 Electronic imaging part 3 Imaging control part 8 Image processing part 8a Cutout image generation part 8b Subject acquisition part 8c Background acquisition part 8d Image composition part 8e Calculation part 8f Composition control part 12a Shutter button 12b Selection button 13 CPU

Claims (6)

Subject acquisition means for continuously capturing a predetermined number of images at a predetermined imaging frame rate and acquiring a plurality of subject images in which a subject exists in the background ;
Subject cutout means for extracting a subject area including the subject from the plurality of subject images acquired by the subject acquisition means and generating a plurality of subject cutout images;
A number calculation means for calculating the number of the plurality of subject cutout images generated by the subject cutout means;
Background acquisition means for acquiring a background image different from the background in which the plurality of subject images acquired by the subject acquisition means are the same as the number of the plurality of subject cut-out images calculated by the number calculation means ;
The plurality of subject cutout images generated by the subject cutout unit and the plurality of background images acquired by the background acquisition unit are combined to obtain a plurality of composite images, and the combined moving image using the plurality of composite images as frames. A synthesis means for generating
An image composition device comprising: Subject recording means for recording a plurality of subject cutout image groups composed of the plurality of subject cutout images generated by the subject cutout means;
Designating means for designating any one of the subject cutout image groups from among the plurality of subject cutout image groups recorded in the subject recording means;
Further comprising
2. The image synthesizing apparatus according to claim 1, wherein the background acquisition unit acquires the same number of background images as the number of a plurality of subject cutout images constituting one subject cutout image group designated by the designation unit. . An imaging frame rate acquisition unit that acquires an imaging frame rate when the subject acquisition unit acquires a plurality of subject images;
The image synthesizing apparatus according to claim 1, wherein the background acquisition unit acquires a background image at the same frame rate as the imaging frame rate acquired by the imaging frame rate acquisition unit. Comprising imaging means,
The background acquisition unit acquires at least the same number of background images as the number of the plurality of subject cutout images generated by the subject cutout unit by imaging by the imaging unit. The image synthesizing device according to item. A subject acquisition step of capturing a plurality of subject images in which a predetermined number of images are continuously captured at a predetermined imaging frame rate and the subject is present in the background ;
A subject cropping step of extracting a subject region including the subject from the plurality of subject images acquired by the subject acquisition step and generating a plurality of subject cropped images;
A number calculation step of calculating the number of the plurality of subject cutout images generated in the subject cutout step;
And background acquiring many as the number of the plurality of subject cut-out image calculated by the number calculation step different background image more acquired background plurality of subject images were present was the object acquisition step,
Wherein the subject cut-out step said plurality of subject cut-out image generated by by synthesizing the plurality of background images acquired by the background acquisition step to obtain a plurality of synthesized images, and a plurality of synthetic images and frame synthesis A composition step for generating a moving image;
An image composition method including: The computer of the image synthesizer
Subject acquisition means for acquiring a plurality of subject images in which a predetermined number of images are continuously captured at a predetermined imaging frame rate and the subject is present in the background ;
Subject cutout means for extracting a subject area including the subject from the plurality of subject images acquired by the subject acquisition means and generating a plurality of subject cutout images;
A number calculating unit for calculating the number of the plurality of subject cutout images generated by the subject cutting unit;
Number as many acquisition background acquisition means of the object acquisition means is acquired Ri by the a plurality of subject cut-out image calculated by the number calculator means different background image as a background in which the plurality of subject images is present,
The plurality of subject cutout images generated by the subject cutout unit and the plurality of background images acquired by the background acquisition unit are combined to obtain a plurality of composite images, and the combined moving image using the plurality of composite images as frames. A synthesis means for generating
A program characterized by functioning as

Images