JP5743769B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to a technique for generating a new image using images taken from multiple viewpoints.

  In a conventional camera, light rays pass through the camera lens and collect at a point on the digital sensor. The camera acquires the light intensity, but cannot obtain information about the direction in which the light enters.

  In recent years, there is a light field camera devised by Ren Ing (Non-Patent Document 1). In a light field camera, by arranging a plurality of microlenses between a main lens and a sensor, the intensity of all incident light rays can be acquired and the direction of the light rays can also be recorded. A technique called light field photography has been developed that uses such a camera to obtain an image in which each subject is in focus with a different depth by image processing after shooting.

  In light field photography, the direction and intensity of light passing through a plurality of positions in space (light field, hereinafter abbreviated as LF) are calculated from multi-viewpoint images. Then, using the LF information, an image when passing through the virtual optical system and forming an image on the virtual sensor is calculated. By appropriately setting the virtual optical system and the virtual sensor, the above-described focus adjustment after photographing can be performed. Hereinafter, the process of calculating the image obtained by the virtual sensor from the multi-viewpoint image is referred to as a refocus process.

  As an imaging device for acquiring LF, a Plenoptic Camera in which a microlens array is placed behind a main lens and a camera array in which small cameras are arranged are known. In any case, a multi-viewpoint image obtained by shooting the subject from different directions can be obtained by one shooting.

  As the refocus processing, it is known to project-convert the acquired multi-viewpoint image onto one virtual sensor and average the pixel values (Patent Document 1).

  In the refocus processing, the pixel values on the virtual sensor are averaged using the corresponding pixels of the multi-viewpoint image. Usually, a plurality of pixels of a multi-viewpoint image correspond to one pixel on the virtual sensor.

Japanese Patent Application 2008-541051

`` Light Field Photography with a Hand-held Plenoptic Camera '' by R.NG, M.Levoy, M.Bredif, G.Duval, M. Horowitz, P. Hanrahan (Stanford Tech Report CTSR 2005-02, 2005)

  However, in the method of Patent Document 1, the average is obtained using all the regions of the pixels on the virtual sensor corresponding to the pixels of the input image, and the pixel values of many images are averaged. There exists a subject that sharpness will fall. Details will be described later.

The image processing apparatus according to the present invention is an image processing apparatus for synthesizing multiple images taken by the multi-view, comprising: input means for inputting image data representing the plurality of images, on the sensor For each pixel of a plurality of pixels included in a projection area which is an area corresponding to one pixel and is projected onto a virtual sensor determined according to the position of the subject focused by the synthesis, Usage rate setting means for setting a usage rate for combining; and combining means for combining the plurality of images indicated by the image data based on the usage rate set in the usage rate setting means. It is characterized by.

  According to the present invention, it is possible to provide an image processing apparatus and an image processing method capable of synthesizing a sharp image from multi-viewpoint images.

It is a block diagram showing the apparatus structure of an image processing apparatus. It is a block diagram showing a signal processing part. It is a figure showing the structure of a normal imaging optical system. It is a figure showing the 1st example of a structure of an imaging part. It is a figure showing the 2nd example of a structure of an imaging part. 3 is a block diagram illustrating a configuration of an image composition unit in Embodiment 1. FIG. 3 is a flowchart illustrating processing in the first embodiment. The figure showing the correspondence between the pixel position of the input image and the pixel position in the composite image It is a figure showing the utilization factor of a projection area | region. It is a figure showing a response | compatibility of the pixel position of a several sheets input image, and the pixel position in a synthesized image. FIG. 10 is a block diagram illustrating a configuration of a pixel value synthesis unit according to a second embodiment. 10 is a flowchart illustrating processing in the second embodiment. FIG. 10 is a block diagram illustrating a configuration of a projection area usage rate setting unit in Embodiment 3. 10 is a flowchart illustrating processing in the third embodiment. It is a figure which shows the example of the effect in Example 1. FIG. FIG. 10 is an image diagram of pixel skip in Example 3.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, the constituent elements described in this embodiment are merely examples, and are not intended to limit the scope of the present invention.

  The present embodiment aims to improve the sharpness of a composite image.

<Camera system configuration>
FIG. 1 is a diagram illustrating an example of a configuration of an image processing apparatus according to the present exemplary embodiment.

  The imaging unit 101 includes a zoom lens, a focus lens, a shake correction lens, an aperture, a shutter, an optical low-pass filter, an iR cut filter, a color filter, and sensors such as CMOS and CCD, and detects the amount of light of the subject. The configuration of the imaging unit 101 will be described later.

  The A / D conversion unit 102 converts the amount of light of the subject into a digital value. The signal processing unit 103 performs white balance processing, gamma processing, noise reduction processing, and the like on the digital value to generate a digital image. The D / A conversion unit 104 performs analog conversion on the digital image. The encoder unit 105 performs processing for converting the digital image into a file format such as JPEG or MPEG. The media interface 106 is an interface for connecting a PC or other media (eg, hard disk, memory card, CF card, SD card, USB memory) to the image processing apparatus.

  The CPU 107 sequentially reads and interprets instructions stored in the ROM 108 and the RAM 109, and controls processing by each component according to the result. The ROM 108 and the RAM 109 provide the CPU 107 with programs, data, work areas, and the like necessary for the processing.

  The imaging system control unit 110 controls the imaging system in accordance with control commands from the CPU 107, such as focusing, opening a shutter, and adjusting an aperture.

  The operation unit 111 corresponds to a button, a mode dial, and the like, and receives a user instruction input via these buttons. An instruction for a refocus position when generating a composite image after image capture is also received.

The character generation 112 generates characters and graphics.
The display unit 113 is generally configured using a liquid crystal display. The display unit 113 displays captured images and characters received from the character generation unit 112 and the D / A conversion unit 102. Further, the display unit 113 may have a touch screen function, and in that case, an instruction from the user via the touch screen function can be handled as an input by the operation unit 111.

  In addition, although the component of an apparatus exists other than the above, since it is not the main point of this invention, description is abbreviate | omitted.

  In addition, the constituent elements of the present invention may be applied to a system composed of a plurality of devices or an apparatus composed of a single device. The object of the present invention can be achieved even when a computer (or CPU or MPU) of the system executes a program that realizes the functions of the above-described embodiments described later. In this case, the program itself realizes the functions of the above-described embodiments, and the storage medium storing the program constitutes the present invention. As a storage medium for supplying the program code, for example, a floppy disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a nonvolatile data storage unit, a ROM, or the like can be used. .

  In addition, the functions of the above-described and below-described embodiments are not only realized by executing the program read by the computer. It goes without saying that an OS running on a computer performs a part of the actual processing based on the instructions of the program, and the functions of the above-described and below-described embodiments are realized by the processing. Furthermore, it goes without saying that the instruction contents of the program are executed by a CPU or the like provided in the function expansion board of the system, and the processing of the above-described and later-described embodiments is realized by the processing.

  The image composition processing in the present embodiment is preferably executed by the signal processing unit 103.

  FIG. 2 shows the configuration of the signal processing unit 103. In the signal processing unit shown in FIG. 2, various processes are performed on the input digital image to improve the image quality. For example, edge enhancement processing for improving the sharpness of an image, color conversion processing for improving color reproducibility, and the like. The image composition processing of the present embodiment is preferably performed as one of various types of processing in the signal processing unit. In the configuration of the signal processing unit shown in FIG. 2, it is described that the image synthesis is processed before other processing, but the present invention is not limited to such configuration. For example, the image composition process may be performed after the noise reduction process. Details of the image composition unit 201 will be described later.

<Refocusing principle>
FIG. 3 shows the configuration of a normal imaging optical system. In FIG. 3, illustrations of configurations such as an iR cut filter, a zoom lens, and a diaphragm are omitted. Further, in FIG. 3, the detailed lens configuration is not shown, and only the main lens 303 is shown as a representative of the lens group.

  FIG. 3 shows the state of the imaging optical system when the focus is shifted. Light from the object point 301 is collected by the main lens 303 and reaches a partial region 305 of the sensor 302. At this time, the light condensed by the main lens 303 reaches the sensor 302 before forming an image at one point. For this reason, in the partial area 305 of the sensor 302, the image of the object point 301 is spread and recorded, and the captured image is blurred. In order to obtain an image with high sharpness, it is necessary to adjust the focus position and perform imaging again so that the object point 301 is focused on one point on the sensor 302.

  FIG. 4 shows a first example of the configuration of the imaging unit 101 in the present embodiment. In FIG. 4, the light from the object point 401 is collected by the main lens 403, but is recorded by the sensor 405 through the microlens array 406 before being imaged. Since the optical image on the sensor generated by one microlens is an image obtained by observing the object point 401 from different directions, the sensor 405 records a multi-viewpoint image as a single image. Here, the pixel value of the pixel 412 on the sensor 405 is recorded according to the intensity of the light beam 411. When the light intensity in the virtual sensor I407 and the light intensity in the virtual sensor II408 are averaged when the light beam group that has passed through the microlens array 406 is extended, an image recorded in each virtual sensor can be obtained by calculation. When the image at the virtual sensor I407 is calculated, the light at the object point 401 spreads and an image with incorrect focus adjustment as shown in FIG. 3 is obtained. When the image of the virtual sensor II 408 is calculated, the light emitted from the object point 401 gathers at one point, and an in-focus image is obtained.

  FIG. 5 shows a second example of the configuration of the imaging unit 101. In FIG. 5, the imaging unit 101 includes a plurality of imaging units. Light from the object point 501 is recorded by a plurality of sensors of the imaging unit. For example, 502 pixels on the sensor record the intensity of the light beam 503. Considering the case where a virtual sensor is placed on the object side, the light beam group is extended in the direction of the virtual sensor, and the light intensity is averaged in the virtual sensor. Then, the image calculated by the virtual sensor I508 is an image in which the object point 501 is blurred, and the image calculated by the virtual sensor II 509 is an image in which the object point 501 is in focus.

  As described above, in both the first example and the second example of the imaging unit 101, a multi-viewpoint image obtained by photographing the subject from a plurality of different angles can be obtained. The refocus processing is processing for calculating light received by the virtual sensor based on the light ray information obtained from the multi-viewpoint image in both the first and second examples. At that time, adjusting the position of the virtual sensor and calculating it corresponds to adjusting the focus position. The above is the principle of refocus for obtaining an image whose focus position is adjusted after imaging.

<Configuration and processing of image composition unit>
FIG. 6 is a block diagram showing a detailed configuration of the image composition unit 201. 6 will be described with reference to the flowchart shown in FIG. As the overall operation of the image composition unit 201, the image composition unit 201 sequentially acquires input images from the A / D conversion unit 102, and outputs the pixel values to the noise reduction processing unit as soon as the calculation of the pixel values of the composite image is completed. To do.

Hereinafter, in order to simplify the description, the input image is described as one-dimensional.
First, in step S701, the input pixel acquisition unit 603 acquires an input pixel value and its pixel position on the input image from the memory 602 and the bus. The memory 602 temporarily stores the input image. The stored input image is read out as necessary when the input pixel acquisition unit 603 and the pixel value synthesis unit 607 perform processing.

  In step S702, the virtual sensor position acquisition unit 604 acquires the refocus position from the bus, and acquires the virtual sensor position by referring to the ROM 108 using the refocus position.

  Here, the refocus position is input to the virtual sensor position acquisition unit 604 via the operation unit 111 and the bus in accordance with an instruction from the user. The virtual sensor position is stored in the ROM 108 by obtaining a position at which a subject existing at a certain focus distance forms a clear image as a virtual sensor position by optical simulation in advance. The virtual sensor position acquisition unit 604 acquires the virtual sensor position by appropriately referring to the ROM 108 using the refocus position.

In this embodiment, the distance between the sensor 405 that records the input image and the microlens is indicated as f _real, and the distance between the virtual sensor (407 or 408) that records the refocused image and the microlens is indicated as f _virtual . Since the virtual sensor position varies depending on the refocus position, f _virtual is set according to the refocus position.

  In step s703, the projection area endpoint calculation unit 605 calculates the projection area endpoint on the virtual sensor using the input pixel, the input pixel position, and the virtual sensor position. In FIG. 8A, the end points 806 of the projection area in this embodiment are indicated by P1 and P2. A method for obtaining the coordinates of P1 and P2 will be described later in the detailed description of the endpoint calculation unit 605 of the projection area. The projection area is an area on the virtual sensor corresponding to the sensor pixel, and is an area surrounded by the end points of the projection area. FIG. 8A shows a projection area 808 in this embodiment. The projection area projected on the virtual sensor 803 in an enlarged manner is equally divided into the same size as the sensor pixel 804. Referring to FIG. 8B, it can be seen that there are a plurality of pixels in the projection region 808.

  That is, the projection area is an area corresponding to one pixel on the sensor, and is an area projected on a virtual sensor determined according to a position focused by synthesis (refocus position). The projection area includes a plurality of pixels.

  In step s704, the projection area usage rate setting unit 606 sets the usage ratio of the projection area on the virtual sensor used for image synthesis. In the conventional method, the entire projection region is used for image synthesis. In this embodiment, a usage rate is set for each pixel in the projection area, and image composition is performed in accordance with the usage rate. As a result, according to the image composition according to the present embodiment, it is possible to reduce a reduction in image sharpness caused by averaging pixel values of a plurality of images.

  Although details will be described later, in the processing of the present embodiment, the usage rate of the projection area is controlled by multiplying the pixels of the projection area by a coefficient of 0 to 1. The coefficient is called a projection area usage rate, and how to determine the coefficient will be described later in the detailed description of the projection area usage rate setting unit 606.

  In step s705, the pixel value composition unit 607 multiplies the pixels in the projection area by the projection area usage rate for a plurality of input images, and averages the pixels that overlap the pixel positions in the composite image to obtain the pixel values of the composite image. Calculate and output. Since there are a plurality of input image pixels, there is a projection area corresponding to each input pixel. For this reason, pixels in a plurality of projection regions overlap with pixel positions of the composite image. The pixel value of the composite image is obtained by multiplying the pixel value in the projection area of the plurality of input pixels by the corresponding projection area usage rate and averaging the pixel values of the plurality of pixels overlapping the pixel position of the composite image. Desired.

<Detailed Operation of Projection Area Endpoint Calculation Unit>
A detailed operation of the projection area end point calculation unit 605 will be described with reference to FIG.
In FIG. 8A, reference numeral 801 represents a subject (object point), reference numeral 802 represents a sensor, and reference numeral 803 represents a virtual sensor. Reference numeral 804 represents one pixel on the sensor 802, and reference numeral 805 represents an optical axis of the microlens. Reference numeral 806 represents the end points P1 and P2 of the projection area when the sensor pixel 804 is projected onto the virtual sensor 803. Reference numeral 808 represents a projection area when the sensor pixel 804 is projected onto the virtual sensor 803.

  The sensor pixel 804 is projected onto the virtual sensor 803 with reference to the optical center 805 of the microlens, and an end point 806 and a projection area 808 of the projection area 808 are formed.

From FIG. 8A, the projection area 808 is f _virtual / f _real times the sensor pixel 804. The projection area end point calculation unit 605 calculates the x coordinate of the projection area end point 806 (P1, P2) according to Equations 1 and 2.

  Here, l is the distance between the optical axes of adjacent microlenses. x is the distance from the center of the input pixel position to the optical axis of the main lens. s is the size of the sensor pixel 804. A pixel range surrounded by end points including P1 and P2 is a projection region. In other words, it can be said that the projection area end point calculation unit 605 determines the area of the composite image by calculating the coordinates of P1 and P2.

  FIG. 8B is an image diagram showing the input image in two dimensions. In FIG. 8B, it can be seen that one pixel 804 on the sensor is enlarged and projected as a projection region 808 on the virtual sensor. In the two-dimensional case, the projection area end point calculation unit 605 calculates the coordinates of the four corners 809P1, P2, P3, and P4 of the projection area 808 on the virtual sensor 803 in FIG.

<Detailed Operation of Projection Area Usage Rate Setting Unit>
The detailed operation of the projection area usage rate setting unit 606 will be described using a specific example with reference to FIGS. 9 and 10.

FIG. 9 shows a specific example of the usage rate of the projection area 808. In this embodiment, f _virtual / f _real is assumed to be 5. The sensor pixel 804 is projected on the virtual sensor 803 after being magnified 5 × 5 times. The projection area enlarged to the virtual sensor 803 is divided by the size of the sensor pixel 804, and (f _virtual / f _real ) ² (5 × 5 in this example) pixels are in the projection area. Each of these pixels is multiplied by the projection area utilization.

  An example of the projection area usage rate is shown in FIG. Reference numeral 901 denotes the usage rate of the conventional method. Conventionally, the sensor pixel 804 is simply multiplied by 5 × 5, and all the pixels in the projection area 808 are used to calculate the pixel value of the composite image, so that the projection area usage rate is all 1.

  Reference numeral 902 is an example of the projection area usage rate of the present embodiment. In this example, the projection area usage rate of the outer peripheral pixels (pixels on the frame of the projection area) in the projection area is set to zero. By using the projection area usage rate set as indicated by reference numeral 902, the effective range of the area projected onto the virtual sensor is reduced as compared with the case of reference numeral 901. That is, in the example of reference numeral 902, the effective range of the projection area is 9 pixels with the projection area usage rate set to 1.

  In the example of reference numeral 902, the usage rate is configured by only 0 and 1. However, as indicated by reference numeral 903, a value between 0 and 1 may be provided.

  As described above, in the examples indicated by reference numerals 902 and 903, different projection area usage rates are set for the pixels in accordance with the distance from the center pixel of the projection area. Also, a higher projection area usage rate is set for pixels closer to the center.

  Reference numerals 904, 905, and 906 in FIG. 9B are functions for creating the reference numerals 901, 902, and 903, respectively. The horizontal axis indicates the pixel position of the sensor pixel 804 unit, and the vertical axis indicates the usage rate of each pixel. Thus, it is also possible to generate the projection area usage rate using an arbitrary function.

  The respective functions of reference numerals 904, 905, and 906 are specifically shown in Expression 3, Expression 4, and Expression 5.

R1 and R2 in Equation 4 can take arbitrary values. Μ in Equation 5 is here
It is assumed that 0.5 × f _virtual / f _real , and σ is an arbitrary value.

  As described above, one pixel of the composite image on the virtual sensor is divided by the pixel size of the sensor pixel 804, and the effective range of the projection area 808 is determined by multiplying the divided area by the projection area usage rate.

  FIG. 10 shows projection areas on the virtual sensor calculated using the projection area usage rate 901 of the conventional method shown in FIG. 9 and the projection area usage rate 902 of this embodiment.

  FIG. 10A shows a case where the projection area usage rate 901 is used, and the pixel is projected by simply multiplying it by 5 × 5. Reference numerals 903 and 904 represent effective ranges of the projection area 808. On the other hand, FIG. 10B shows a case where the projection area usage rate 902 is used, and the effective range of the projection area 808 is represented by reference numerals 1005 and 1006. It can be seen that the effective range of the projection area is reduced in the areas 1005 and 1006 compared to the areas 903 and 904. When an image is synthesized using a projection area having a narrow effective range, overlapping of projection areas of different input pixels is reduced, and the number of pixels in the projection area used when averaging the pixels of the synthesized image is reduced. A contrast image can be generated, which leads to sharpening of the composite image.

  FIG. 15 shows an example of the result of image composition according to this embodiment. FIG. 15A shows a part of a refocus image obtained by combining images obtained by photographing a plurality of viewpoints of the CZP chart. An image 1501 in FIG. 15A is a result of generating a refocus image using the projection area usage rate 901 of the conventional method shown in FIG. An image 1502 in FIG. 15B is a result of generating a refocus image using the projection area usage rate 902 of the present invention shown in FIG. The image 1502 has a better contrast than the image 1501, and it can be seen that the sharpness of the composite image is improved by this embodiment.

  FIG. 15B shows the signal response value when the signal response value is acquired at the same place (position indicated by reference numeral 1503) in the result image of the conventional method and this embodiment. A chart 1504 is a signal response of the conventional method, and a chart 1505 is a signal response of this embodiment. In the chart 1505, it can be seen that the signal response varies greatly. Since the signal response fluctuates greatly, it can be confirmed that the contrast of the composite image generated by this embodiment is larger than the composite image generated by the conventional method.

  As described above, according to the present embodiment, a sharp refocus image can be generated in the Plenoptic Camera as shown in FIG. 4 or the camera array as shown in FIG.

  If the effective range of the projection area based on the set projection area usage rate is too narrow, an area in which the pixel average cannot be obtained occurs, and the value cannot be filled in the pixel of the composite image. A case where such a pixel exists is called a pixel skip. A method for setting the projection area usage rate for solving this problem will be described with reference to FIGS. 11 and 12 in this embodiment.

  This embodiment is the same as the first embodiment except for the processing performed by the pixel value synthesis unit 607 in step S705.

  In the first embodiment, after the projection area usage rate setting process in step S704, the process is performed in step S1201 of FIG.

  In step S1201, −1 is set as an initial value for each pixel of the composite image before the pixel value is set.

  In step S1202, the initial value is set in step S1201 using the projection usage rate set in the first embodiment, but the pixel value is set for the image to generate a composite image.

  In step S1203, the composite image inspection unit 1105 checks whether or not a pixel for which -1 is set exists for the generated composite image. That is, it is confirmed whether or not there is a pixel for which a pixel value is not set by image synthesis processing, and such a pixel is detected.

  If -1 is present, in step S1204, the projected area usage rate changing unit 1104 increases the projected area usage rate, and in step S1202, a composite image is generated again with the changed usage rate. If there is no pixel for which -1 is set on the composite image, the composite image is output in step 1205.

  In this embodiment, -1 is set as the initial value for each pixel of the composite image, and the presence of pixels that have been set with the initial value after the image composition processing is confirmed to check for pixel skipping. However, it is not limited to this method. Any method can be used as long as it can detect a pixel in which a pixel value is not set to a pixel that should originally have a pixel value in a combined image due to reducing the usage rate of the projection area and narrowing the effective area. The method can be used.

  As described above, according to this embodiment, it is possible to output a composite image without pixel skipping.

  There is a phenomenon in which the number of pixels associated with the pixels of the input image is reduced in a region outside the center in the composite image. This is because when the same projection area usage rate is used outside and at the center in the composite image, there are multiple pixel overlaps in the part close to the center, but there are few pixel overlaps in the peripheral part, resulting in the center in the composite image. This is because the pixel skip occurs outside the area.

  FIG. 16 shows pixel skipping in the peripheral portion of the composite image. FIG. 16A shows an example of using the projection area usage rate of the first embodiment. Reference numeral 1601 denotes an area of the synthesized refocus image. Reference numeral 1602 denotes a central portion of the composite image. Reference numeral 1603 indicates a peripheral region (region outside the center) of the composite image. Reference numerals 1604 and 1605 indicate pixel skipping. Reference numeral 1606 denotes a use range of the projection area according to the projection area use rate of the first embodiment.

  FIG. 16B shows an example in which the projection area usage rate in the peripheral area is improved. Reference numeral 1607 indicates the use range of the projection area after the projection area use rate is improved. Since the use range 1607 of the projection area is larger than the use area 1606, it is possible to prevent occurrence of pixel skipping as indicated by reference numerals 1604 and 1605.

  A method for setting the projection area usage rate for solving the pixel skip problem will be described with reference to FIGS. 13 and 14. FIG.

  The processing by the image composition unit 201 in this embodiment is the same as that in the first embodiment except for the processing performed by the projection area usage rate setting unit 606 in step S704 in FIG.

  After the process before entering the composite image generation process in step S704 in the first embodiment, the process in step S1401 in FIG. 14 which is the process in the present embodiment is continued.

  In step S1401, the distance calculation unit 1302 with the center pixel of the composite image calculates the distance between the pixel of the composite image associated with the pixel of the input image and the center pixel of the composite image. In this embodiment, the distance is a linear distance.

  In step 1402, the projection area usage rate setting unit 1303 sets the usage rate according to the distance.

  In this embodiment, the distance between the center pixel and the pixel on the frame of the composite image is the maximum distance dmax. The distance between the target pixel and the center pixel is d. In order to determine the usage rate according to the distance between the target pixel and the center pixel, d / dmax is calculated as a weighting factor.

  The usage rate can be controlled by multiplying R1 of Equation 4 by d / dmax and multiplying R2 by dmax / d. Also, the usage rate can be controlled by multiplying σ in Equation 5 by a weighting factor of d / dmax.

  In the present embodiment, the method of controlling the usage rate using a specific formula has been described, but the present invention is not limited to this method. Another method may be used as long as the usage rate can be set so that the usage rate of the pixel is increased in the image of the projection region synthesized outside the center in the synthesized image.

  As described above, according to the third embodiment, the peripheral pixels of the composite image can be output without pixel skipping.

  In the third embodiment, it is assumed that the image size of the input image is the same value in the horizontal and vertical directions, and the arrangement intervals of the microlenses are uniform.

  When the horizontal and vertical sizes of the input image are different, it is impossible to determine whether or not the pixel to be processed on the composite image is in the peripheral portion only by the distance of the central pixel. Therefore, in order to control the usage rate according to the distance from the center pixel, in addition to the position information of the pixel to be processed and the center pixel on the composite image, information on the horizontal and vertical sizes of the composite image is required. is there.

  On the other hand, when there is unevenness in the arrangement interval of the microlenses, the ratio of the portion where the arrangement interval of the microlens is wide overlaps with the projection area of the other image is low, and it is necessary to use a lot of the projection area projected by the input pixel on the virtual sensor is there. In other words, the usage rate of the projection area must be determined using the arrangement distance between the microlenses.

  As described above, in the case where the horizontal and vertical sizes of the input image are different from each other and in the case where there is a difference in the arrangement interval of the microlenses, the method described in the second embodiment so that the pixel skip does not occur on the composite image. Need to check for pixel skipping.

[Other Examples]
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (8)

An image processing apparatus for synthesizing multiple images taken by the multi-view,
Input means for inputting image data indicating the plurality of images;
For each pixel of a plurality of pixels included in a projection area that is an area corresponding to one pixel on the sensor and projected onto a virtual sensor determined according to the position of the subject focused by the synthesis Usage rate setting means for setting a usage rate for the composition ;
An image processing apparatus comprising: a combining unit configured to combine the plurality of images indicated by the image data based on the usage rate set in the usage rate setting unit .   The image processing apparatus according to claim 1, wherein the usage rate setting unit sets a different usage rate according to a distance from a center of the projection area.   The image processing apparatus according to claim 1, wherein the usage rate setting unit sets a higher usage rate for a pixel closer to the center in the projection area.   The image processing apparatus according to claim 1, wherein the usage rate setting unit sets the usage rate of outer peripheral pixels in the projection region to 0. 5. Detecting means for detecting a pixel whose pixel value is not set in the combined image by setting the usage rate;
The image processing apparatus according to claim 1, wherein the usage rate setting unit sets the high usage rate for the detected pixel. The usage setting means, the image after the synthesis, sets high the utilization than a pixel of the image of the projection area of the distance from the center of the image after the synthesis is synthesized in greater position The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. An image processing method for synthesizing multiple images taken by the multi-view,
An input step of inputting image data indicating the plurality of images;
For each pixel of a plurality of pixels included in a projection area that is an area corresponding to one pixel on the sensor and projected onto a virtual sensor determined according to the position of the subject focused by the synthesis A usage rate setting step for setting a usage rate for the composition ;
An image processing method comprising: a combining step of combining the plurality of images indicated by the image data based on the usage rate set in the usage rate setting step .   A program for causing a computer to function as the image processing apparatus according to any one of claims 1 to 6.

Images