JP6168220B2 - Image generation apparatus, image processing apparatus, image generation method, and image processing program - Google Patents

Description

  The present invention relates to an image generation device, an image processing device, an image generation method, and an image processing program.

  Conventionally, an image obtained by picking up an image of an object is arbitrarily adjusted by adjusting sharpness (focus adjustment) for each depth by substituting multiple images with different focal positions (multifocal images) into an identity based on a superposition model. A technique for generating a focus image or the like has been proposed (Patent Document 1). In the invention described in Patent Document 1, in order to convert the blur state, the blur state is converted for each multifocal image, and then the intensity sum of the blur image obtained by converting the plurality of blur states is obtained.

US Pat. No. 6,571,023

  However, in this method, when the blur state is changed for each image in the multifocal image group, a large conversion noise is generated. Therefore, an out-of-focus area becomes dirty and a good image cannot be obtained. In order to improve it, it has also been proposed to perform iterative calculation, but it was insufficient.

  The present invention has been made in view of the above circumstances, and an image generation apparatus, an image processing apparatus, an image generation method, and an image processing capable of generating an arbitrary blurred image so that an out-of-focus area is beautifully blurred. The purpose is to provide a program.

In one aspect of the image generating apparatus illustrating the present invention, a blur processing unit that performs a blur process assuming a common focus distance on a plurality of focus images having different focus states and a common viewpoint, and a blur process A sharpening processing unit that generates a blurred image with a predetermined blur from the plurality of focus images by performing a sharpening process on a focus stack image that is a sum of intensities of the plurality of images subjected to Is provided.
In one aspect of the image processing apparatus illustrating the present invention, an image acquisition unit that acquires a plurality of images with different viewpoints and a common shooting distance, or a plurality of images with different focus states and a common viewpoint, and the image A focus image generation unit that generates a plurality of focus images having different focus states and a common viewpoint from a plurality of images acquired by the acquisition unit, and the plurality of focus images generated by the focus image generation unit A blur processing unit that performs blur processing assuming a common in-focus distance, and a sharpening processing unit that performs sharpening processing on a focus stack image that is the sum of the intensities of a plurality of images subjected to blur processing .

According to one aspect of the image generation method of the present invention, a blur processing step of performing a blur process assuming a common focus distance on a plurality of focus images having different focus states and a common viewpoint, and the blur process A sharpening process step of generating a blurred image with a predetermined blur from the plurality of focus images by performing a sharpening process on a focus stack image that is a sum of intensities of the plurality of images subjected to Prepare.
According to one aspect of the image processing program of the present invention, a blur processing procedure for performing a blur process assuming a common focus distance on a plurality of focus images having different focus states and a common viewpoint, and a blur process Sharpening processing is performed on the focus stack image, which is the sum of the intensities of the plurality of images, and a sharpening procedure for generating an arbitrary blurred image from the plurality of focus images is executed by the computer.

According to the present invention, when a plurality of pupil regions arranged so as to surround the viewpoint on the pupil plane or its conjugate plane are sliced by a radius emitted from the viewpoint on the same plane, 80% of the deflection angle of the radius
With respect to a plurality of in-focus images acquired from a multi-view imaging unit that forms a light beam that passes through a plurality of pupil regions and forms an image of a subject individually for each pupil region. Each blur process set at an arbitrary focus position is performed, a focus stack image that is an intensity sum image of a plurality of images subjected to the blur process is calculated, an image sharpening process is performed on the calculated focus stack image, and an arbitrary sharpness process is performed. Since an image with blur is generated, it is possible to obtain an image with good blur that suppresses the generation of noise and becomes a blur that is beautiful outside the focus area.

It is a front side perspective view which shows the multi-lens camera which has arrange | positioned imaging | photography opening on the circumference. It is a back side perspective view which shows the multi-lens camera demonstrated in FIG. It is a front side perspective view which shows the multi-lens camera which has arrange | positioned imaging | photography opening in matrix form. It is explanatory drawing which shows the numerical sequence of a pseudorandom series of M series. It is explanatory drawing which shows the two-dimensional arrangement | sequence of the imaging opening based on M series pseudo-random number sequence "110100" by a x, y direction point sequence, and has shown the position of the imaging opening by "(circle)". It is a block diagram explaining the electrical structure of a multi-view camera. It is a block diagram which shows other embodiment which produces | generates the focus image f. It is explanatory drawing explaining the state which convolved image point p1, p2 with PSF, respectively. When the focal length is 5.34 mm, F1.4, and the image height × 2 = 3.17 mm, the image is formed in the optical axis direction z of the image space x, y, z when the image is formed from a short distance of 530 mm to infinity. It is explanatory drawing explaining a position. It is explanatory drawing explaining the outline of focusing by the synthetic aperture method using a some camera (cam). It is explanatory drawing explaining the parallax amount which generate | occur | produces by arrangement | positioning of cam1-cam5 arranged in the x or y direction demonstrated in FIG. 10 with four straight lines. It is explanatory drawing explaining the state by which the bokeh kernel is convolved with the object which has a focus with parallax z = 4 with respect to 5x5 eye camera arrangement | positioning. It is explanatory drawing explaining the state in which the blur kernel is convolved with the object in focus by the blur kernel z = 4 of arbitrary blur amount. It is explanatory drawing which represented the blur kernel demonstrated in FIG. 12 in x, y plane. FIG. 13 is an explanatory diagram illustrating the blur kernel (parallax z = 0 to z = 4) described in FIG. 12 in the x and y planes in the case of an M-series camera arrangement. FIG. 13 is an explanatory diagram illustrating the blur kernel (parallax z = 4 to z = 8) described in FIG. 12 in the x and y planes in the case of an M-series camera arrangement. It is explanatory drawing which shows the focus stack kernel which integrated the blur kernel in the parallax direction (z direction) in the focus position. It is explanatory drawing explaining the flow of an image process from the image capture of a multi-lens camera. It is explanatory drawing explaining the arbitrary blur algorithm which an image process performs. It is explanatory drawing which shows the several image taken in changing focus. It is an explanatory view showing a plurality of images captured by changing the focus and explaining that a convolution result between an object and an optical system can be obtained (explaining the equation described in [Equation 15]). It is explanatory drawing explaining the formula as described in [Formula 16]. FIG. 10 is an explanatory diagram in which an arbitrary defocus kernel is convolved in each x, y plane with a plurality of images (images obtained by a plurality of focuses) captured by changing the focus. It is a block diagram which shows the structure of the image process part which performs an arbitrary blurring algorithm. It is explanatory drawing which shows the focus stack kernel in the case of the multi-lens camera which arranged the imaging lens with the predetermined | prescribed periodic pattern according to the M series which is a pseudorandom series. It is a flowchart explaining the operation | movement procedure of a multiview camera. FIG. 10 is a diagram illustrating a part of the arbitrary blur algorithm, and is an explanatory diagram illustrating the integration range of the expressions described in [Expression 12] and [Expression 19] with respect to the expression described in [Expression 16]. It is a block diagram which shows other embodiment which produces | generates the image which provided the blur. It is explanatory drawing explaining the integration range of the formula as described in [Equation 19]. It is explanatory drawing which shows the outline of the cone of the light of the third order distribution (PSF) of point image intensity distribution. It is a graph which shows the relationship between the radius of the blur kernel of a Gaussian distribution, and a standard deviation. It is a graph which shows the relationship between a synthetic | combination blur radius and a standard deviation. It is a case where the lens of an annular zone is used for a blur algorithm, and is a graph which shows the PSF of the annular zone pupil which has the intensity distribution of the difference of two Gaussian distributions. It is a case where the lens of an annular zone is used for a blur algorithm, and is a graph which shows the relationship between a synthetic blur radius and a standard deviation. It is a perspective view which shows two Gaussian functions and the ring zone intensity distribution of the difference, and has shown the outer Gaussian distribution. It is a perspective view which shows two Gaussian functions and the ring zone intensity distribution of the difference, and has shown the inner Gaussian distribution. It is a perspective view which shows two Gaussian functions and the annular intensity distribution of the difference, and has shown the annular Gaussian distribution. The blur kernel of the annular optical system is shown. A blur image when using an annular optical system is shown, the left side is a blur image without using a blur algorithm with a chart placed at the focus position of the optical system (hereinafter referred to as an “optical blur image”), and the right side is a blur image. An image (hereinafter referred to as a “blurring algorithm image”) to which blur is applied by a blur algorithm in a no-blur state with the kernel focus position as a chart position is shown. The size of the blur kernel of the blur algorithm is 48 mm on the pupil plane. The blur image when using the annular optical system is shown. The left side shows the optical blur image without using the blur algorithm with the chart placed at the focus position of the optical system. The right side shows the blur image with the focus position of the blur kernel as the chart position. The blur algorithm image by the blur algorithm of the state of nothing is shown. The size of the blur kernel of the blur algorithm is set to 36 mm on the pupil plane. The blur image when using the annular optical system is shown. The left side shows the optical blur image without using the blur algorithm with the chart placed at the focus position of the optical system. The right side shows the blur image with the focus position of the blur kernel as the chart position. The blur algorithm image by the blur algorithm of the state of nothing is shown. The size of the blur kernel of the blur algorithm is 24 mm on the pupil plane. The blur image when using the annular optical system is shown. The left side shows the optical blur image without using the blur algorithm with the chart placed at the focus position of the optical system. The right side shows the blur image with the focus position of the blur kernel as the chart position. The blur algorithm image by the blur algorithm of the state of nothing is shown. The size of the blur kernel of the blur algorithm is 12 mm on the pupil plane. The blur image when using the annular optical system is shown. The left side shows the optical blur image without using the blur algorithm with the chart placed at the focus position of the optical system. The right side shows the blur image with the focus position of the blur kernel as the chart position. The blur algorithm image by the blur algorithm of the state of nothing is shown. The size of the blur kernel of the blur algorithm is 6 mm on the pupil plane. The blur image when using the annular optical system is shown. The left side shows the optical blur image without using the blur algorithm with the chart placed at the focus position of the optical system. The right side shows the blur image with the focus position of the blur kernel as the chart position. The blur algorithm image by the blur algorithm of the state of nothing is shown. The size of the blur kernel of the blur algorithm is 1.2 mm on the pupil plane. The blur image when using the annular optical system is shown, the left side shows the optical blur image without using the blur algorithm with the chart placed at the focus position (Z = 10.7 m) of the optical system, and the right side shows the focus of the blur kernel The blur algorithm image by the blur algorithm in the state of no blur with the position as the chart position is shown. It is a front view which shows arrangement | positioning of the camera (single-eye imaging part) of a multi-lens camera, (1) is a rectangular pupil densely arranged in a rectangular shape, (2) is a configuration in which rectangular pupils are arranged in a rectangular shape. (3) shows the size of the blur kernel with a circular radius r on the pupil planes s and t, (4) shows the circular pupil densely in a rectangular shape, and (5) shows the circular pupil flying in a rectangular shape. The configuration arranged in FIG. It is a front view which shows arrangement | positioning of the camera (single-eye image pick-up part) of a multi-lens camera, (1) is the structure which arranged the rectangular pupil densely in the vertically long rectangle shape, and (2) was arranged in the shape of the vertically elongated rectangle. (3) shows the size of the blur kernel by the circular radius r on the pupil planes s and t, (4) shows the rectangular pupil densely in a diamond shape, and (5) shows the rectangular pupil in a diamond shape. Fig. 2 shows a configuration arranged in a jump. It is a front view which shows arrangement | positioning of the camera (single-eye imaging part) of a multi-lens camera, (1) is a structure which arrange | positioned the circular pupil to the rhombus shape, and narrowed the space | interval to the rhombus shape (2). Is shown. As shown in FIG. 46 (4), the image on the left side is a synthetic aperture blur image (using a synthetic aperture method) generated using an image captured by a multi-lens camera in which the pupils of each optical system are arranged on a square line without any gaps. A defocused image at the time of focusing is shown. The pupil has a square arrangement with a pitch (p) of 12 mm and a circular shape with a diameter (k) of 12 mm. The image shown on the right side is a circular blur algorithm image in which the size of the blur kernel is a radius (r) of 18 mm on the pupil plane st plane, as shown in FIG. As shown in FIG. 46 (4), the image on the left side shows a synthetic aperture blurred image generated using an image captured by a multi-lens camera in which the pupils of each optical system are arranged on a square line without a gap. The pupil has a square arrangement with a pitch (p) of 12 mm and a circular shape with a diameter (k) of 10.8 mm. The image shown on the right side is a circular blur algorithm image in which the size of the blur kernel is a radius (r) of 18 mm on the pupil plane st plane, as shown in FIG. As shown in FIG. 46 (4), the image on the left side shows a synthetic aperture blurred image generated using an image captured by a multi-lens camera in which the pupils of each optical system are arranged on a square line without a gap. The pupil has a square arrangement with a pitch (p) of 12 mm and has a circular shape with a diameter (k) of 9.6 mm. The image shown on the right side is a circular blur algorithm image in which the size of the blur kernel is a radius (r) of 18 mm on the pupil plane st plane, as shown in FIG. As shown in FIG. 46 (1), the image on the left side shows a synthetic aperture blurred image generated using an image captured by a multi-lens camera in which the pupils of each optical system are arranged on a square line without a gap. The pupil has a square arrangement with a pitch (p) of 12 mm and a square with a side length (l) of 12 mm. The image shown on the right side is a circular blur algorithm image in which the size of the blur kernel is a radius (r) of 24 mm on the pupil plane st plane, as shown in FIG. 46 (3). As shown in FIG. 46 (1), the image on the left side shows a synthetic aperture blurred image generated using an image captured by a multi-lens camera in which the pupils of each optical system are arranged on a square line without a gap. The pupil has a square arrangement with a pitch (p) of 12 mm and a square with a side length (l) of 12 mm. The image shown on the right side is a circular blur algorithm image in which the size of the blur kernel is a radius (r) of 18 mm on the pupil plane st plane, as shown in FIG. As shown in FIG. 46 (1), the image on the left side shows a synthetic aperture blurred image generated using an image captured by a multi-lens camera in which the pupils of each optical system are arranged on a square line without a gap. The pupil has a square arrangement with a pitch (p) of 12 mm and a square with a side length (l) of 12 mm. The image shown on the right side is a circular blur algorithm image in which the size of the blur kernel is a radius (r) of 12 mm on the pupil plane st plane, as shown in FIG. 46 (3). As shown in FIG. 46 (1), the image on the left side shows a synthetic aperture blurred image generated using an image captured by a multi-lens camera in which the pupils of each optical system are arranged on a square line without a gap. The pupil has a square arrangement with a pitch (p) of 12 mm and a square with a side length (l) of 12 mm. The image shown on the right side is a circular blur algorithm image in which the size of the blur kernel is a radius (r) of 6 mm on the pupil plane st plane, as shown in FIG. As shown in FIG. 46 (2), the image on the left side shows a synthetic aperture blurred image generated using an image picked up by a multi-lens camera in which the pupils of each optical system are arranged on a square line without a gap. The pupil has a square arrangement with a pitch (p) of 12 mm and a square with a side length (l) of 10.8 mm. The image shown on the right side is a circular blur algorithm image in which the size of the blur kernel is a radius (r) of 18 mm on the pupil plane st plane, as shown in FIG. As shown in FIG. 46 (1), the image on the left side shows a synthetic aperture blurred image generated using an image captured by a multi-lens camera in which the pupils of each optical system are arranged on a square line without a gap. The pupil has a square arrangement with a pitch (p) of 12 mm and a square with a side length (l) of 9.6 mm. The image shown on the right side is a circular blur algorithm image in which the size of the blur kernel is a radius (r) of 18 mm on the pupil plane st plane, as shown in FIG. As shown in FIG. 46 (1), the image on the left side shows a synthetic aperture blurred image generated using an image captured by a multi-lens camera in which the pupils of each optical system are arranged on a square line without a gap. The pupil has a square arrangement with a pitch (p) of 12 mm and a square with a side length (l) of 8.4 mm. The image shown on the right side is a circular blur algorithm image in which the size of the blur kernel is a radius (r) of 18 mm on the pupil plane st plane, as shown in FIG. As shown in FIG. 47 (1), the image on the left side shows a synthetic aperture blurred image generated using an image captured by a multi-lens camera in which the pupils of each optical system are arranged on a square line without any gap. The pupil has a square arrangement with a pitch (p) of 12 mm and a square with a side length (l) of 12 mm. The image shown on the right side is the same optical system as shown in FIG. 47 (3), and is a circular blur algorithm image in which the size of the blur kernel is a radius (r) of 4.8 mm on the pupil plane st plane. . As shown in FIG. 47 (1), the image on the left side shows a synthetic aperture blurred image generated using an image captured by a multi-lens camera in which the pupils of each optical system are arranged on a square line without any gap. The pupil has a square arrangement with a pitch (p) of 12 mm and a square with a side length (l) of 12 mm. The image shown on the right side is a circular blur algorithm image in which the size of the blur kernel is a radius (r) of 6 mm on the pupil plane st plane, as shown in FIG. 47 (3), in the same optical system. As shown in FIG. 47 (1), the image on the left side shows a synthetic aperture blurred image generated using an image captured by a multi-lens camera in which the pupils of each optical system are arranged on a square line without any gap. The pupil has a square arrangement with a pitch (p) of 12 mm and a square with a side length (l) of 12 mm. The image shown on the right side is a circular blur algorithm image in which the size of the blur kernel is a radius (r) of 7.2 mm on the pupil plane st plane as shown in FIG. 47 (3). . As shown in FIG. 47 (4), the image on the left side shows a synthetic aperture blurred image generated using an image captured by a multi-lens camera in which the pupils of each optical system are densely arranged in a diamond shape. The pupil has a square shape with a pitch (p) of 12 mm and a side length (l) of 12 mm. The image shown on the right side is a circular blur algorithm image in which the size of the blur kernel is a radius (r) of 5.4 mm on the pupil plane st plane, as shown in FIG. 47 (3), in the same optical system. . As shown in FIG. 47 (4), the image on the left side shows a synthetic aperture blurred image generated using an image captured by a multi-lens camera in which the pupils of each optical system are densely arranged in a diamond shape. The pupil has a square shape with a pitch (p) of 12 mm and a side length (l) of 12 mm. The image shown on the right side is a circular blur algorithm image in which the size of the blur kernel is a radius (r) of 6 mm on the pupil plane st plane, as shown in FIG. 47 (3), in the same optical system. As shown in FIG. 48 (1), the image on the left side shows a synthetic aperture blurred image generated using an image captured by a multi-lens camera in which the pupils of each optical system are densely arranged in a diamond shape. The pupil has a circular shape with a pitch (p) of 12 mm and a diameter (k) of 12 × √ (2) mm. The image shown on the right side is the same optical system, as shown in FIG. 47 (3), and a circular blur with a blur kernel having a radius (r) of 6 × √ (2) mm on the pupil plane st plane. It is an algorithm image. As shown in FIG. 46 (1), the image on the left side is a multi-lens having a pupil of each optical system arranged on a square line with a side length (l) of 12 mm and a 12 mm pitch (p). It is a synthetic | combination aperture blur image produced | generated using the image imaged with the camera. The right-side image is a circular blur algorithm image in which the size of the blur kernel is a radius (r) of 36 mm on the pupil plane st plane, as shown in FIG. As shown in FIG. 46 (1), the image on the left has a pupil of each optical system that is randomly arranged on a square line with a length (l) of one side of 12 mm and a 12 mm pitch (p). It is a synthetic | combination aperture blur image produced | generated using the image imaged with the multi-view camera. The right-side image is a circular blur algorithm image in which the size of the blur kernel is a radius (r) of 36 mm on the pupil plane st plane, as shown in FIG. As shown in FIG. 46 (5), the image on the left is taken by a multi-lens camera having a circular shape with a diameter (k) of 6 mm and a pupil of each optical system arranged on a square line at a pitch of 12 mm (p). It is a synthetic aperture blurred image generated using the obtained image. The right-side image is a circular blur algorithm image in which the size of the blur kernel is a radius (r) of 36 mm on the pupil plane st plane, as shown in FIG. As shown in FIG. 46 (4), the image on the left is taken by a multi-lens camera having a circular shape with a diameter (k) of 12 mm and a pupil of each optical system arranged on a square line at a pitch of 12 mm (p). It is a synthetic aperture blurred image generated using the obtained image. The right-side image is a circular blur algorithm image in which the size of the blur kernel is a radius (r) of 36 mm on the pupil plane st plane, as shown in FIG. It is explanatory drawing which shows the entrance pupil which is an image of the aperture stop by the optical system ahead of an objective lens. It is explanatory drawing explaining the synthetic | combination optical axis in the case of arrange | positioning a some objective lens. It is explanatory drawing explaining the change of a blur kernel in the multi-lens camera which has arrange | positioned the objective lens to symmetry and non-object. It is explanatory drawing which shows the structure which has arrange | positioned the annular pupil densely on the periphery of the radius r. It is explanatory drawing which shows the zone-shaped structure which has closely arranged the zone pupil on the double circumference from which a radius differs. It is explanatory drawing which shows the structure of the annular zone pupil in the state where there is no cut | interruption in an arc direction and a symmetry is high in a rotation direction. It is explanatory drawing which shows the outline of the slice by the radial radius of the annular pupil of the state which becomes substantially uniform and dense. It is explanatory drawing which shows the structure which has arrange | positioned 24 square openings in the shape of a ring zone. FIG. 77 is an explanatory diagram showing a main part of the rectangular opening described in FIG. 76. It is explanatory drawing which shows another embodiment which carries out the amplitude division | segmentation of the to-be-photographed object light which injects from an objective lens, and is simultaneously guide | induced to several focus positions. It is explanatory drawing which shows other embodiment which multiplexes a pupil by an annular pupil optical system, and divides | segments a wave front. FIG. 80 is an explanatory diagram showing an example in which a single image sensor is used using a spatial light modulator as another example of the wavefront division described in FIG. 79; It is explanatory drawing which shows the example which uses a blur algorithm by pupil-dividing using a microlens array. 81 shows a light receiving element group (segment) corresponding to the microlens described in FIG. 81, (1) shows a light receiving element located on the annular zone in the light receiving element group, and (2) receives light only on the annular zone. (3) shows an example in which light receiving elements arranged on concentric circles are increased. (4) is an explanatory diagram showing an example in which the blur kernel is asymmetric before and after the focus position and the sizes of the front blur and the rear blur are adjusted. It is explanatory drawing which shows the example which arranged closely the camera group which arranged multiple cameras with a synthetic aperture with a micro lens in the shape of a ring zone. It is explanatory drawing which shows the relationship between a focus position when an optical pupil is arrange | positioned at annular | circular shape, and the position of an imaging surface. It is explanatory drawing which shows the relationship between the to-be-photographed object distance of a structure using the annular pupil optical system containing the imaging lens which has a rotationally symmetric aberration, and an imaging surface. It is explanatory drawing which shows the example which made the blur kernel asymmetric before and behind a focus position, and adjusted the magnitude | size of front blur and back blur. It is explanatory drawing which shows the example which arranged the lens whose external shape was a quadrangle densely on the circumference. FIG. 88 is a main part explanatory view showing the main parts of the lens described in FIG. 87; It is explanatory drawing which shows the example which arranged the crescent-shaped lens densely on the circumference. It is explanatory drawing which shows the example which adhere | attached the optical element on the glass substrate and comprised the several objective lens demonstrated in FIG. It is explanatory drawing which shows embodiment which virtually produces | generates the image obtained when tilting an image surface and performing a tilt photography using a synthetic aperture method.

  As shown in FIG. 1, the multi-lens camera 10 is provided with a plurality of circular shooting openings 11 arranged circumferentially on the front surface of the camera body 12. A photographing lens (imaging optical system) 23 and an image sensor (not shown) are disposed in the back of each photographing aperture 11. In this example, a single-eye imaging unit including the imaging aperture 11, the imaging lens 23, and the imaging element is configured, and a configuration in which a plurality of single-eye imaging units are integrated is a multi-eye imaging unit. The plurality of single-eye imaging units are arranged so that the optical pupil formed by the photographing lens 23 of each single-eye imaging unit is dense on the circumference in a plane orthogonal to the photographing optical axis. Although not shown, the stop may be provided in the imaging optical system or outside the imaging optical system.

  As each image sensor, for example, an image sensor having low pixels and low power consumption is used. Further, the optical axes of the photographing lenses are substantially parallel. Each imaging element is arranged so that the imaging surface is perpendicular to the optical axis of the imaging lens.

The multi-lens camera is provided with a shutter button 13 and a power switch 14 on the upper surface of the camera body 12, and simultaneously acquires a plurality of image data with different viewpoints by one shutter release, and uses these image data as parameters. By performing image processing accordingly, sharpness adjustment (focus adjustment) for each depth, that is, an arbitrary focus image generated by intentionally blurring a specific subject in the screen, the intensity and arrangement of blur ( Blur generation image generated by adjusting blur size, shape, and weight), and arbitrary focal point of arbitrary viewpoint for stereoscopic viewing with a viewpoint generated by combining the process of blur generated image and arbitrary focus image, arbitrary It is also possible to generate a blur generation image. That is, (1) `` Focus position in the optical axis direction '', (2) `` Bokeh size '', (3) ``
A camera capable of freely selecting three parameters of “viewpoint” can be configured. Further, by using a plurality of cameras, it is possible to increase the number of pixels, and it is also possible to generate high-resolution image data, that is, image data with multiple pixels.

  An LCD 15 is arranged behind the camera body 12 as shown in FIG. The LCD 15 forms a touch panel together with the transparent tablet 34. The touch panel displays images such as menu characters and operation buttons over a through image (or an image that has been focused using the synthetic aperture method described below, or an image that has been blurred by the blur algorithm), and touch operations. The input of parameters such as zooming operation, designation of focal plane, selection of blur parameters (virtual aperture size, shape, weight) is performed.

The present invention relates to a camera using a blur algorithm, which will be described later, and particularly relates to a camera in which the optical pupil distribution is a circle or a rectangle. By using such an optical pupil distribution, the optical pupil is arranged in a grid. Further, it is possible to give a more beautiful blur than the multi-lens camera as shown in FIG. First, the blur algorithm will be described with reference to an example of a normal multi-lens camera in which the optical pupil is arranged in a grid as shown in FIG. 3, and the arrangement of the optical pupil that gives more beautiful blur will be described later with reference to FIG.
The multi-view camera 9 shown in FIG. 3 has the optical apertures arranged in a two-dimensional manner, that is, the photographing apertures 11 are arranged in a matrix. Each of these apertures is equipped with a two-dimensional image sensor, which is literally a multi-lens camera. The intervals in the x direction and the y direction of the imaging aperture 11 are set to a predetermined periodic pattern according to the M sequence that is a pseudo-random sequence. In this case, the autocorrelation function of the M sequence is close to the delta (δ) function, and has a feature that the correlation function value is constant except for the peak. Since the autocorrelation function is a delta function, the power spectrum is flat (constant without depending on frequency). This is a desirable property because a specific frequency component is not emphasized when generating blur.

  In the case of a 5 × 5 25-eye grid camera, the weights described in, for example, [Equation 1], [Equation 2], and [Equation 3] are multiplied when calculating the synthetic aperture output by [Equation 6]. (The calculation of the synthetic aperture output by the synthetic aperture method with the camera arrangements in FIG. 46 and thereafter is calculated by [Equation 6] with the weight of the image sensor of each camera as 1.) . In the case of 16 eyes selected with the M-sequence arrangement described with reference to FIG. The combined optical axis in this case is, for example, the camera with X = 3 and Y = 3 described in FIG.

  The M-sequence pseudorandom number sequence of the sequence Xi (i = 1 to m, where m is a cycle) is generally “0” having a cycle m of “2n−1” for an integer n of “1” or more. And “1” in a binary number sequence, the number of the number sequences in one period m is an odd number, and the difference between the number of “0” and the number of “1” is “1”. Therefore, combinations (n, m) of the integer n and the period m include (3, 7), (4, 15), (5, 31), (6, 63), and the like.

  As shown in FIG. 4, for example, in a sequence where the integer n is “2” and the period m is “3”, it is “1, 0, 1”, and for example, the integer n is “3” and the period m is “7”. In the numerical sequence of “1, 1, 1, 0, 1, 0, 0”, for example, in the numerical sequence in which the integer n is “4” and the period m is “15”, “1, 1, 1, 1, 0, 1, 0, 1, 1, 0, 0, 1, 0, 0, 0 ". In the present embodiment, for example, a numerical sequence “1, 1, 1, 0, 1, 0, 0” having an integer n of “3” and a period of “7” is employed. In this case, a two-dimensional array pattern as shown in FIG. 5 is obtained. In this pattern, the position indicated by “◯” is the position where the photographing aperture 11 is arranged.

In addition, the method of making this number sequence is, for example, “Correlation Function and Spectrum-Its Measurement and Application” published by Takashi Isobe, published by the University of Tokyo in February 1968 and the second edition in January 1971. 170-175.

  By the way, the interval of the two-dimensional array of the photographing apertures 11 may be set to a predetermined periodic pattern which is a binary pseudo-random sequence such as a Q sequence (square residue sequence), a Gold sequence, a Walsh code, etc. in addition to the M sequence. it can. Further, the intervals in the two-dimensional direction of the imaging openings 11 may be equal. However, since it is desirable that there is no zero point in the pupil sampling points before the focus stack, which will be described in detail later, it is desirable to set the interval in the two-dimensional direction to a predetermined periodic pattern of M series. The number of imaging apertures 11 is not limited to 16 (16 eyes).

  As shown in FIG. 6, each image sensor 20 is connected to an AFE 21 and a frame memory 22, and captures a subject image formed by the photographing lens 23 and outputs an image signal to the AFE 21. Here, reference numeral 40 indicates the single-eye imaging unit 40.

  The AFE 21 includes a well-known CDS (correlated double sampling) / AGC (gain control amplifier circuit), A / D, and a signal processing circuit, and operates in synchronization with a pulse supplied from the TG together with the CPU 24. (Not shown). The signal processing circuit captures digital image data and performs corrections such as pixel defect correction, white balance correction, and gamma correction.

  The CPU 24 controls the charge accumulation time (electronic shutter) of the image sensor 20 and measures the luminance of the subject based on the image data obtained from the specific image sensor 20, and all the image sensors based on the measurement result. The exposure is controlled by changing the value of 20 electronic shutters to be constant.

  The AF unit 26 fetches at least two of the plurality of image data in response to the half-pressing operation of the shutter button, and obtains the relative shift amount between the two images based on the two image data. Subject distances for a plurality of subjects are calculated from the shift amounts, and one focusing driver 27 is controlled to move the photographing lens 23 to a focusing position corresponding to the subject distance, and the remaining focusing drivers 27 are controlled. Then, with the in-focus position corresponding to the subject distance as a reference, the in-focus positions of the remaining imaging lenses 23 are sequentially shifted so as to be shifted with a uniform blur amount. Note that the image pickup device 20 may be moved, or both the photographing lens 23 and the image pickup device 20 may be moved to shift the focus.

  As a result of estimating the necessity of AF, it is desirable to use an image sensor (including an image sensor and a driver circuit) having an optical size of 1/8 inch or less in order to realize a thin camera with a thickness of 3 mm. This is because if the diagonal length of the imaging surface of the imaging sensor is “Dg”, the total optical path length is “Lp”, and a lens design with sufficient aberration characteristics is satisfied, Lp> 0.9 Dg.

In the case of a multi-lens camera, the number of pixels can be increased up to several times as many as the camera, so each camera module may have fewer output pixels than the final camera. Depending on the algorithm for increasing the number of pixels, if an algorithm such as the pixel data of the nearest imaging sensor is used, the generated image will not fail if the number of pixels is reduced to 0.56 times the number of cameras or less. . For example, in the case of a camera with 6M (mega, 1 million) pixel output, it is easy to increase the number of pixels by about 9 times when 16 eyes are used. In that case, 6M × 1/9 = 0.6 from 1 / (16 × 0.6) = 1/9 of 6M
Any camera module having a pixel number of 7M pixels or more may be used.

  An optical system for resolving 6M with an image sensor size of 1/8 inch image height is required. Specifically, the MTF at the Nyquist wavelength must be at least 10%. In this case, the Nyquist wavelength is 1.169 μm. For this reason, the F number of a lens needs to be 1.4 or less. When the thickness of the 6M output camera is 5 mm, the nominal size of the image sensor can be 1/4 inch, and the F number of the lens may be 2.8 or less.

  The presence or absence of AF was examined for the above-mentioned 3 mm and 5 mm thick card cameras including lens aberrations. If the camera focus position was set to about 2 m to the subject, the distance to the subject was 1 m. It was found that the change in MTF from infinity to infinity is a change of 50% or less at the Nyquist frequency and can be regarded as pan focus. It is also possible to perform deconvolution so as to compensate for the change in MTF (PSF also changes at the same time) according to the focus position. When refocusing (focusing on an arbitrary subject distance after shooting), it is possible to compensate for the degradation of MTF according to the distance. That is, as shown in FIG. 7, a storage unit 49 that stores PSF or MTF for each focus position in advance, and image processing so as to cancel PSF or MTF changes due to the focus position when generating a focus image by the synthetic aperture method. And an image processing unit 41 that performs the above. The storage unit 49 and the image processing unit 41 are connected together with the CPU 42 by a bus 43 and are controlled by the CPU 42 in an integrated manner. Reference numeral 44 denotes an image acquisition unit for capturing a plurality of images, such as a multi-lens camera, a scanner, and an external input unit. Even with this configuration, the out-of-focus image is blurred according to the present invention, so that the out-of-focus image is overcorrected and the image does not become unnatural.

  The AF actuator of the camera module may be used when setting the focus at a distance of 2 m to the subject. This is because the focus adjustment on the image side of the size of the image sensor of 1/8 inch is severe, and the adjustment and stability at the level of several microns are required.

  By the way, the focus of one photographing lens 23 is adjusted to the reference subject distance measured, and the remaining photographing lenses 23 are sequentially shifted in focus so that the amount of blur is uniform with respect to the in-focus position. Going out of focus is difficult to control. Therefore, the AF unit 26 divides the shooting distance range from the shortest shooting distance to infinity by the number of the single-eye imaging units 40, and each focusing driver so that the shooting distance corresponding to each division position is in focus. It is preferable to move each photographic lens 23 individually by controlling 27. In addition, when the focus is not performed after the photographing, the pan focus may be configured by the diaphragm and the photographing lens 23. In this case, the AF unit 26 and the like can be omitted.

  The ROM 28 stores various programs and setting values necessary for executing the programs in advance. The RAM 29 is used as a work memory for the CPU 24 and as a temporary memory for each unit. The CPU 24, ROM 28, RAM 29, LCD driver 33, I / F 30, and image processing unit 25 are connected by a bus 32.

  The photographing lens 23 may be a zoom lens or a focal length switching type lens. In this case, all the photographing lenses 23 may be configured to be zoomed in the same manner in synchronization with the zooming operation.

  Various operation signals are input to the CPU 24 from the tablet 34 (not shown). The CPU 24 simultaneously loads a plurality of image data (multifocal image data) having different focal points into each frame memory 22 in response to one release operation. The multifocal image data is individually taken into each frame memory 22 and then output to the image processing unit 25.

<Three-dimensional shift invariant imaging>
Considering the blur due to defocusing in the optical axis direction of the point image PSF formed by the optical system, it is necessary to consider a three-dimensional PSF. An object is also three-dimensional, and can be regarded as shift-invariant imaging within an isoplanatic range, and imaging can be expressed with the same PSF anywhere on the object surface. For example, when there are image points p1 and p2 in the image space x, y, z, P
The state in which the SF is convolved is the image formation state. In general, an object has irregularities, and as shown in FIG. 8, image points corresponding to object points on the surface visible from the camera are convolved with PSFs as shown in FIG. The three-dimensional PSF shown in the figure may be called a three-dimensional blur kernel. When obtaining such a three-dimensional imaging state, in the case of normal lens imaging, for example, as shown in FIG. 9, focal length 5.34 mm, F1.4, diagonal imaging range = image height × 2 = 3 In the case of .17 mm, even if an image is formed from a short distance of 530 mm to infinity, the imaging position in the optical axis direction z of the image space x, y, z is 5.34 mm and 5.39 mm, and only a change of 50 μm occurs at most. However, this range can be regarded as an isoplanatic range in the z direction.

Since the PSF does not change significantly with respect to x and y as well, a three-dimensional image is obtained by moving and positioning a two-dimensional image sensor in parallel with the image plane I in the normal lens imaging state. It is possible to acquire image state data. It is desirable in terms of signal processing that the data acquisition intervals in the optical axis z direction of the two-dimensional imaging device are equal intervals. Note that z min corresponds to the infinity position. Although it is desirable that z max sufficiently include the latest position from the lens of the subject, there is no problem in a range where refocusing may be practically performed. When z min is focused only in the near view, the data of the two-dimensional image sensor up to infinity is not necessarily required, and the point farthest in the depth direction of the refocus range may be set as z min. Particularly in macro photography, since data in the z direction tends to increase, it is necessary to set appropriate ranges to z min and z max. The imaging magnification is obtained from the formula of the lens imaging formula “1 / f = 1 / Z + 1 / b” where “Z” is the distance from the lens to the subject and “b” is the distance from the lens to the image point. .

<About Focusing by Synthetic Aperture Method>
As shown in FIG. 10, the focusing (focusing) by the synthetic aperture method is as follows. When focusing on the distance Z, if the distance from the center camera is “s”, the amount of parallax = camera interval × imaging magnification The image is shifted by = s × b / Z so that the center camera image and the image point at the focus position overlap.

  When a pinhole is used instead of a lens, the imaging magnification is obtained from the formula “f / Z”. When a pan focus lens is used, the focus position of the lens calculates the change in MTF, and the fixed focus setting distance is determined so that the degradation of the shortest shooting distance and the MTF at infinity is comparable. For example, when the shortest shooting distance is 1 m, the focus position is set to 2 m. The value of b is obtained from the lens imaging formula. ([Equation 8])

  In the following description, the unit of parallax is often expressed in pixels, but in the present invention, the focus image is synthesized by a synthetic aperture method at a subject distance in which the parallax amount is an integral multiple of the pixel pitch of the image sensor. This is because it is efficient to perform the processing, and the processing result also has good image quality. As the pixel pitch, the original value of the image sensor may be used, or the pitch of the image sensor after the number of pixels is increased may be used. In addition, when there are too many focus points, such as during macro shooting, it is not necessary to generate a synthetic aperture image with the parallax amount corresponding to each pixel, and thinning can be performed as appropriate. It is necessary to judge the thinning ratio while viewing the image.

Consider acquiring shift-invariant three-dimensional imaging state data using the synthetic aperture method. For that purpose, it is desirable to express the optical axis direction z of the image space by the amount of parallax. The reason is as follows.

Consider the coordinate system of an object on an X, Y, Z linear (equally spaced) scale. The optical axis is taken in the Z-axis direction. A coordinate system (x, y, z) of an image space for focusing (focusing) by a synthetic aperture method of a multi-lens camera is used. As in the case of a pinhole camera, each camera of a multi-lens camera usually uses a camera with a deep focal depth and can be refocused. First, consider a pinhole camera as shown in FIG.

  Each pinhole camera does not have information in the Z direction, and an object at any distance is imaged on an imaging plane located at a focal length f from the pinhole. However, when focusing is performed by the synthetic aperture method, parallax = blurring due to camera arrangement occurs. Focusing with a multi-lens camera will be described with reference to FIG.

  The camera has a 5 × 5 arrangement in the X and Y directions. Since a 25-eye image is taken with a pinhole camera and focusing is performed by the synthetic aperture method, the imaging magnification of the subject is determined at the time of shooting and does not depend on the focusing. The imaging magnification is the magnification of the pinhole camera: f / Z, where Z is the distance to the subject.

FIG. 10 represents a multi-view camera as a plurality of pinhole cameras, and the viewpoint of each camera, that is, the position of the pinhole, is located on the camera plane (pupil conjugate plane) (s-t plane). The camera at the position (equivalent to the viewpoint position and pinhole position) (s, t) is denoted by D (s, t). The X, s, and x axes are parallel. The Y axis, t axis, and y axis are parallel. The object plane and the camera plane are parallel and represent a state in which the object plane is in focus at an arbitrary distance Z from the camera plane by the synthetic aperture method. In the figure, the pinhole of the pinhole camera is represented by a triangular apex. Further, the side facing the apex representing the pinhole represents the imaging surface.

Assume that the coordinates of the imaging surface of the imaging sensor of each camera are xD (s, t) and yD (s, t). The origin and pinhole of the sensor coordinates of each camera are on the optical axis of each camera, and the optical axis of each camera is parallel to the Z axis. The camera surface and the imaging surface are parallel, and the optical axis of each camera is orthogonal to them. The coordinates xD (s, t) and yD (s, t) are values without error after calibration. The point P (X, Y, Z) on the object plane (X, Y) is at a distance Z from the camera plane, and x′D (s, t), y′D (of each camera D (s, t). s, t). This is shown by the equations described in [Equation 4] and [Equation 5]. If there is cam3 on the Z axis, xD (0,0) = X × f / Z, yD (0,0) = Y × f / Z.

  Here, fD (s, t) described in [Equation 4] and [Equation 5] are measured focal lengths, and const_xD (s, t) and const_yD (s, t) are constants for tracking calibration errors.

Let C (x′D (s, t), y′D (s, t)) be the output of the image pixels on the imaging surface. The synthetic aperture output I (x, y, Z) focused on the distance Z can be calculated from the equation shown in [Equation 6]. The substantial viewpoint of the image obtained by the synthetic aperture method and the synthetic aperture image is called a synthetic viewpoint. The composite viewpoint is the origin of s and t coordinates from [Equation 4] and [Equation 5].

  However, the synthetic aperture output is the synthetic output of any selected camera. For example, 16 of 25 units. Schematically, the state of image synthesis by the synthetic aperture method is to perform focusing (focusing (focusing)) by relatively moving the image according to the focus position.

[Expression 4], [Expression 5], and [Expression 6] are the expressions described in the example of the pinhole camera. However, a multi-lens camera with a camera lens is actually used. In this case, the parallax amount is expressed by the equation described in [Equation 7] or [Equation 8].

  It is efficient to acquire an image at an object distance Z at which the parallax amount z is equally spaced, and if the parallax amount z is determined, the distance Z to the subject is expressed by the equation described in [Equation 7] or [Equation 8]. Is uniquely determined by The amount of parallax z in the case of a pinhole camera is proportional to the product of camera interval (distance between any two viewpoints on the pupil plane): s0 and imaging magnification: b / Z, and is given by the equation shown in [Equation 7]. Indicated. Therefore, the larger the camera interval, the greater the amount of change in parallax with respect to the subject distance. In a multi-lens arrangement camera, the amount of change in parallax with respect to the subject distance is maximized at the interval of two viewpoints: s max where the distance between the cameras is the longest. Therefore, in the case where the sampling pitch of the parallax amount is acquired most finely, the parallax amount z in which s0 = smax in [Equation 7] is equal to an integer multiple of the pixel pitch of the image sensor, and a plurality of focus points are obtained. You can sample or combine images.

[Equation 7]
Parallax amount (z) = camera interval (s0) × imaging magnification = s0 × b / Z

<About the amount of parallax>
As seen above, the amount of parallax is expressed by the equation described in [Formula 7] in the case of a pinhole camera or a pan focus lens. On the other hand, consider the case of focusing with a monocular lens as shown in FIG. In this case, the entrance pupil of the lens has a certain area, and the distance between any two points in the pupil plane (sp) of the lens is s0. The maximum distance between any two points: s0 is the pupil diameter Ap. The parallax amount z2 is given by the equation shown in [Equation 8] where Z is the distance to the object point and f is the focal length. Also, when focusing on each photographic lens in the configuration of a multi-lens camera, it is expressed by the equation described in [Equation 8].

[Equation 8]
Parallax amount (z2) = imaging magnification (m) × distance between any two viewpoints on the pupil plane (s0) = f × s0 / (Z−f)

  The pupil plane shown in FIG. 9 and the camera plane shown in FIG. 10 are both entrance pupils that define the incident light flux, and play an equivalent role. In addition, when a lens of a multi-lens camera with a synthetic aperture method is extended for each focus position of a subject to perform focusing, the distance between the image plane and the lens: b is a function of Z, so that the parallax is [ Equation 8] is obtained. In general, a synthetic aperture method using a multi-lens camera uses fixed focus and pan focus lenses, and thus the parallax amount z is derived from the equation described in [Equation 7].

Since the parallax amount z between the two cameras is the amount of image movement when the lens is relatively shifted in parallel, the increase in the parallax amount from the in-focus state is equal to the blur amount. For example, in a state where the focus is set to infinity, a subject at a subject distance corresponding to one pixel of parallax is photographed with a blur radius of one pixel, and a subject at a distance of one pixel of parallax is focused. In total, an object at infinity is blurred with a blur radius of 1 pixel. When calculating the parallax amount z when focusing is performed by changing the position of the imaging surface with a single lens without performing the focusing by the synthetic aperture method, it is particularly closer to use the formula described in [Equation 8]. Although the error is reduced in the calculation of the distance parallax amount z, the equation described in [Expression 7] may be substituted.

  The act of setting the parallax amount z at an arbitrary distance to zero can be said to be focusing (focusing (focusing)). The action of focusing is to make the image plane coincide with the focus position in a normal lens. In the synthetic aperture method, a plurality of images are shifted by the amount of parallax so that the amount of parallax between the images of a plurality of cameras becomes zero.

  In the present invention, since the depth direction of the optical axis is replaced with parallax, in the function representing blur, the magnitude of blur is proportional to the amount of parallax. For example, a circular opening has a conical shape (cone shape).

<About the data acquisition interval>
An image is acquired from a camera in which x and y directions are arranged at equal intervals. This is achieved by sampling with imaging sensors arranged in two dimensions at equal intervals. On the other hand, the parallax amount z is obtained by being divided so that the pixel count (parallax amount / pixel pitch (P)) of parallax generated between the two cameras is linear (equal intervals). In other words, the focus position is set so that the parallax amount z is an integer multiple of the pixel pitch, and images in a plurality of focus states are synthesized by the synthetic aperture method. Thereby, a calculation error can be reduced. The pixel pitch may be the pixel pitch of the image sensor, or the pixel pitch of the output image output from each camera (single-eye imaging unit). In the latter case, it is “1” corresponding to an integer of the pixel pitch of the original image sensor for the multi-pixel processing.

<About the distance s0 between two points on the pupil plane>
Here, the number of times of focusing by the synthetic aperture method, that is, the resolution in the focus direction is determined. How to determine the distance s0 between two points on the pupil plane will be described.

In order to increase the number of data in the focus direction (the number of focus images to be synthesized by the synthetic aperture method), the parallax amount with s0 = smax or s0 = AP in the equations described in [Equation 7] and [Equation 8] is used. Calculate
A focus image obtained at a distance Z, where the parallax amount is an integral multiple of the pixel pitch of the image sensor, may be synthesized by the synthetic aperture method. This method is a method of generating a focus image at the most different subject distances Z. However, practically, it has been found that there is no problem even if thinning is performed to about a fraction. In particular, when the multi-lens cameras are arranged at equal intervals (including the M-sequence arrangement), the parallax amount z is s0 =
By setting the distance between adjacent cameras, it is not necessary to shift the subpixel image laterally (shifting by a shift amount of one pixel or less) at the time of focusing (focusing) by the synthetic aperture method. Since the resulting quantization error does not occur, it is possible to prevent the occurrence of blur due to the pixel pitch. As a result, the number of data is thinned out. For example, if the number of cameras is about s × t = 5 × 5, the number of data is thinned out to about 1/4 of the maximum number of data, but it has been found that the number is practically sufficient. In macro photography, it was found that there is no problem even if the number of data is reduced to about half.

<About the amount of parallax in a monocular lens>
When using a monocular lens to determine the parallax according to the equation described in [Equation 8] and acquiring image data with multiple focus, the distance between two points on any pupil: s0 is set to about Ap / 4, and the data is thinned out. There is no practical problem even if it is performed.

Since the value of the parallax amount z is distance: Z = infinity and parallax: z = 0, the coordinates of the parallax are preferably taken as shown in FIG. x and y are two-dimensional coordinates in the imaging surface, and z is a parallax amount. In FIG. 11, a multi-lens camera arranged in a two-dimensional grid at equal intervals is assumed, and the parallax amount z is set as s0 = interval between adjacent cameras in the equation shown in [Expression 7]. The four diagonal straight lines shown in FIG. 11 indicate that the cameras D (s, t) described in FIG. 10 are arranged at equal intervals in the s, t plane and 5 × 5 grids parallel to the s, t axes. It shall be. The five cameras with s = 0 are named cam1 to cam5. Cam1, cam5, cam2, cam4 above and below cam3 at the center of the camera (cam, origin of s, t coordinates) in a row of cam1 to cam5 in a 5 × 5 grid arrangement in the s, t plane. The size of parallax or blur occurring in the x, y plane depending on the arrangement (the size of the x, y cross section of the three-dimensional PSF kernel) is shown. It is shown that a parallax of ± 4 times and ± 8 of the pixel (P) is generated in x or y when the parallax amount z = 4. In this way, by dividing the depth direction (optical axis direction) so that the parallax amount z is equally spaced, a three-dimensional PSF kernel (see FIG. 12) representing blur due to camera arrangement or an arbitrary three-dimensional blur kernel The change in size (see FIG. 13) changes linearly to the amount of parallax, and the shape of the three-dimensional blur kernel changes depending on the position of the parallax z-axis along the optical axis. Disappear. Thereby, shift invariant, that is, data of a three-dimensional imaging state that can be expressed by a three-dimensional convolution can be acquired (formula described in [Equation 17]).

As shown in FIG. 7, the maximum distance between any two points in the s and t directions on the pupil plane (camera plane) is ca.
The distance is m1 and cam5, and the parallax amount z shown in FIGS. 12 and 13 is four times the distance between adjacent cameras. That is, since the data in the three-dimensional imaging state has s0 set between adjacent cameras, the number is thinned to ¼ from the maximum number of data. Since this number is sufficient for practical use, there is no problem.

FIG. 12 is a rewrite of the three-dimensional PSF kernel of FIG. 11 with z = 4 and in focus. The 3D PSF kernel is a blur kernel that is determined by the camera's optical hardware. The camera entrance pupil shape and the combined viewpoint position (including the pupil plane arrangement in the camera plane st plane (incident pupil plane) and the combined viewpoint arrangement) ) Figure 12 shows a 3D PSF representing blur
The three-dimensional shape of the kernel and the size of the blur kernel in the x and y sections are shown. x and y are two-dimensional coordinates in the imaging surface, and z is a parallax amount. In FIG. 12, similarly to FIGS. 10 and 11, a multi-view camera arranged in a two-dimensional grid at equal intervals is assumed, and the parallax amount z is set as [Equation 7] and s0 = interval between adjacent cameras. The size of the blur kernel changes linearly according to the amount of parallax. This figure shows a state in which the three-dimensional PSF kernel Bc is convolved with an object in focus with parallax z = 4 (pixels). The x, y cross section of the blur kernel is a two-dimensional distribution function representing blur in the x, y plane (imaging plane), and is also a function of parallax z in the present invention. For example, as described in [Equation 9], if f (x, y, z) is set, the function is normalized with an area of 1 as described in [Equation 10]. To obtain a function that does not blur at the focus position, a delta function, that is, a function of area 1 with a size of zero is used. A two-dimensional image can be blurred by convolving an arbitrary blur kernel with an image on the imaging plane (xy plane).

A distribution function whose spread changes in accordance with the parallax z in this case (proportionally in many cases) is hereinafter referred to as an x- and y-section of a three-dimensional blur kernel or simply a blur kernel, a blur function, a three-dimensional PSF kernel, and the like.

  FIG. 13 shows a state where the blur kernel B is convoluted with an object that is in focus with an arbitrary blur amount of blur kernel z = 4 (pixels). From comparison with FIG. 11 and FIG. 12, it can be seen that FIG. 13 is a three-dimensional blur kernel representing blur due to a lens having a circular aperture with a radius s0 on the entrance pupil plane. x and y are two-dimensional coordinates in the imaging surface, and z is a parallax amount. When the radius of the blur kernel is set to zero, an omnifocal image is generated. A detailed description of the blur generation algorithm will be described later.

Assuming multi-lens cameras arranged in a two-dimensional grid at equal intervals in the camera plane, the parallax amount z is the equation described in [Equation 7], where s0 = interval between adjacent cameras, adjacent camera interval 12 mm, focus Distance 5.34
Table 1 shows the parallax amount z, the distance Z to the subject, and the distance b between the image plane and the lens when mm and the pixel pitch P = 6 μm. The distance b between the lens and the image plane was determined by the lens imaging formula 1 / f = 1 / Z + 1 / b. In this way, when the cameras are equally spaced, the parallax is obtained from the interval between adjacent cameras, and when the parallax value used for image processing is determined at an interval such that the parallax is an integer multiple of the pixels, focusing by the synthetic aperture method (focusing) ) Can be set to an integral multiple of the pixel pitch in any camera, and the occurrence of quantization errors can be prevented. “b” indicates an image forming position by the image forming action of the lens, and is a distance from the lens to the image. db represents the amount of change of b. Strictly speaking, db is not equally spaced. Therefore, even when the blur processing of the present invention is performed on an image of a monocular camera, the amount of parallax is more equal than in the conventional method of acquiring data by moving the imaging sensor at equal intervals in the optical axis direction in the image space. It is desirable to obtain the subject distance to be obtained from the equation [Equation 7] or [Equation 8].

  On the other hand, imaging of a lens by a synthetic aperture is generally photography using a pan focus lens, and is irrelevant to an imaging formula as described above. Strictly speaking, when a lens is used instead of a pinhole, the position of the image plane also changes according to the position of the subject. The focal length of the camera lens with a focal length of 5.34 mm and F2.8 is adjusted so that the distance to the subject is about 2 m and focused on the subject distance. The change of is not large (change of about 10 to 20%). There is no problem considering this state as pan focus. Since the change in MTF is known, it is possible to store MTF data for each shooting distance in the camera in advance and correct image blur from the MTF data for each shooting distance.

The focus image at the distance Z corresponding to the parallax (z) is focused by the synthetic aperture method. For example, synthetic aperture images at 20 locations (parallax z = 1 to 20 (pixels)) shown in [Table 1] are generated. In FIG. 11, the parallax amount z is [Equation 7], s0 = interval between adjacent cameras, and an image focused on subject distance corresponding to nine parallaxes with parallax z = 0 to 8 pixels is synthesized by the synthetic aperture method. Represents what to do. Dividing the parallax with a resolution of less than a pixel is meaningless from the viewpoint of generating a blurred image. Further, when thinning out, it is important to thin out so that the parallax intervals are equal. When the number of pixels is increased, the pixel pitch is used.

When focusing with the synthetic aperture, the image of the surrounding camera is shifted in the xy plane in parallel with the amount of parallax with respect to the image of the camera at the center (center of s, t plane). The amount and the amount of parallax in [Table 1] match. When the parallax is zero, it is at infinity, and when it is 1 pixel, the focus is in front of 10.68 m. When there are 25 cameras, once 25 images are captured, an arbitrary focus image can be obtained by selecting an arbitrary parallax from [Table 1]. The image thus obtained is schematically shown by a plurality of vertical lines shown in FIG.

FIG. 13 is an explanatory diagram of blur due to the circular pupil of the normal lens. First, assume a multi-lens camera arranged in a two-dimensional grid at equal intervals in the camera plane. The parallax amount z is [Equation 7] and s0 = interval between adjacent cameras. The blur generated by the three cameras cam2 to cam4 arranged with the adjacent lens interval of 12 mm is, for example, when the focus is made by the synthetic aperture at z = 4 (pixel), and the parallax amount 1 in the optical axis direction intersecting at z = 4 (pixel) Is represented by two straight lines that generate a lateral shift of ± 1 pixel in a direction perpendicular to the increase / decrease. This is the parallax due to the three camera arrangements = the amount of blur. If it is assumed that the cameras are distributed innumerably in a circle having a diameter of cam2 to cam4, it is possible to handle a circular ordinary lens blur similarly. That is, a cone having a straight line intersecting at z = 4 (pixels) as a generating line corresponds to a blur of a lens having a lens pupil diameter of 24 mm. If the camera interval is replaced with the lens pupil radius, the lens blur of the circular pupil can be considered. Larger lens diameter blur and smaller lens diameter blur can also be given by a cone with the focus position at the apex. The blur blur kernel is given by a circle of diameter at each parallax of this cone. As will be described in detail later, the weight in the circular aperture can be arbitrarily changed, such as a uniform distribution or a Gaussian distribution.

<Bokeh shape of camera placement>
FIG. 12 assumes a multi-lens camera arranged in a two-dimensional grid at equal intervals, and the amount of parallax z is expressed by [Equation 7] and s0 = interval between adjacent cameras. Represents the shape. The blur shape with respect to the parallax in the case of a 5 × 5 arrangement with equal intervals as shown in FIG. 10 is shown. Since the focus direction needs only to have data so that the parallax is equally spaced, the black circle in the figure represents a blurred shape or a blur kernel. When this is expressed in the xy plane, it is as shown in FIG. The M series is as shown in FIGS. These are 3D PSF
Called the kernel. Arrows represent delta function weights. At the focal position, the strength of the arrow increases by a factor of several. When these blur kernels are integrated in the parallax direction (the direction is the same as the Z (large jet) direction in the depth direction of the object), the result is as shown in FIG. This is called the focus stack PSF kernel.

FIG. 18 shows the flow of processing from image capture. A plurality of focus images obtained by the synthetic aperture method are passed to an arbitrary blur algorithm. The contents of the arbitrary blur algorithm are described in FIG. 19, and a plurality of images are convolved with a blur kernel having a radius proportional to the parallax. These are added together to integrate the parallax of the optical system PSF (camera placement blur shape). Deconvolution is performed based on the result of integrating the parallax of the bokeh kernel.

  The relationship between FIG. 19 and the equation is shown in FIG. 20 and 21 show images obtained by changing the focus (parallax). 22 and 23 are explanatory diagrams for deriving the equation described in [Equation 18]. FIG. 22 explains that the calculation time can be shortened by approximating the equation [Equation 18]. In the present invention, the description based on the approximate expression of [Equation 18] is referred to as “blur generation algorithm” or “blur algorithm”.

As shown in FIG. 24, the blur processing unit 36 constituting the blur algorithm for generating an arbitrary blur has a circular opening having a blur radius proportional to the parallax from a desired focus position, or a plurality of multi-focus image data, or Convolution processing is performed using a two-dimensional section (two-dimensional distribution function representing blur) of an arbitrary blur kernel such as Gaussian. Here, “proportional to the parallax from the focus position” means, for example, that the convolution process is performed by increasing the radius of the two-dimensional section of the blur kernel in proportion to the parallax from the desired focus position. This means that a three-dimensional blur kernel as shown in FIG. 13 is used.

  The focus stack image calculation unit 37 obtains a focus stack image that is an intensity sum image of a plurality of image data processed by the blur processing unit 36. In addition, you may obtain | require an intensity | strength average. The PSF kernel calculation unit 38 calculates a focus stack PSF kernel corresponding to the focus stack of the pupil sampling point of the synthetic aperture (see FIGS. 14 to 17).

  The pupil sampling point represents the arrangement of the photographing lens 23 of the multiview camera 10. Here, since the arrangement of the photographing lens 23 is determined in advance, it is not necessary to calculate the focus stack PSF kernel. The focus stack PSF kernel is stored in advance in the RAM 29 and is read and used. FIGS. 17 and 25 show focus stack PSF kernels when the pseudo-random arrangement of the 16 photographing lenses 23 is a predetermined periodic pattern according to the M series.

  The image sharpening processing unit 39 performs deconvolution processing using a predetermined kernel on the focus stack image. As the deconvolution process, an in-focus (focused) object is obtained by performing a deconvolution process using the focus stack PSF kernel read from the RAM 29 on the focus stack image calculated by the focus stack image calculation unit 37. Generates a blur algorithm image in which an arbitrary blur is added to an out-of-focus (out-of-focus) object while maintaining a sharp image.

  The deconvolution process using the focus stack PSF kernel is a focus that is a two-dimensional distribution function that represents an integration result in the optical axis direction of the three-dimensional PSF that is a three-dimensional blur function of the single-eye imaging unit. This is a deconvolution process using two kernels: a stack PSF kernel and a focus stack blur kernel corresponding to a focus stack of an arbitrary blur kernel used when the blur processing unit 36 performs the convolution process.

The CPU 24 in FIG. 6 records the image data with an arbitrary blur on the recording unit 31 via the I / F 30. Note that a compression unit may be provided, and image data with arbitrary blurring may be recorded in a compression format such as the JPEG method. A plurality of multifocal image data obtained from each frame memory 22 may be stored in the recording unit 31. In this case, it is preferable to store a plurality of multifocal image data in association with image data generated by adding a blurred image. In the case of a multi-lens configuration as in the present embodiment, image processing after the synthetic aperture may be performed at any time if the images output from the image pickup devices 20 are stored. That is, the function of saving the raw data of the multi-view image is at least necessary for the camera, and the image processing can be performed offline. The captured image may be stored in the cloud computer, and the stored image may be a moving image or a still image.

  Here, the operation of the configuration of the multi-view camera 9 in which the photographing apertures 11 shown in FIG. 1 are two-dimensionally arranged will be briefly described with reference to FIG. When the power switch 14 is turned on, a through image is generated based on the image data obtained from any one single-eye imaging unit 40 and displayed on the LCD 15. (Alternatively, a synthetic aperture image of the images of the plurality of imaging units 40 may be displayed on the LCD 15 and the composition and focus may be determined while viewing this, or if the generation time of the blur algorithm image can be ignored, the blur is generated. The composition and focus may be determined using an algorithm image.) The composition is determined while viewing the through image, and the shutter button 13 is half-pressed. In response to this half-press operation, the CPU 24 controls AE / AF. The AE control is performed so that each single-eye imaging unit 40 has the same exposure. AF control is focus shift control. In this control, the CPU 24 controls the AF unit 26 to move the photographic lens 23 to the in-focus position so that each photographic lens 23 is completely out of focus.

  When the shutter button 13 is fully pressed as it is, the CPU 24 (FIG. 6) controls each single-eye imaging unit 40 to capture a plurality of images from a multi-eye camera, and generates a multi-focus image by the synthetic aperture method. Then, the multifocal image data group is taken into each frame memory 22, and the plurality of taken multifocal image data is sent to the image processing unit 25.

  The blur processing unit 36 of the image processing unit 25 in FIGS. 6 and 24 uses, for multi-focus image data, a circular aperture having a blur radius proportional to a desired focus position, or an arbitrary three-dimensional blur kernel such as Gaussian. To perform two-dimensional convolution processing. The focus stack image calculation unit 37 calculates a focus stack image, which is a pixel intensity sum image, for each of a plurality of image data subjected to the blur processing by the blur processing unit 36.

  The focus stack kernel calculation unit 38 reads a predetermined focus stack PSF kernel and focus stack blur kernel from the RAM 29 (or may calculate them one by one). The image sharpening processing unit 39 uses the focus stack PSF kernel read from the RAM 29 and the focus stack blur kernel to deconvolute the focus stack image calculated by the focus stack image calculation unit 37 and give image data with arbitrary blurring. (Image processing of the contents described in FIG. 19). The generated image data with blur is recorded by the recording unit 31.

  Here, the features of the image processing of the present invention will be briefly described. First, an arbitrary blur process is performed on a plurality of focused images, and then the intensity sum (average) of these images is obtained to obtain an intensity sum (average) image. There is no noise at this stage. Next, an image sharpening process is performed on the obtained intensity sum (average) image to generate an image with arbitrary blurring. The image sharpening corresponds to the focus stack of the pupil sampling point of the synthetic aperture. This is done by deconvolution of the “stack PSF kernel”. As this deconvolution kernel, a Fourier transform having no zero point is used to suppress the occurrence of restoration noise. As a result, it is possible to form an image to which blur with good photographic image quality is imparted. The focus stack may be performed while changing the focus state by the synthetic aperture method or the movement of the image sensor in the optical axis direction.

<Bokeh algorithm>
A specific blur generation method will be described using an example of a synthetic aperture method using a multi-lens camera. Specifically, the three-dimensional intensity distribution of the object to be imaged is Io (X, Y, Z). The Z axis is the optical axis direction of the photographing lens 23, and the x and y directions are the horizontal and vertical directions of the image sensor 20. The z-axis is the amount of parallax (parallax = camera interval × focal length / Z, Z is the distance from the lens pupil position to the subject). The direction xy in the imaging plane of the imaging sensor is photographed at a magnification as can be understood from the formula “x = X × f / Z, y = Y × f / Z” as described above. An image taken with a camera in which the third-order distribution of the point image intensity distribution representing the degree of lens blur is represented by PSFc (x, y, z), with the focus position set to the parallax zr along the optical axis of the camera. Ic (x, y, zr) is a projection of Ic (x, y, z, zr) shown in the equation described in [Equation 11] onto the x, y plane, and the equation is an equation described in [Equation 12]. Indicated by PSFc (x, y, z) in the equation of [Equation 11] is represented by a function that has a conical function along the optical axis in the case of a circular aperture, for example, so that the focal position of the apex of the cone is equivalent to the parallax zr. be able to. An image obtained by integrating the focus position in the depth direction of the scene to be photographed is expressed by the equation [Equation 6]. The integration range (zmin to zmax) is approximately matched with the region where the object exists. Io (x, y) is equivalent to a picture (omnifocal image) in which the scene is projected onto the image plane by an ideal pinhole camera without blur.

For example, in the case of a circular aperture, PSFc (x, y, z) in the equation described in [Equation 11] is represented by a function that has a conical function along the optical axis so that the apex of the cone is at the focus position zr. be able to. An image obtained by integrating the focus position in the depth direction of the scene to be photographed is expressed by the equation described in [Equation 13]. The integration range (z min to z max) is generally matched with the region where the object exists. Io (x, y) is equivalent to a picture (omnifocal image) projected on the image plane by an ideal pinhole camera without blur.

Since Ic_pn (x, y) is a convolution of PSFc_pn and Io (x, y) from the equation described in [Equation 13], Io (x, y) is obtained from the equation described in [Equation 15] (Io (x, y) is not used for blur generation according to the present invention but is calculated for reference). Here, PSFc_pn is expressed by the equation described in [Equation 14], and is an integral of PSFc (x, y, z) in the parallax z direction. This is called the focus stack PSF kernel. Note that the subscript c indicates the state of the camera. pn indicates pan focus.

Consider the integration range described in the equation [14]. PSFc uses the z-axis origin as the z-axis origin (see FIG. 27, for example). The range of the object is infinity = zmin to close distance = z ma
Since x, the convolved PSFc may exist in the range of PSFc (x, y, z) to PSFc (x, y, z−z max) illustrated in FIG. 27. The integration range of the focus stack is “−z max˜0” for an object at infinity, and “
0 to z max ”. The results of these integrations are preferably averaged when the PSF shape is asymmetrical at the front and rear pins, for example, in the case of the examples described in FIGS. On the other hand, it was confirmed by a live-action video that the integration range described in the equation [Equation 14] is sufficient. This integration range is optimally about “−z max / 2 to + z max / 2” for the intermediate portion between the close distance and infinity, so the integration range may be variable according to the distance to the subject. As described above, in short-distance shooting, z max may be the farthest position from the camera within the range where refocusing is possible in the subject.

In the same way as the equation described in [Equation 11], the blur function PSFc (x, y, z) is changed to PSFa (x, y, z) and the parallax z1 corresponding to an arbitrary focus position is focused and photographed. The image is shown by the equation described in [Equation 16]. The equation described in [Equation 17] is a three-dimensional image obtained by convolving PSF three-dimensionally, and indicates a luminance distribution that can be acquired while shifting the focus position under the optical condition of PSFc. This is three-dimensional data.

The equation of [Equation 18] is obtained from the equation of [Equation 17]. The part deconvoluted by PSFc_pn (x, y) in the equation [18], that is, “A” in the equation [19] is obtained by applying PSFa (x, It is shown that y, z-zr) is convolved and further integrated with respect to parallax z. This corresponds to the processing of the blur processing unit 36 and the focus stack image calculation unit 37 of the above-described embodiment.

It should be noted that the equation described in [Equation 20] can be obtained from the equation described in [Equation 18]. For this reason, images Ic_p (x, y, z) at a plurality of focus positions are picked up by a synthetic aperture method or the like, and this is subjected to three-dimensional deconvolution by PSFc (x, y, z) to obtain Io (x, y , z) Obtain the intensity distribution of the object, and perform two-dimensional convolution of PSFa (x, y, z-zr) (arbitrary three-dimensional blur kernel) set to the parallax zr corresponding to the arbitrary focus position. The result may be integrated with the parallax z and projected onto the x and y planes to obtain an arbitrary blurred image and an image refocused at an arbitrary focus position.
In this case, as a means for generating a blurred image, a three-dimensional deconvolution processing unit 45, a two-dimensional convolution processing unit 46, and an image generation unit 47 may be configured as shown in FIG. The three-dimensional deconvolution processing unit 45 regards a multifocal image (an image obtained by focusing at a plurality of positions) acquired from the image acquisition unit 48 and having parallaxes at equal intervals as three-dimensional data. Deconvolution processing is performed to obtain the intensity distribution of the object. The two-dimensional convolution processing unit 46 performs two-dimensional convolution processing using an arbitrary three-dimensional blur kernel on an image that has been subjected to three-dimensional deconvolution processing. The image generation unit 47 projects an image that has been subjected to the two-dimensional convolution process onto a plane and generates an image with arbitrary blurring.

Note that the right side of the equation described in [Equation 18] is a desired three-dimensional blur kernel when the measured value Ic_p (x, y, z) and the parallax zr corresponding to the device-specific focus setting value are in focus: PSFa (x, y, z-zr), focus stack PSF kernel: PSFc_pn (x, y). The subscript a indicates the state of the camera. In the following, constants are omitted.

PSFa_stk (x, y) in the equation [Equation 18] is expressed by the equation [Equation 21] (see FIG. 27). Similarly to PSFc, the integration range of the equation described in [Equation 21] has the relationship shown in FIG. 29, and the integration range of the focus stack is “−z max to z max” from FIG. This may also be changed to an optimum range such as “−z max / 2 to z max / 2” when focusing on an object at an intermediate distance between the close distance and infinity. There is no problem with “−zmax to zmax”.

The equation of [Equation 21] can be confirmed experimentally as follows. When PSF is represented by a Gaussian distribution, PSFa (x, y, z-zr) for newly generating a blur with respect to PSFc (x, y, z) of the camera is [Equation 22] and [Equation 23], respectively. It is shown by the formula of C1 (z) and c2 (z) are standard deviations of the Gaussian distribution and are expressed by the equations [Equation 24] and [Equation 25]. The radius of a Gaussian blur kernel is proportional to the standard deviation. zr indicates a set focus position when an arbitrary blurred image is generated, and indicates a position where a light cone of an arbitrary three-dimensional blur kernel converges. That is, as shown in FIG. 30, the three-dimensional blur kernel is a point at which the cone of light (conical blur shape or blur kernel) is most narrowed (converged) at a distance zr to be focused.

The trace 2 shown in FIG. 31 shows the standard deviation c1 (z) of PSFc (x, y, z), and the trace 3 shows the standard deviation c2 (z-zr) of PSFa (x, y, z-zr). Here, the parallax corresponding to defocus is 5 pixels (for example, zr = 5 (pixels)). The horizontal axis shown in the figure indicates the amount of parallax z taken in the direction of the optical axis. The vertical axis shows the standard deviation. Trace 1 shows the standard deviation cca (z, zr) of PSFca (x, y, z, zr) in the equation [Equation 26]. The standard deviation cca (z, zr) is given by the equation [Equation 27].
In this case, the object is at z = 0. Although the object takes an arbitrary value of z according to the unevenness, the light of PSFc (x, y, z) is observed in Ic_p (x, y, z) observed as a result of the convolution of the equation [17] The center of the cone is always located on the object surface (see FIG. 30). That is, the blur distribution acting on the object surface is PSFca (x, y, z, zr) by the convolution of the formula [Equation 19].

On the other hand, the blur radius PSF3 (x, y, z) of the formula [Equation 29] is set as the blur radius at the desired defocus position.
Assuming that c3 (z) = k3 = constant, the standard deviation of the combined blur radius PSFca3 (x, y, z) is shown by trace 4 in FIG. The same figure shows the standard deviation c1 (z) of PSFc (x, y, z) and the standard deviation c2 (z) of PSFa (x, y, z). The synthetic blur radius PSFca3 (x, y, z) is expressed by the equation described in [Equation 28], and the standard deviation is expressed by the equation shown in [Equation 30]. Also, the blur radius c3 (z) = k3 is shown as a trace 6.

  For example, the coefficients k1 = 2, k2 = 1, and k3 = 5. As shown in FIG. 32, the value of trace 1 is 5 when z = 0. Since the trace 3 in FIG. 31 is k2 = 1 and the focus is 5, the standard deviation is 5 at z = 0. Under this condition, the minimum value of the trace 1 is about 4.5, which is about 10% smaller than 5, but the trace 1 can be regarded as a translation of the trace 2 in the z direction. Since the information in the z direction is lost after the integration in the z direction of the formula [19] is performed, the integration results in the z direction of the trace 1 and the trace 2 can be regarded as the same. This means an approximation of the equation of [Equation 18].

As described above, the function representing the PSF is a Gaussian function. As a result of simulating the function representing the circular aperture and the function representing the pupil sampling point of the synthetic aperture method, it was found that substantially the same effect can be obtained. In other words, this method can give images of various blur shapes to hardware with various blur functions.

  Thus, in the present invention, a desired image can be obtained directly from an acquired image. In the present invention, the three-dimensional imaging state is represented by a three-dimensional convolution model based on the three-dimensional luminance distribution of the object and the three-dimensional blur function, as shown by the equation described in [Equation 17]. Therefore, a desired two-dimensional image can be obtained by deconvolution calculation of the two-dimensional image of the formula [18].

The multi-lens camera 10 performs camera calibration by a six-point method or the like. Measurement values are used for constants such as the position (s, t) of the lens center in the camera plane and the focal length of each photographing lens 23. Note that an error amount such as a tilt of the optical axis of the camera is removed by camera calibration. The tilt of the optical axis is corrected as a distortion component.

In FIG. 10, the center of the photographing lens is located on the camera plane (s, t), and 25 (5 rows × 5 columns) single-eye imaging units are denoted by D (s, t). The object plane and the camera plane are parallel, and the object plane is focused at a distance Z corresponding to an arbitrary parallax z from the imaging plane by the synthetic aperture method.

  The depth of focus of the photographing lens 23 of the multi-lens camera 10 of the present embodiment is deep, and the photographing lens 23 may be considered as pan focus from a short distance to infinity. When the photographing lens 23 is bright and the focal depth is shallow, and AF for each photographing lens 23 (ordinary focusing before and after a normal lens) is required, AF may be applied to the subject to be focused in advance for each photographing lens 23. . In this case, when refocusing, it is desirable to compensate for the degradation of MTF from known MTF and PSF data.

  Regardless of the synthetic aperture method, the image of a plurality of focus positions can be obtained by adjusting the focus of the normal monocular photographing lens 23 (adjustment by converting the aerial image and the relative position of the image sensor 20 in the optical axis direction in the image space). In the case of obtaining the parallax amount, the parallax amount is obtained from the equation described in [Equation 7] or [Equation 8] from the pupil radius instead of the lens interval used when obtaining the parallax amount in the case of a multi-lens camera. Determine the focus distance so that For example, in the case of a 35 mm full-size camera, when f = 50 mm, for example, the F-number is 2, and the pupil radius is 50 / (2) = 25 mm. As described above, this value “25” is ¼. It should be about. If an image at each focus position is acquired and an image group with an equal interval of parallax is obtained, the subsequent blur algorithm processing is the same as when the multi-eye camera synthetic aperture method is used. (Focus adjustment is a method such as changing the position of the imaging sensor in the optical axis direction, changing the aberration of the lens, changing the image plane, and acquiring an image at each focus position, etc. There is.)

<Arbitrary blur generation>
The point that processing is performed using a multifocal image group adjusted to a plurality of focus positions by the focusing of the synthetic aperture method will be described below.

[Bokeh generation kernel: circular aperture]
When defocus blur of a circular aperture is generated as the blur generation kernel, the three-dimensional blur kernel PSFa (x, y, z) is expressed by the equation described in [Equation 31]. The radius ra (z) of the blur kernel that is arbitrarily set is given by the equation described in [Equation 32]. A desired blur size can be obtained by arbitrarily setting the constant Apa. Further, by giving an offset, for example, zr in the z direction (replace “z” with “z-zr”), it is possible to focus on an arbitrary focus position zr.

[Bokeh generation kernel: Gaussian aperture]
In the case of Gaussian blur, the three-dimensional blur kernel PSFgss (x, y, z) expressed by [Equation 33] is used. C is a constant. The size of the Gaussian blur can be adjusted by arbitrarily setting σa according to the equation described in [Equation 34].

[Arbitrary viewpoint]
If an arbitrary viewpoint three-dimensional PSF kernel PSFshift (x, y, z) is used, an all-focus image of an arbitrary viewpoint can be obtained as shown in FIG. Here, the constants axs and ays are arbitrarily set, and the viewpoint is arbitrarily set. (Note that since the origin of the s, t coordinates is the composite viewpoint, the same effect can be obtained even if the camera arrangement is reset to the origin of any s, t coordinates and the 3D PSF kernel is obtained from the amount of parallax. In the case of 14, 15, and 16, the s and t coordinate origins correspond to the center camera position.)

Further, an image Ic_shift_p (x, y, z) measured by focus scanning is expressed by an equation described in [Equation 34] instead of the equation [Equation 17].

An image refocused to the parallax z1 of the viewpoint of the formula [Expression 35] including the blur of PSF and PSFa (x, y, z) indicating an arbitrary blur is expressed by the expression described in [Expression 37]. . The focus stack of the PSF with the viewpoint shifted is expressed by the equation described in [Equation 38].

The lens arrangement of the multi-eye camera 10 represents a pupil sampling point. The pupil sampling point acts as a focus stack PSF kernel when focus stacking is performed. The focus stack PSF kernel is represented by PSFc_pn (x, y) in the equation described in [Equation 14]. The pupil sampling point before the focus stack is PSFc (x, y, z). Since the desired image is obtained by the equation described in [Equation 18], there is no zero point in the amplitude of the Fourier transform of PSFc_pn (x, y) in order to accurately perform the deconvolution of the focus stack kernel PSFc_pn (xy). desirable. This is because deconvolution can be performed by applying the inverse of the Fourier transform of PSFc_pn (x, y) and then performing the inverse Fourier transform. The deconvolution of the equation described in [Equation 18] is represented by the equation described in [Equation 39].

By the way, it is desirable that the amplitude of the Fourier transform of PSFa_stk (x, y) and PSFc_pn (x, y) has no zero point. PSFa (x, y) is preferably a circular aperture or a Gaussian function. When a circular aperture is used for PSFa (x, y), PSFa_stk (x, y) is expressed by [Equation 41] and its Fourier transform is expressed by [Equation 42]. U and v are expressed by [Equation 43] and [Equation 44].

The amplitude of the Fourier transform of PSFc_pn (x, y) needs to have no zero point, and the arrangement of the taking lens 23 is a circumferential arrangement with many arrangements including a predetermined pattern using vertical and horizontal M-sequence pseudo-random number sequences, The rectangular layout and grid layout also satisfy this condition. The focus stack kernel with the M-sequence random arrangement is, for example, as shown in FIG. There is no zero point in the amplitude of the Fourier transform of this kernel, and there is no inconvenience such as divergence due to zero division in the equation [39]. Also, since the change in amplitude is about 20 times, noise of a specific frequency is not picked up.

  For the calculation of the equation shown in [Equation 19], the equation shown in [Equation 40] is actually used using Fourier transform. The Fourier transform of a Gaussian function is represented by a Gaussian function, and the Fourier transform of a circular aperture is represented by a Bessel function.

<First Embodiment>
[Application to blurring algorithm of annular optical system]
The example in which the blur algorithm is applied to the multi-lens camera 9 in which the photographing aperture 11, the photographing lens 23, and the like are two-dimensionally described has been described above. However, as described with reference to FIG. An example in which the above-described blur algorithm is applied to the multi-lens camera 10 having an annular optical system in which the photographing lens 23 and the like are arranged circumferentially will be described.

  In this embodiment, when blur that does not fail by the blur algorithm, that is, circular blur, is obtained, the slice based on the radius of the pupil centered on the virtual optical axis of the objective lens is zero at approximately 80% or more of the declination angle. It has been experimentally found that it is not necessary.

In FIG. 33 and FIG. 34, a description will be given of a case where an annular pupil lens is used for the blur algorithm. FIG. 33 shows a PSF (three-dimensional distribution of a point image intensity distribution indicating the degree of lens blur) having an intensity distribution of a difference between two Gaussian distributions. z indicates the amount of parallax. x and y indicate the size of the blur. z is a value obtained by multiplying the distance between two points on the pupil by the imaging magnification, and is a function of the distance to the object. In a pinhole lens, the focal length is f, and the distance Z from the pinhole is an object. FIG. 35 is a perspective view of the difference intensity distribution of the Gaussian distribution. The inner Gaussian distribution is expressed by the equation described in [Equation 45]. The outer Gaussian distribution is expressed by the equation described in [Equation 46]. These differences are expressed by the equation shown in [Equation 47] indicating a zonal Gaussian distribution.

  The reason why the annular zone may be regarded as a Gaussian distribution in this way is to convolve a circular kernel or a Gaussian kernel in the blur algorithm. Convolution of these results in a Gaussian distribution even if it is a uniformly distributed ring zone, so it can be considered as a Gaussian ring zone from the beginning. Note that even if a Gaussian distribution or a uniform distribution is convolved with the Gaussian distribution, it can be regarded as a Gaussian distribution.

  If it is assumed that the ring body is composed of two Gaussian distributions with different variances as described above, convolution by an arbitrary blur kernel can be easily explained as in the case of the prior application of the blur algorithm.

FIG. 33 is a profile corresponding to the equation described in [Equation 45], in which trace (5) indicates the inside of a gaussian blur of a Gaussian distribution. Trace (6) is a profile corresponding to the equation described in [Equation 46] indicating the outer side of the gaussian blur of the Gaussian distribution. A profile indicating an arbitrary blur function is a blur function in which the diameter of the Gaussian distribution changes in proportion to the parallax z in the trace (7). Arbitrary Gaussian blur is expressed by the equations described in [Equation 33] and [Equation 34]. x and y are two-dimensional coordinates on the imaging surface, and here indicate the size of the imaged blur. The parallax z is relative display, and the focus position of the optical system is parallax = 0. The focus position of the optical system being zero parallax means that focusing (focusing) is performed on an object at a distance Z so that parallax = 0 by a multi-lens camera or a monocular camera. The unit of the parallax amount is, for example, a pixel count of the image sensor.

The results of performing the two-dimensional convolution of the trace (7) in the x and y planes on the traces (5) and (6) in FIG. 33 are traces (3) and (4), respectively. Traces (1) and (2) will be described with reference to FIG. In FIG. 34, trace (5) is a profile corresponding to the equation described in [Equation 46], showing the inside of the gaussian blur of the Gaussian distribution. Trace (6) is a profile corresponding to [Equation 46] indicating the outside of the annular blur of the Gaussian distribution. A profile indicating an arbitrary blur function is a blur function in which the diameter of the Gaussian distribution changes in proportion to the parallax z in the trace (7). In this case, the blur function has a minimum radius when the amount of parallax is shifted by 5. Therefore, when the blur algorithm is used, the blur function is such that a picture in focus can be obtained at the focus position corresponding to the parallax amount = 5. The two-dimensional convolutions in the x and y planes of the trace (7) and the traces (5) and (6) are the traces (1) and (2).

  On the other hand, the traces (3) and (4) are traces (8) in the x and y planes indicating the Gaussian blur having a constant radius regardless of “z” in FIG. 34 in the traces (3) and (4) in FIG. It is the result of two-dimensional convolution with. Since the traces (3) and (4) in FIG. 33 almost overlap each other when the traces (1) and (2) are slightly shifted to the z-axis, the integration results regarding the parallax z of the traces (3) and (4) are as follows: This is almost the same as the result of integration on the parallax z of the traces (1) and (2). In other words, by the two-dimensional convolution of the blur function by the trace (7), the blur equivalent to the result of the two-dimensional convolution of the Gaussian blur having a constant radius is focused on the traces (3) and (4) of FIG. The position and parallax = 0 are given in the vicinity of the trace (1).

  On the other hand, in the convolution by the blur function of the trace (7) in FIG. 34, the object having a distance that focuses on the position of parallax = 5 does not blur. With this effect, it is possible to focus on an object at a subject position corresponding to an arbitrary parallax. This is similar to the solid Gaussian blur optical system described above. It should be noted that the focus position of the profile of the optical system in FIGS. 33 and 34 is one point (delta function), not a ring-shaped blur. For this reason, after convolution with traces (7) and (8) in FIG. 34, a disk-shaped Gaussian blur occurs at parallax = 0, that is, at the best focus of the optical system (including the best focus due to the synthetic aperture of the multi-lens camera). It is to be. Of course, in other cases where the parallax z is other than zero, the zonal blur remains (the story before focus stacking, which will be described later).

  Here, we are talking about blurring of the zonal optical system, but if we consider that the radius of the inner and outer rims of the annular zone is basically determined by two Gaussian distributions, the blurring algorithm will be effective as in the case of solid. I understand that.

  FIGS. 35 to 37 are bird's-eye views of the ring body strength distribution based on two Gaussian functions and their differences. The x and y cross-sectional intensities at a parallax z with a blur function are shown. The outer Gaussian distribution, inner Gaussian distribution, and ring Gaussian distribution are shown.

  38 to 44 are simulations of a blur algorithm when the annular optical system is used. [Table 2] shows the parallax amount z (pixel) and the distance to the subject when the focal length is “5 mm”, the pixel pitch of the image sensor is “6 μm”, and the distance between two points on the pupil plane is “12 mm”. The relationship of Z (mm) is shown (z = s * f / Z).

When the outer diameter of the annular optical system pupil is 48 mm, the inner diameter is 43.2 mm, the resolution chart is placed at a distance of 970 mm, and the focus is 10.7 m, the optical blur image of the optical system alone is left The blur algorithm image of the blur algorithm is arranged on the right. 39 shows the size of the blur kernel of the blur algorithm on the pupil plane of 48 mm, FIG. 40 shows the size of the blur kernel of the blur algorithm on the pupil plane with a diameter of 36 mm, and FIG. 41 shows the size of the blur kernel of the blur algorithm on the pupil. In FIG. 42, the size of the blur kernel of the blur algorithm is 12 mm on the pupil plane, and in FIG. 43, the size of the blur kernel of the blur algorithm is 6 mm on the pupil plane. In FIG. 44, the size of the blur kernel of the blur algorithm is 1.2 mm in diameter on the pupil plane. FIG. 45 is a right view of an image obtained by a blur algorithm in a state where there is no blur with the focus position of the blur kernel as the chart position. The left figure shows a state in which the blur algorithm is not used with the chart placed at the focus position (Z = 10.7 m) of the optical system. As is apparent from these figures, the ring algorithm peculiar to the circular opening is completely eliminated by the blur algorithm. However, in FIG. 39, the size of the blur kernel of the blur algorithm is φ48 mm on the pupil plane, which is equal to that of the optical system, and thus some artifacts are generated. However, if it is less than that, there is no problem. This gives great freedom to the design of the optical system. In addition, since there is no restriction on the width of the annular zone aperture, the optical system can be miniaturized while obtaining a large blur. Specific examples will be described later. In the figures after FIG. 39, the blur given by the optical or algorithm is a blur calculated by simulation, but these are given to the out-of-focus object, and the in-focus object is not blurred. Please be careful. Since the purpose of the simulation is to determine whether the blurred image is good or bad, the simulation of the in-focus object is omitted.

The amount of blur generated by these blur kernels may be considered in proportion to the ratio of the blur kernel size on the pupil plane to s = 12 mm. That is, when s = 12 mm, Z = 50000 mm and Z = 10000 mm, resulting in a blur of translation of 1 pixel with respect to 1 pixel of parallax (z). On the other hand, with a blur of 24 mm in diameter on the pupil plane, Z = 5000 mm and Z = 10000 mm, and a blur kernel represented by a cone in which a blur with a radius of 1 pixel occurs with respect to 1 pixel of parallax (z). In a blur with a diameter of 48 mm on the pupil plane, Z = 5000 mm and Z = 10000 mm, and a blur kernel represented by a cone in which a blur with a radius of 2 pixels occurs with respect to 1 pixel of parallax (z). In this simulation, the optical system is an annular aperture, and the intensity distribution in the aperture is uniform. In addition, the x and y cross-sections of the blur kernel with an arbitrary blur amount were also made uniform. When the PSF of the optical system and an arbitrary blur kernel are two-dimensionally convolved, the zonal intensity distribution becomes a mountain shape and can be regarded as Gaussian. Therefore, a function obtained by integrating this with parallax z is described in [Equation 48]. It becomes the following formula. Here, the integration result when the outer and inner radii are close can be regarded as the same as the integration result of the solid Gauntian aperture and the focus stack. That is, the Gaussian blur is softer than the circular aperture blur of the reference FIGS. Deconvolution can be performed as usual with the focus stack (focus stack PSF kernel) of the optical system PSF and the focus stack (focus stack blur kernel) of the blur kernel. Consider other patterns.

[Table 3]
Optical system Bokeh kernel Convolution result Ring Gaussian distribution Uniform distribution ≒ Gaussian distribution Gaussian distribution Gaussian distribution ≒ Gaussian distribution Uniform distribution Uniform distribution ≒ Gaussian distribution Uniform distribution Gaussian distribution ≒ Gaussian distribution

[Table 4]
Optical system Bokeh kernel Convolution result Solid Gaussian distribution Uniform distribution ≒ Gaussian distribution Solid Gaussian distribution Gaussian distribution ≒ Gaussian distribution Solid uniform distribution Uniform distribution ≒ Gaussian distribution ~ Uniform solid distribution Uniform Gaussian distribution ≒ Gaussian distribution ~ Uniform distribution

  As shown in [Table 4], even when the optical system is not a ring body and is solid, even if the same processing is performed, there is no major failure. Needless to say, in the frequency space (the blurred kernel is solid), the equation described in [Equation 42] is used.

<Ring arrangement or opening>
Annular aperture and annular camera arrangement is not limited to a circle, but an annular aperture surrounding a virtual optical axis or a synthetic optical axis, or when an annularly arranged camera is subjected to two-dimensional convolution with a blur algorithm Since there is a premise that it always has an annular Gaussian distribution, it can be regarded as an annular Gaussian distribution (before the two-dimensional convolution). An annular Gaussian distribution is seen as the difference between two closely spaced Gaussian distributions as seen above. The PSF focus stack of an optical system with an annular aperture or an annular camera arrangement is essentially the same as the original solid Gaussian PSF focus stack. The solid Gaussian distribution is known to be able to generate arbitrary blur with the blur algorithm, and as a result, arbitrary blur can be generated by combining the blur algorithm with the annular aperture of the arbitrary shape and the annular camera arrangement. I knew it was possible.

<Simulation of blurred image of annular optical system>
Annular blur was simulated as follows. In FIG. 38, the PSF of the optical system displays the parallax amount in pixels when the camera interval or the distance between two points on the st plane is s = 12 mm. Since the pupil of the annular optical system has an outer diameter of 48 mm and an inner diameter of 43.2 mm, the radius of the blurring of the outer annular zone is 20 pixels when the parallax amount is 10 pixels. This is the PSF of the optical system shown in FIG. The chart of the object is Z = 10.7m, parallax amount 19pixel
The optional blur kernel was a circular aperture with a diameter of 36 mm on the pupil. A blur kernel having a parallax amount of 10 pixels and a diameter of zero to 30 pixels is obtained. Parallax amount z = 20pixel
The focus position of the PSF of the optical system is positioned from 0 to 0 pixel, and two-dimensional convolution of x and y is performed. Actually, since the object has only z = 19 pixels, it is sufficient to perform convolution only at this position. In this way, 20 so-called focus swept images that are focused on a plurality of focus positions are obtained. An x, y two-dimensional convolution is performed by aligning an arbitrary three-dimensional blur kernel having an arbitrary blur diameter with a focus position at an arbitrary focus position. This time, x, y two-dimensional convolution was performed focusing on the parallax amount z = 9 pixels. Then integrate these with z. (Projection on x, y plane) A so-called focus stack image is obtained. Deconvolution processing is performed on the frequency spectrum obtained by Fourier transform of the focus stack image using the filter of the formula described in [Equation 42] (filtering + inverse Fourier transform).

<Multi-view configuration>
Below, the possibility of multi-viewing is examined. 49 to 64 show blurred images of a multi-lens camera with a square arrangement. The left image in each figure shows an optical blur image of only the optical system, and the right image shows a blur algorithm image after image processing by the blur algorithm. The camera arrangement for outputting the blur algorithm image shown in each figure will be described below.

[Example of placing a pupil on a square line]
(1) As shown in FIG. 46 (4), the left image shown in FIG. 49 is a composite generated using an image captured by a multi-lens camera in which the pupils 50 of the respective optical systems are arranged on a square line without any gap. It is an opening blurred image. The pupil 50 has a square arrangement with a pitch (p) of 12 mm and has a circular shape with a diameter (k) of 12 mm. The right image shown in the figure is the same optical system, and as shown in FIG. 46 (3), a circular blur algorithm image in which the size of the blur kernel 51 is set to a radius (r) of 18 mm on the pupil plane st plane. It is. There is no problem with the bokeh in the image on the right.

(2) As shown in FIG. 46 (5), the left image shown in FIG.
k) A synthetic aperture blurred image generated by using an image captured by a multi-lens camera having a circular shape of 10.8 mm and arranged on a square line at 12 mm pitch (p) (square arrangement). The right image shown in the figure is the same optical system, and as shown in FIG. 46 (3), a circular blur algorithm image in which the size of the blur kernel 51 is set to a radius (r) of 18 mm on the pupil plane st plane. It is. The circular opening that becomes the pupil 52 has a diameter of 90% with respect to the pitch (p). Some blurring artifacts appear in the blur on the right side of the image.

(3) As shown in FIG. 46 (5), the left image shown in FIG.
It is a synthetic aperture blurred image generated using an image captured by a multi-lens camera arranged in a 6 mm circle and 12 mm pitch (p) on a square line. As shown in FIG. 46 (3), the image on the right side shown in FIG. 46 is a circular blur algorithm image in which the size of the blur kernel 51 is set to a radius (r) of 18 mm on the pupil plane st plane. It is. The circular opening that becomes the pupil 52 has a diameter of 80% with respect to the pitch (p). Some blurring artifacts appear in the blur on the right side of the image, which is the limit.

  As described above, the circular aperture in which the pupils are arranged on the square line is not rough as shown in FIG. 46 (5), and is dense regardless of the declination as shown in FIG. 46 (4). If the blur generation radius (r) is 1.5 p or less, the blur generation radius does not reach the opening, so there is no failure.

Next, FIGS. 52 to 55 are examples of blur generation at an arbitrary blur generation radius.
(4) The left image shown in FIG. 52 has a side length (l) of 12 m as shown in FIG. 46 (1).
It is a synthetic aperture blurred image generated by using an image captured by a multi-lens camera having a pupil 53 of each optical system arranged on a square line with a square of m and a pitch of 12 mm (p). The right image shown in the figure is the same optical system, and as shown in FIG. 46 (3), a circular blur algorithm image in which the size of the blur kernel 51 is a radius (r) of 24 mm on the pupil plane st plane. It is. The image on the right side is a simulation image of a blur amount change without any problem with respect to the blur.

(5) As shown in FIG. 46 (1), the left image shown in FIG. 53 has a side length (l) of 12 m.
It is a synthetic aperture blurred image generated by using an image captured by a multi-lens camera having a pupil 53 of each optical system arranged on a square line with a square of m and a pitch of 12 mm (p). The right image shown in the figure is the same optical system, and as shown in FIG. 46 (3), the size of the blur kernel 51 is a circular blur algorithm image with a radius (r) of 18 mm on the pupil plane st plane. It is. The image on the right side is a simulation image of a blur amount change without any problem with respect to the blur.

(6) As shown in FIG. 46 (1), the left side image shown in FIG. 54 has a side length (l) of 12 m.
It is a synthetic aperture blurred image generated by using an image captured by a multi-lens camera having a pupil 53 of each optical system arranged on a square line with a square of m and a pitch of 12 mm (p). The right image shown in the figure is the same optical system, and as shown in FIG. 46 (3), a circular blur algorithm image in which the size of the blur kernel 51 is a radius (r) of 12 mm on the pupil plane st plane. It is. The image on the right side is a simulation image of a blur amount change without any problem with respect to the blur.

(7) As shown in FIG. 46 (1), the left image shown in FIG. 55 has a side length (l) of 12 m.
It is a synthetic aperture blurred image generated by using an image captured by a multi-lens camera having a pupil 53 of each optical system arranged on a square line with a square of m and a pitch of 12 mm (p). The right image shown in the figure is the same optical system, and as shown in FIG. 46 (3), a circular blur algorithm image in which the size of the blur kernel 51 is set to a radius (r) of 6 mm on the pupil plane st plane. It is. The image on the right side is a simulation image of a blur amount change without any problem with respect to the blur.

  In the following, FIGS. 56 to 58 see the limit of the size of a square opening arranged on a square line. There is no failure when the rectangular opening serving as the pupil 53 is 80% one side long with respect to the pitch (p). The blur generation radius is a radius (r) of 1.5p that does not cover the rectangular opening. See (1), (2), and (3) in FIG.

(8) As shown in FIG. 46 (2), the left image shown in FIG.
It is a synthetic aperture blur image generated using an image captured by a multi-lens camera having a pupil 54 of each optical system arranged on a square line at a square of 8 mm and at a pitch of 12 mm (p). The right image shown in the figure is the same optical system, and as shown in FIG. 46 (3), a circular blur algorithm image in which the size of the blur kernel 51 is set to a radius (r) of 18 mm on the pupil plane st plane. It is. The rectangular opening serving as the pupil 54 has a side length of 90% with respect to the pitch (p). There is no problem for the right image.

(9) The left image shown in FIG. 57 has a side length (l) of 9.6, as shown in FIG. 46 (2).
It is a synthetic aperture blurred image generated using an image captured by a multi-lens camera having a pupil 54 of each optical system arranged on a square line with a square of 9 mm and a pitch of 12 mm (p). The right image shown in the figure is the same optical system, and as shown in FIG. 46 (3), a circular blur algorithm image in which the size of the blur kernel 51 is set to a radius (r) of 18 mm on the pupil plane st plane. It is. The rectangular opening serving as the pupil 54 has a side length of 80% with respect to the pitch (p). There is no problem with the image on the right.

(10) As shown in FIG. 46 (2), the left image shown in FIG. 58 is arranged on a square line with a side length (l) of 8.4 mm and a square of 12 mm pitch (p). It is a synthetic aperture blurred image produced | generated using the image imaged with the multiview camera which has the pupil 54 of each optical system. The right image shown in the figure is the same optical system, and as shown in FIG. 46 (3), a circular blur algorithm image in which the size of the blur kernel 51 is set to a radius (r) of 18 mm on the pupil plane st plane. It is. The rectangular opening serving as the pupil 54 has a side length of 70% with respect to the pitch (p). There is a problem with the blur in the image on the right.

  Hereinafter, an example of a biased multi-lens camera in which the pupils of the optical systems are arranged on a rectangular line will be described. It can be seen that no blur generation radius is applied to the rectangular opening, and there is no failure if the rectangular opening is dense. As shown in FIG. 47 (2), it is not rough, and it is sufficient if it is dense as shown in FIG. 47 (1). In the case of the pupil arrangement shown in FIG. 47, the blur generation radius is a radius (r) of 0.5 p or less.

[Example of placing the pupils of each optical system on a rectangular line]
(11) As shown in FIG. 47 (1), the image on the left side shown in FIG. 59 is a square with a side length (l) of 12 mm and is arranged on a vertically long rectangular line at a pitch of 12 mm (p). It is a synthetic aperture blurred image generated using an image captured by a multi-view camera having a pupil 55 of the system. The right image shown in the figure is the same optical system, and as shown in FIG. 47 (3), a circular blur with a blur kernel 56 having a radius (r) of 4.8 mm on the pupil plane st plane. It is an algorithm image.

(12) As shown in FIG. 47 (1), the left image shown in FIG. 60 is a square with a side length (l) of 9.69 mm and is arranged on a vertically long rectangular line at a pitch of 12 mm (p). It is a synthetic aperture blurred image produced | generated using the image imaged with the multi-eye camera which has the pupil 55 of each optical system. The right image shown in the figure is the same optical system, and as shown in FIG. 47 (3), a circular blur algorithm image in which the size of the blur kernel 56 is set to a radius (r) of 6 mm on the pupil plane st plane. It is.

(13) As shown in FIG. 47 (1), the left image shown in FIG. 61 is a square with a side length (l) of 9.69 mm and is arranged on a vertically long rectangular line at a pitch of 12 mm (p). It is a synthetic aperture blurred image produced | generated using the image imaged with the multi-eye camera which has the pupil 55 of each optical system. The right image shown in the figure is the same optical system, and as shown in FIG. 47 (3), a circular blur with the size of the blur kernel 56 set to a radius (r) of 7.2 mm on the pupil plane st plane. It is an algorithm image.

  Hereinafter, in the case of a rectangular opening having a 45 degree arrangement (rhombic arrangement) in which each pupil is arranged on a rhombus line inclined by 45 degrees with respect to the square, it is not rough as shown in FIG. As shown in 4), there is little failure if the area is dense and the blur radius does not cover the rectangular opening, but the sliced pupil width varies greatly with the radius compared to the 0 degree arrangement (square arrangement). , Artifacts are strongly generated in the bokeh.

[Square opening, 45 degree (diamond) arrangement]
(14) As shown in FIG. 47 (4), the left image shown in FIG. 62 is a square with a side length (l) of 12 mm and is arranged on a rhombus line at a pitch of 12 mm (p). It is a synthetic aperture blurred image generated using an image captured by a multi-lens camera having a pupil 58 of the system. The right image shown in the figure is the same optical system, and as shown in FIG. 47 (3), a circular blur having a blur kernel 56 with a radius (r) of 5.4 mm on the pupil plane st plane. It is an algorithm image.

(15) As shown in FIG. 47 (4), the image on the left side shown in FIG. 63 is a square with a side length (l) of 12 mm and is arranged on a rhombus line at a pitch of 12 mm (p). It is a synthetic aperture blurred image generated using an image captured by a multi-view camera having a pupil 59 of the system. The right image shown in the figure is the same optical system, and as shown in FIG. 47 (3), a circular blur algorithm image in which the size of the blur kernel 56 is set to a radius (r) of 6 mm on the pupil plane st plane. It is.

In the case where the circular openings described below are arranged at 45 degrees, the openings are made larger and densely arranged than in the arrangement example shown in FIG. can get.

[Circular opening, 45 degree (diamond) arrangement]
(16) As shown in FIG. 48 (1), the left image shown in FIG. 64 is a circle with a diameter (k) of 12 × (√2) mm and is arranged on a rhombus line at a pitch of 12 mm (P). It is a synthetic aperture blurred image obtained from the image imaged with the multi-lens camera which has the pupil 60 of each optical system. The right image shown in the figure is the same optical system. As shown in FIG. 47 (3), the size of the blur kernel 56 is set to a radius (r) of 6 × (√2) mm on the pupil plane st plane. This is a circular blur algorithm image.

In order to obtain a blur image similar to the optical blur image of the circular aperture from the blur algorithm image of each example described above, the camera pupil shape on the pupil conjugate plane or the camera placement plane (s, t plane) is It has been found that the higher the objectivity, the better, and the slice by the radius around the virtual optical axis may be non-zero at 80% or more of the declination.

  The images shown in FIGS. 65 to 68 show optical blur images when the optical system has a circular aperture. The blur algorithm is not processed. The object is positioned at z = 19 pixel, and the focus is set at z = 9 pixel. The diameters on the blurred pupil plane are 48 mm, 36 mm, 24 mm, and 12 mm, respectively. It can be seen from the comparison with these images that the blur algorithm images of the blur algorithm described so far have no color.

[Reference circular aperture blur] (Example of circular blur using an ideal optical system)
(1) As shown in FIG. 46 (1), the left image shown in FIG. 65 has a side length (l) of 12 m.
It is a synthetic aperture blurred image generated by using an image captured by a multi-lens camera having a pupil 53 of each optical system arranged on a square line with a square of m and a pitch of 12 mm (p). The right image shown in the figure is the same optical system, and as shown in FIG. 46 (3), a circular blur algorithm image in which the size of the blur kernel 51 is a radius (r) of 36 mm on the pupil plane st plane. It is.

(2) The left image shown in FIG. 66 has a side length (l) of 12 m as shown in FIG. 46 (1).
It is a synthetic aperture blurred image generated by using an image captured by a multi-lens camera having pupils 53 of each optical system randomly arranged on a square line at a square of m and at a pitch of 12 mm (p). The right image shown in the figure is the same optical system, and as shown in FIG. 46 (3), a circular blur algorithm image in which the size of the blur kernel 51 is a radius (r) of 36 mm on the pupil plane st plane. It is.

(3) As shown in FIG. 46 (5), the left image shown in FIG. 67 is a pupil of each optical system arranged in a circle having a diameter (k) of 6 mm and a square line at a pitch of 12 mm (p). 5 is a synthetic aperture blurred image generated using an image captured by a multi-lens camera having 52. The right image shown in the figure is the same optical system, and as shown in FIG. 46 (3), a circular blur algorithm image in which the size of the blur kernel 51 is a radius (r) of 36 mm on the pupil plane st plane. It is.

(4) The left image shown in FIG. 68 is a pupil of each optical system arranged on a square line with a diameter (k) of 12 mm and a 12 mm pitch (p) as shown in FIG. 46 (4). 50 is a synthetic aperture blurred image generated using an image captured by a multi-lens camera 50. The right image shown in the figure is the same optical system, and as shown in FIG. 46 (3), a circular blur algorithm image in which the size of the blur kernel 51 is a radius (r) of 36 mm on the pupil plane st plane. It is.

[Entrance pupil]
As shown in FIG. 69, the entrance pupil (Ent_p) is an image of the aperture stop (AS) by the optical system (FE) in front of the lens. The exit pupil (Ext_p) is an image of the aperture stop (AS) by the lens optical system (RE). In the figure, an object OB is formed as an image IM, and a lens front optical system FE and a rear optical system RE are positioned on the optical axis OA. The optical axis can be regarded as a light ray passing through the center of the lens, and is a light ray that goes straight without being refracted because it passes through the center of the lens. When the object point is on the optical axis, the optical axis becomes the central ray. When the object point has a finite height, the ray passing through the center of the entrance pupil becomes the central ray. This is called principal ray PL (principal ray or chief ray). Since the entrance pupil and the exit pupil are conjugate, the principal ray PL reaches the image plane through the center of the exit pupil on the image side. When the pupil is circular, the optical axis passes through the center of the pupil plane. The diameter PD of the entrance pupil is obtained by PD = f / F from the F number and the focal length f. The position of the entrance pupil is usually indicated by a distance LP from the first surface. The entrance pupil diameter PD is about 80 to 90% of the maximum lens diameter OD in the case of a normal camera lens tesser type. The distance LP from the first surface of the entrance pupil is about LP = 10 mm in the case of a Tesser type lens having a focal length of 45 mm. In the following embodiments, a plurality of camera lenses are arranged. In this case, the entrance pupil is projected onto the camera plane (or pupil plane). Since the camera surface and entrance pupil are the same size as seen from the object, it is convenient for design.

[Arrangement of camera lens]
When arranging a plurality of camera lenses, the entrance pupil is projected onto the camera plane (st plane). Since the camera surface and entrance pupil are the same size as seen from the object, it is convenient for design. In the case of the example shown in FIG. 70, the combined optical axis passes through the origin of the st plane. The entrance pupil Ent_p of the cameras 1 and 2 is positioned on the st plane. The optical axis of each camera is located at (s, t) = (0, t0) and (s, t) = (0, −t0). The origin (s, t) = (0, 0) is the composite viewpoint.

[About the arrangement of multi-lens cameras]
When multiple cameras are arranged in multiple eyes, a straight line that passes through the origin (0,0) of the camera plane st plane and is perpendicular to the st plane is regarded as the optical axis of the synthetic aperture method and is called the synthetic optical axis. To. Also origin (0,0)
Is called a composite viewpoint. In the synthetic aperture method, a synthetic image is output according to equations 4, 5, and 6. The amount of blur in the synthetic aperture method is proportional to the camera distance from the origin (0, 0) of the camera plane st plane. Therefore, even if the cameras are arranged in a ring shape, the way of blurring by the synthetic aperture method depends on the origin (0, 0) of the camera plane st plane, that is, the way of selecting the composite viewpoint. In addition, the shape of the blur kernel that depends on the camera arrangement also changes. FIG. 71 shows an example in which the blur kernel changes due to the bias of the composite viewpoint. As shown in FIG. 71, when arranging multi-lens cameras in an annular shape, it is possible to select an arbitrary composite viewpoint, but in the example described below, as a basic arrangement, The case where the composite viewpoint is the center of the annular zone is shown. If the synthetic viewpoint is the center of the annular zone, there is an advantage that the size of the generated blur can be increased. This is because the size of the generated blur is limited to the same level as the diameter of the annular zone of the optical system, as is noticeable in the example of the camera arrangement of (1) and (2) in FIG. The viewpoint position and the amount of generated blur may be adjusted so that the generated blur falls within the annular pupil of the optical system, thereby generating a blur algorithm image viewed from an arbitrary viewpoint within the annular pupil. May be.
In the example of FIG. 10 described above, cam3 is located at the origin (0, 0) of the st plane and is a composite viewpoint.

[Multi-camera annular arrangement]
When the annular pupil shape is configured by a multi-view camera, as shown in FIG. 72, the camera is arranged on the circumference of the radius r. The surface on which the photographing lens of the camera is arranged or the surface on which the entrance pupil is arranged is defined as s and t planes. In the figure, the circular shape of the photographing lenses C1 to C12 represents the size of the entrance pupil. A two-dimensional image sensor is attached to each objective lens (photographing lens) to constitute a camera unit, and a plurality of these are arranged to constitute a multi-view camera. (Focus alignment (focusing) by a synthetic aperture method using a multi-lens camera is as described with reference to FIG. 10. The synthetic aperture output is calculated by using [Equation 6] with the weight of each camera as 1.)
As shown in FIG. 74, in the camera arrangement, when the planes where the entrance pupils are arranged are s and t planes, as shown in FIG. It is desirable that there is no cut in the direction of the arc having a radius r (a cut having an angle of 20% or less). For this reason, it is desirable that the information collected by the lens of the camera can be taken all the way inside the lens C. As shown in FIG. 73, the entrance pupils of the camera lens may be arranged in a concentric direction. Lenses are arranged in a staggered pattern inside and outside. There is no break in the arc direction and the higher the uniformity, the better (the details will be described later with reference to FIG. 75). The size of the blur that is actually reproduced by the blur algorithm is arbitrary, but the limit is that the generated blur radius of about r (+ 50% of the aperture width) is large on the pupil.

FIG. 75 is an example of an annular pupil. Assume that the composite viewpoint is at the origin (0,0) of the st coordinate. If the inner radius is “r2” and the outer radius is “r1”, the pupil region where the light beam enters is “f (s, t)”. Consider a slice by a straight line s ′ (θ) that originates in one direction from the origin (0,0) and forms θ on the s-axis (this is called a slice by the radius s ′ (θ)). Slicing is to make zero except on a straight line. The sliced result is a line segment s ″ (θ). The length of the line segment s ″ is | s ″ (θ) | (there is a plurality of light receiving regions, and the sliced result is a plurality of line segments. In this case, a total of a plurality of line segments is considered.) When the weights relating to the intensity of each point in the pupil region are uniform, | s ”(θ) |
It is desirable to be constant (50 to 100%) between ˜360 degrees. Conversely, if the weight related to the intensity of each point in the pupil region can be set with an intensity filter, the length of the line segment (| s ”(θ) |)
* The line length, | s ”(θ) |, may be set by making the intensity weight constant (50 to 100%). According to the experiment, the intermittent region of the pupil, that is, | s” (θ) | = 0 is 20% in the angle range of θ
It was found that the following (7.2 degrees or less) is sufficient.

  In this embodiment, in order to obtain a blur without a failure by the blur algorithm, when obtaining a circular blur and when the weights regarding the intensity of each point in the pupil region are uniform, the pupil conjugate plane or the camera placement plane It is clear from the images described with reference to FIGS. 39 to 45 and FIGS. 49 to 68 that the slice of the shape of the upper pupil region by the straight line in the radial direction originating from the synthesis viewpoint should be dense and as uniform as possible. . The pupil intermittent region may be less than 20% in angle. In the multi-lens camera arrangement for the annular zone and the rotation target, there is no major failure even with the blur amount corresponding to the pupil diameter about the outside of the camera arrangement. However, in the case of a biased camera arrangement, such as a rectangular arrangement, it is necessary to set the amount of blurring enough to be inscribed in the camera group with the combined viewpoint of the cameras as the center. Considering this point, it is clear that the circular arrangement is most efficient. However, in order to obtain the effect of the present invention, any camera arrangement may be used in principle as long as the camera is arranged so as to surround the composite viewpoint. That is, as hardware, when a plurality of pupil regions arranged so as to surround the viewpoint on the pupil plane or its conjugate plane are sliced by a radius emitted from the viewpoint on the same plane, the deviation angle of the radius is 80. It is only necessary to have a multi-lens imaging unit that forms an image of a light flux that exists in% or more and passes through a plurality of pupil regions by an optical system and individually captures a subject image for each pupil region. The pupil conjugate plane may be a plurality of pupil conjugate planes formed by a microlens array positioned on the imaging plane, or may be combined with a so-called light field camera. Such an embodiment will be described later with reference to FIGS. When a microlens array is used, it is easy to increase the number of pupil divisions, and the pinhole diameter of each pinhole camera is reduced, so that the geometrical depth of focus of the pinhole camera can be increased.

  In order to efficiently place the lens on the pupil plane or the camera placement plane, a classic symmetrical lens design such as triplet type, tesser type, gauss type, etc. is desirable, and the optical system for recent mobile phones has a rear group diameter. Difficult to place large. This seems to be due to the design with short back focus. However, if a lens design is made in which the entrance pupil is positioned immediately after the first surface, the pupil aperture ratio (= incidence pupil diameter / lens outer shape) can be set to 100%. When the rear group has a large diameter, it is possible to design the rear group to be folded back by a mirror.

When a thin camera is configured, it can be considered that the camera is appropriately bent with a mirror or the like. In addition, it is also possible to increase the pupil aperture ratio (= incidence pupil diameter / lens outer shape) using a rod lens, an image fiber, a Selfoc lens (registered trademark), or the like. EDOF (Extended Depth of Field) in which the diameter of each lens of a multi-lens camera becomes large and the focal depth of a single lens is insufficient, giving a cubic wave pupil aberration and increasing the depth of focus.
) It may be used as a lens. However, it is convenient to use a microlens optical system described later. Since refocusing (focusing on an arbitrary object surface after photographing) by the synthetic aperture method is an important function, it is desirable that each camera lens has a deep focal depth and pan focus. (Or a single light field camera.) Therefore, it is desirable to reduce the entrance pupil of each lens or to reduce the photographing magnification. In the case of f = about 4 mm, F2.8, angle of view of about 50 degrees, and 1/4 inch sensor lens, the MTF change at a Nyquist frequency equivalent to 5 to 6 M pixels from infinity to 1 m when focused on 2 m is 20% Degree. In this case, the diameter of the entrance pupil is 1.4 m. A plurality of such lenses are arranged. If anything, it seems to be good to construct a Tesser type with a wafer level lens. If it is difficult to make the pupil width as a result of slicing with a straight line centered on the composite viewpoint as shown in FIG. 75, a density filter is provided at the pupil conjugate position to adjust the light intensity transmittance of the pupil. Such filtering may be performed.
When the change due to the focus position of the MTF becomes a problem, it can be dealt with by appropriately changing the image sharpening process according to the focus position. As another means, each of the multi-lens cameras that divide the pupil may be initially adjusted so as to gradually change the focus position from infinity to a short distance. With this setting, the characteristics of the synthetic PSF of the image after synthetic aperture processing do not change much depending on the focus position.

  As shown in [Table 5], if the pupil size is 50 mm, 112 objective lenses (cameras) are required. The number of pixels can be increased if the resolution of the lens has a margin up to the number of pixels corresponding to a value obtained by multiplying the number of cameras by about 0.5 times by super-resolution processing. In addition, the focus position of the objective lens is gradually changed from a short distance to a long distance for each lens, and the focus image after focusing by the synthetic aperture method is set to be the same PSF in any case. You can keep it.

[Rectangular arrangement of rectangular openings]
Of course, as shown in FIG. In this case, the photographic lens approaches a fly-eye lens. C1 to C24 indicate the pupil shape of the lens. As shown in FIG. 77, when the Tesser lens is cut into a square, an entrance pupil 71 that is about 80 to 90% effective area with respect to the lens outer shape 70 can be obtained, and a multi-lens camera is configured with a small number of lenses. be able to.

Second Embodiment
The second embodiment is an invention for simultaneously obtaining a plurality of focus images as multiple annular pupils. The optical system may be a refractive system, a catadioptric system, or a reflective system. An imaging sensor is prepared for each focus. In order to acquire images with multiple focus completely simultaneously, we propose a multi-eye arrangement as well as a multi-zone pupil optical system. In addition, it goes without saying that images of a plurality of focus can be acquired at the same time even if the image sensors are arranged at a plurality of focus positions by dividing the amplitude. An example of this is shown in FIG. The light emitted from the object O and collected by the photographing lens Ob is
The amplitude is divided by the half mirrors p1 to p5, and images are formed on the light receiving surfaces of the imaging sensors s1 to s5 including the imaging elements by the relay lenses Or1 to Or5. Each of the image sensors s1 to s5 is given a desired focus offset. For this reason, images at a plurality of focus positions can be acquired simultaneously. In order to obtain almost the same time even if they are not simultaneous, the focus sensor and a part of the lens are driven in the focus direction in a vibrating manner, and the focus positions of the imaging sensors s1 to s5 are continuously changed, with a time difference. A plurality of focus images may be acquired almost simultaneously.

  FIG. 79 shows another embodiment in which the pupil is multiplexed and divided by the ring-to-pupil optical system. In this case, the pupil plane is divided into three zones with different diameters by the mirrors p1 to p3 in the vicinity of the pupil conjugate plane, and images are taken by the imaging sensors s1 to s3, respectively. At this time, each of the imaging centers s1 to s3 is given a predetermined focus offset, and a plurality of focus images can be obtained simultaneously. In this embodiment, the upper limit at which blur can be formed by the blur algorithm is up to the inner diameter AP of the multiple ring zone of the optical system. Even in the case where the image side NA changes for each focus as in the present embodiment, there is no problem as long as the integral IPSF in the focus direction of the PSF is within the range of the depth of the target object and can be regarded as virtually unchanged.

  When the pupil division of the half mirrors P1 to P5 is not the amplitude division but the wavefront division, it is effective to divide in a ring shape, but as another example, it is conceivable to divide in a random pattern. At that time as well, as described above, the radial segment centered on the synthetic viewpoint occupies about 80% of 360 degrees, and if possible, the segment length after slicing is uniform (50 to 100%) ) To determine a random pattern. A random pattern is generated by a genetic algorithm or the like under the above constraint conditions.

  As another example of wavefront division, as shown in FIG. 80, a single imaging sensor s1 is used, and a mirror is used as a spatial light modulation element p1 such as a so-called DMD (deformable mirror array). The selection area of the annular zone may be switched and changed, and an image may be acquired in synchronization with the switching of the annular zone. In such a case, a rotationally symmetric aberration such as spherical aberration is given to the photographing lens Ob, and the focus offset can be changed simultaneously by selecting the diameter of the annular zone. The lens Or1 is a relay lens. The symbol “p2” is a mirror. If the aberration is changed by changing the lens interval, the aberration will change depending on whether the spherical aberration is positive or negative, such as increasing the front blur or increasing the rear blur. It is done. The amount of aberration may be changed by a spatial light modulation element such as a liquid crystal. Further, the spherical aberration amount may be changed in accordance with the depth of the subject.

[Use of micro lens array]
Next, an embodiment in which a microlens array is used will be described. FIG. 81 shows an imaging unit having a generally known light ray sensor (LLS). In the LLS, the microlens array M is arranged in front of the imaging sensor sa. The microlens array M is obtained by arranging a plurality of microlenses (positive lenses) M1 to Mn in a two-dimensional array. The imaging sensor sa has a plurality of light receiving elements on the imaging surface. A predetermined number of light receiving elements are arranged behind the microlenses M1 to Mn. In such a configuration, the microlenses M1 to Mn correspond to one pixel of a normal image sensor.

Note that four pixels of RGBG arranged in a Bayer arrangement form one pixel. The microlens array M forms an image of the pupil plane of the photographing lens (imaging optical system) Ob on the light receiving surface of the imaging sensor sa. That is, the imaging sensor sa receives the light intensity at each pixel position corresponding to the microlenses M1 to Mn by dividing the pupil. Since the light is divided and received on the pupil plane, this can be focused by the synthetic aperture method as in the multi-lens camera, and it is needless to say that the above-described blur algorithm by the multi-lens camera can be used. If the light receiving element group seg2 corresponding to the pixel of the microlens M2 in the drawing has the s ″ and T ″ coordinates at the center of the segment on the s ′ and t ′ planes,
As shown in 1). A pupil conjugate plane is projected onto the back focal plane of the microlens, and two-dimensional pixels are arranged according to the number of pupil divisions. That is, a plurality of pupil division elements are arranged for one pixel. Normally, the array of image sensors (= pupil splitting elements) and the array of microlenses are arranged in a square grid, and the array of image sensors and microlenses are arranged so that the s, t coordinates and x, y coordinates are parallel. Parallel. A camera system arranged in this way is generally called a light field camera. The present invention can also be applied. For example, when focusing is performed by the synthetic aperture method using only the sensor in the vicinity of the outer periphery of the pupil division sensor, the multi-camera camera arranged in a ring shape around the optical axis is focused by the synthetic aperture method. An image equivalent to the case can be formed. When the division number of the light field camera is, for example, 8 × 8, a pupil division sensor as shown in FIG. 82 (1) is formed. If the 28 pupil division sensors on the outer periphery are used, pupil division of a rectangular area is achieved. Utilizing such a hollow annular zone or rectangular pupil division sensor is particularly compatible with a catadioptric optical system having an annular pupil.
When applying a focus offset using a spherical aberration or a step on the image plane, the portion corresponding to the annular optical system of the pupil division element (selected to be in the annular shape of the light receiving element group seg2 arranged in a matrix shape) Since the integration of the regions divided by the concentric circles is sufficient, the light receiving element group may be arranged concentrically from the beginning as shown in FIG. In this case, the synthetic aperture method cannot be performed in order to obtain a plurality of focused images simultaneously. A step is provided in the focusing direction with respect to the light receiving surface of the light receiving elements arranged concentrically. Alternatively, rotational symmetric aberration such as spherical aberration may be given to the photographing lens Ob, and a focus offset may be given by this aberration. If you change the lens distance to change the amount of aberration, the aberration will change depending on whether the spherical aberration is positive or negative, such as increasing the front blur or increasing the rear blur. It is done. As described above, when the imaging sensor in which the light receiving elements are arranged in a ring shape for each segment corresponding to the image plane formed by the micro lenses M1 to Mn, the number of the light receiving elements can be reduced.

  Also, when blur processing is performed, the blur kernel is made asymmetric before and after the focus position (see (4) in the same figure), and convolution is performed using the asymmetric blur kernel so that the size of the front blur and the rear blur is reduced. You may adjust (control). This effect and the effect of the optical system (the effect of changing the front blur or back blur by changing the spherical aberration to plus or minus) may be complementarily optimized. .

  In the example shown in FIG. 82 (3), the number of concentric segments is increased, and the number of segments to be used is increased or decreased according to the size of blur added by the blur algorithm. In this case, when increasing the blur, an outer segment group such as “as1” is used, and when the blur is small, the segments from “as1 to as3” including the inner “as3” are used. By doing so, the sensitivity can be improved. This is effective only when the blur may be small. Also, the segment to be used can be changed according to the distance to the subject. In such a case, it is desirable to use a segment such as “as3” at the time of short-distance shooting, in which the magnification of the image plane increases and blurring tends to increase. A catadioptric optical system is suitable as the optical system for the annular pupil, particularly in the case of a telephoto lens having a long focal length.

<Third Embodiment>
In the third embodiment, as shown in FIG. 83 (2 ′), a plurality of cameras with a synthetic lens capable of synthetic aperture are arranged. FIG. 1 (1 ′) is a diagram in which the entrance pupil is projected onto the camera plane st plane. As shown in FIG. 1 (1 ′), a camera Cn capable of focusing by a synthetic aperture having a sensor lens array H at a pupil conjugate position is disposed.

Thus, by arranging a plurality of cameras with sensor lens arrays, it is possible to increase the number of pupil divisions in the camera surfaces s and t. In this embodiment, the number of pupil divisions H for each camera Cn is 24, and the number of cameras Cn is 11, so that there are convenient 264 pupil division cameras. All of these can be used for focusing by the synthetic aperture method, and a blur algorithm image with arbitrary blur can be generated using the blur algorithm. The size of the blur is about a radius r2 in the present embodiment. The radius r2 is about the radius where 11 cameras capable of synthetic aperture are arranged. In this embodiment as well, it is desirable to select the pupil division camera so that the center (0, 0) is the synthesis viewpoint, and the slice based on the radial radius is uniform without being interrupted by the declination. It is desirable to select a uniform width for the camera along r2, for example, a width corresponding to two columns. There is room for further optimization regarding the pupil division method. For example, a triangular pupil dividing element, a hexagonal pupil dividing element, a staggered pupil dividing element, or the like.
For example, a triangular pupil dividing element, a hexagonal pupil dividing element, a staggered pupil dividing element, or the like.
In FIGS. 1A and 1B, the way of pupil division is changed according to the (s, t) coordinates. In this example, the image sensor of each camera is tilted according to the (s, t) coordinates. On the other hand, the microlens remains parallel to the s and t coordinates. In this case, it is necessary to perform additional image processing. The additional image processing is processing for generating a pupil dividing element output of a sub-pixel by interpolation processing from pixels imaged with the imaging element inclined.
In the case of using a normal grid-arranged sensor and a microlens, a combination of (1 ′) and (2 ′) in FIG. C1-C11 may use a commercially available light field camera.

In this way, the light field camera can also be applied. In this case, the shape and arrangement of the pupil division element are designed and optimized by projecting onto the camera arrangement plane st plane. Focusing is performed by the synthetic aperture method using s, t coordinates when the sensor array at the pupil conjugate position is projected onto the entrance pupil. The composite viewpoint and the composite optical axis are the center C of the annular arrangement. With such a configuration, the pupil division size by the pupil division camera can be reduced, and the depth of focus of each pupil division element can be increased.

[Use of spherical aberration]
In FIG. 84, U and V coordinates are camera coordinates. The example shown in the figure is a multi-lens camera in which single-eye imaging units are arranged in an annular shape with diameters s1 to s5. These are previously focused on distances z1, z2, z3, z4 and z5. For example, Z1 = s1 * f / z1, Z2 = 2 * s1 * f / z1, Z3 = 3 * s1 * f / z1, Z4 so that the parallax Z = s1 * f / z is equal to Z1. = 4 * s1 * f / z1, and Z5 = 5 * s1 * f / z1 are focused. That is, z2 = z1 / 2, z3 = z1 / 3, z4 = z1 / 4, z5 = z1 / 5.

  The above is an example of a multi-lens camera, but the cameras of the annular zones s1 to s5 may be anything as long as they are optical systems of annular zones with different focus positions. For example, as shown in FIG. 85, a lens having a symmetric aberration such as a spherical aberration, or a Fresnel lens divided into an annular shape may be used. Thus, when dividing | segmenting a pupil surface into concentric form, it is so preferable that the width | variety of the divided | segmented ring zone is small. This is because the amount of blur varies depending on the subject distance as compared to the type in which the annular pupil is not divided.

  As another example, the effect of the blur kernel can be changed between the front blur and the rear blur, and as shown in FIG. 86, the front blur can be increased or the rear blur can be decreased. Moreover, the blur of an existing lens can be emulated by creating a blur kernel based on the aberration amount of the existing lens. In other words, existing lenses include a lens having a large front blur and a lens having a large back blur depending on the tendency of spherical aberration. This is to imitate this. This embodiment can be used in all cases where a blur kernel is used, such as when an arbitrary blur kernel is used in combination with a multi-lens camera.

[Camera division method]
FIG. 87 shows a multi-lens camera in which, for example, 22 cameras cam1 to cam22 are arranged on the circumference. FIG. 88 shows a main part of each of the cameras cam1 to cam4. Each camera cam1-ca
m22 is the width of the light-receiving area sliced by the straight line (the length of the line segment) (the length of the line segment), considering the straight line originating from the center of the s and t coordinates by processing the outer shape of the objective lens (photographing lens) into a quadrangle. In order to suppress fluctuations due to the angle of deviation of the straight line (when there are multiple line segments in the same direction, the total value) (see FIG. 75), the sides where the lenses contact each other are divided so as not to be parallel to the moving radius It is in form. In this case, the side where the lenses are in contact with each other and the radial direction are always at an angle of about 45 degrees. The entrance pupil has an effective area of 80 to 90% with respect to the lens outer shape. The pupil shape is close to the optical system of the annular pupil with the inner diameter r1 and the outer diameter r2. In FIG. 89, the outer shape of the lens is processed into a crescent shape based on the same idea, and fluctuations due to the deviation of the straight line width of the sliced light receiving region due to the straight line originating from the center of the s and t coordinates are suppressed. Thus, by making the lens into a shape other than a polygon or a circle, it is possible to suppress fluctuations due to the deflection angle of the width of the light receiving region. It is also possible to process the objective lens into a single optical element. For example, as shown in FIG. 90, optical elements gm1 and gm2 may be bonded to a glass substrate gk. Also, gm1 and gm2 may be molded with resin to form a hybrid lens. Moreover, it is good also as the glass which carried out integral injection molding of these, or a resin lens.

  As described above, in this embodiment, an image having the same blur as that of the large-diameter lens can be taken with a downsized camera without using the large-diameter lens. In addition, the amount of blur can be set after shooting. Furthermore, when an annular pupil, a microlens, and an annular imaging segment are combined, it is possible to reduce the number of image sensors compared to this type of conventional camera. In addition, when the focus offset is applied with the spherical aberration amount, it is possible to adjust the size of the front blur and the rear blur. Furthermore, the size of the front blur and the rear blur can be adjusted by making the blur kernel asymmetric before and after the focus position.

  In the above embodiment, a multifocal image necessary for obtaining a photograph with one blur is simultaneously obtained using the multi-lens camera 10 described in FIG. 1, but a general single-lens camera is used. Thus, a plurality of multifocal images may be acquired by performing shooting a plurality of times in time series while shifting the focus.

  In the above embodiments, the electronic cameras 9 and 10 are described. However, the present invention is not limited to this, and the photographing lens 23, the image sensor 20, the AFE 21, the frame memory 22 and the like described with reference to FIG. 6 are omitted. However, an image processing apparatus or method having only an image processing unit may be used. In this case, a storage unit for capturing a plurality of multifocal image data that has been captured in advance or a plurality of multiview camera images may be provided.

  The program to be executed by the image processing unit constitutes a program according to the present invention. As a recording medium for recording the program, a semiconductor storage unit, an optical or magnetic storage unit, or the like can be used. By using such a program and a recording medium in a system having a configuration different from that of each of the above-described embodiments and causing the CPU to execute the program, substantially the same effect as the present invention can be obtained. Therefore, the present invention may be a program for causing a computer to function as the above-described image processing apparatus. Moreover, it is good also as a recording medium which recorded the program.

Next, an embodiment in which an image obtained by tilt shooting is virtually generated using the synthetic aperture method will be described below.
(1) [When virtually creating an image obtained when shooting with the image surface tilted]
In FIG. 91, the object space is represented by (X, Y, Z) on the assumption that the origin of the optical axis Z of the lens L is in the lens. The image plane Imp and the distance a0 of the lens L are arbitrarily set. The image plane Imp and the object plane Obp are in a conjugate relationship. This figure shows a tilting optical system, and since the image plane Imp is tilted, the object plane Obp is also tilted. Consider image points I1, I2, and I0 at distances a0, a1, and a2 from the lens L when the image plane Imp is tilted. The distances b0, b1, b2 to the object points O0, O1, O2 conjugate to the image points I0, I1, I2 are determined for each image point position by the lens imaging formula (1 / a + 1 / b = 1 / f). The maximum and minimum values of the distances b0, b1, b2 to the object points O0, O1, O2 are calculated. In the example shown in the figure, the distance b1 corresponds to the maximum value, and the distance b2 corresponds to the minimum value. Since the range between the minimum value b2 and the maximum value b1 of the distance to the object points O0, O1, and O2 is in focus, the distance between them is divided at an appropriate interval, and the divided positions are determined as a plurality of focus distances. . As the focus distance, for example, it is preferable to set the focus range at a position divided at equal intervals. Next, a plurality of images in focus at a plurality of focus distances are generated by performing a blur algorithm process with a desired blur amount. In these images, if there is an object in focus, no blur occurs. If there is an object in focus, the image is blurred. Data for each image point on the image plane Imp when the above-mentioned image plane is tilted from each image focused on multiple focus distances (RGB intensity values. Actually, the same processing for R, G, and B Of course, the data corresponding to the distances b0, b1, b2 for the conjugate object points O0, O1, O2 on the object plane Obp are extracted. In this way, it is possible to obtain the data of a single tilt photograph (in the case of tilt photography, the object of interest is usually photographed in focus (in focus)).

(2) [In the case of virtually generating an image obtained when shooting with an object tilted]
In this case, the surface to be tilted is set to the object plane Obp, but the object plane Obp and the image plane Imp have a conjugate relationship. By using the same processing as in the case of virtually generating an image obtained when shooting at an angle, it is possible to obtain data for a single tilt photograph.

  As described above, the image generation apparatus, the image processing apparatus, the image generation method, and the image processing program according to the embodiment of the present invention have been described in detail. However, the above-described configuration can be appropriately changed without departing from the gist of the present invention. is there.

[Appendix]
As will be understood from the description of the embodiments of the invention described in detail above, the present specification includes disclosure of various technical ideas including at least the invention described below.

(Invention 1): A plurality of in-focus images acquired from a multi-view imaging unit that individually captures subject images with a plurality of photographing lenses are blurred using a two-dimensional filter set in advance at an arbitrary focus position. Image generation that performs processing, calculates a focus stack image that is an intensity sum image of a plurality of images subjected to the blur processing, and generates an image with arbitrary blurring by sharpening the focus stack image In the method, the plurality of photographic lenses are arranged so that a slice by a radius of movement about the virtual optical axis of the pupil of the photographic lens is non-zero at about 80% or more of the deviation angle. An image generation method.

(Invention 2): In the image generation method according to Invention 1, the blur process includes a process of performing a convolution using an arbitrary blur kernel,
The image sharpening process includes a deconvolution process using a predetermined kernel for the focus stack image.

(Invention 3): In the image generation method according to Invention 2, the arbitrary blur kernel is a circular aperture having a blur radius proportional to a desired focus position or a function representing Gaussian. Generation method.

(Invention 4): In the image generation method according to Invention 2 or 3, the deconvolution process using the kernel corresponds to a focus stack of the arbitrary blur kernel used when performing the convolution process in the blur process. An image generation method comprising a deconvolution process using a focus stack blur kernel.

(Invention 5): A plurality of in-focus images acquired from a multi-view imaging unit that individually captures subject images with a plurality of photographing lenses are blurred using a two-dimensional filter set in advance at an arbitrary focus position. Image generation that performs processing, calculates a focus stack image that is an intensity sum image of a plurality of images subjected to the blur processing, and generates an image with arbitrary blurring by sharpening the focus stack image In the method, the plurality of photographic lenses may be configured such that a slice of a pupil region by a radius that rotates in an arbitrary direction around the combined optical axes of the plurality of photographic lenses is non-zero at about 80% or more of the declination. An image generation method characterized by being arranged.

(Invention 6): [Invention to divide the pupil by the annular optical system described in FIG. 79]
Blur processing using a two-dimensional filter set in advance at an arbitrary focus position for a plurality of in-focus images acquired from a multi-view imaging unit that individually captures subject images formed by a plurality of photographing lenses In an image generation method for performing each of these, calculating a focus stack image that is an intensity sum image of a plurality of images subjected to the blur processing, and generating an image with arbitrary blurring by image sharpening the focus stack image The multi-view imaging unit divides the pupil into a plurality of regions in which a radial slice centered on the virtual optical axis of the pupil of the photographing lens is non-zero at approximately 80% or more of the declination angle. An image generation method characterized by the above.

(Invention 7): [Invention using the microlens described in FIG. 81]
A photographic lens that forms a subject image on a predetermined imaging plane, a lens array that is two-dimensionally arranged in the vicinity of the imaging plane, and a rear side of the lens array corresponding to each of the lens arrays A plurality of light receiving elements arranged in a predetermined number, and each region from a plurality of concentric or rectangular light receiving elements in a light receiving element group included in a region corresponding to an image plane formed by each positive lens. For each of a plurality of in-focus images acquired in advance, a blur process set in advance at an arbitrary focus position is performed, and a focus stack image that is an intensity sum image of the plurality of images subjected to the blur process is calculated. An image generation method comprising: generating an image with arbitrary blurring by performing an image sharpening process on the focus stack image.

(Invention 8): The image generation method according to Invention 7, wherein the photographing lens has rotationally symmetric aberration.

(Invention 9): [Invention for controlling the size of front blur and rear blur explained in FIG. 82 (4)]
Using a two-dimensional filter that is set in advance at an arbitrary focus position for a plurality of in-focus images acquired from a multi-lens imaging unit that individually captures subject images that are focused with a plurality of photographing lenses. Each of the blur processing is performed, a focus stack image that is an intensity sum image of the plurality of images subjected to the blur processing is calculated, and the focus stack image is sharpened to generate an image with an arbitrary blur. In the image generation method, the blur processing includes convolution using a blur kernel that is asymmetric before and after the focus position.

9, 10 Multi-lens camera 11 Imaging aperture 20 Imaging element 23 Imaging lens 25 Image processing unit 36 Blur treatment unit 39 Image sharpening processing unit 40 Single eye imaging unit

Claims (13)

A blur processing unit that performs a blur process assuming a common focus distance on a plurality of focus images having different focus states and a common viewpoint;
A sharpening processing unit that generates a blurred image with a predetermined blur from the plurality of focus images by performing a sharpening process on a focus stack image that is a sum of intensities of the plurality of images subjected to the blurring process. When,
An image generation apparatus comprising:   The image generating apparatus according to claim 1, wherein the plurality of focus images are generated using an image group including a plurality of images having different viewpoints and a common shooting distance.   The image generation apparatus according to claim 2, wherein the plurality of images having different viewpoints and a common shooting distance include a plurality of images whose parallax amounts are equally spaced from each other.   The image generating apparatus according to claim 1, wherein the plurality of focus images are configured using an image group including a plurality of images having different focus states and having a common viewpoint.   The image generation apparatus according to claim 4, wherein the plurality of images having different focus states and having a common viewpoint include a plurality of images in which a difference in blur amount between the images is proportional.   The image generation apparatus according to claim 1, wherein the blur image is the blur image at a viewpoint designated by a user or a predetermined viewpoint.   The image generation apparatus according to any one of claims 1 to 6, wherein the sharpening process generates the blurred image using information related to an imaging optical system used for imaging a plurality of images.   The image generation apparatus according to claim 1, wherein the sharpening process is a deconvolution process. An image acquisition unit for acquiring a plurality of images with different viewpoints and a common shooting distance, or a plurality of images with different focus states and common viewpoints;
A focus image generation unit that generates a plurality of focus images having different viewpoints and a common viewpoint from a plurality of images acquired by the image acquisition unit;
A blur processing unit that performs a blur process assuming a common focus distance for the plurality of focus images generated by the focus image generation unit;
A sharpening processing unit that performs a sharpening process on a focus stack image that is a sum of intensities of a plurality of images subjected to the blurring process;
An image processing apparatus.   The image processing apparatus according to claim 9, wherein the image acquisition unit includes a plurality of imaging units.   The image processing apparatus according to claim 10, wherein the plurality of imaging units include a plurality of imaging units arranged on a circumference having a radius R from a position indicating a center of an array of the plurality of imaging units. A blur processing step for performing a blur process assuming a common focus distance for a plurality of focus images having different focus states and a common viewpoint;
A sharpening process step of generating a blurred image with a predetermined blur from the plurality of focus images by performing a sharpening process on a focus stack image that is a sum of intensities of the plurality of images subjected to the blur process When,
An image generation method comprising: A blur processing procedure for performing a blur process assuming a common focus distance on a plurality of focus images having different focus states and a common viewpoint;
A sharpening procedure for performing a sharpening process on the focus stack image, which is a sum of intensities of a plurality of images subjected to the blurring process, and generating an arbitrary blurred image from the plurality of focus images;
An image processing program for causing a computer to execute.