WO2007100057A1

WO2007100057A1 - Imaging device and integrated circuit

Info

Publication number: WO2007100057A1
Application number: PCT/JP2007/053945
Authority: WO
Inventors: Ichiro Oyama
Original assignee: Matsushita Electric Industrial Co., Ltd.
Priority date: 2006-03-03
Filing date: 2007-03-01
Publication date: 2007-09-07

Abstract

An imaging device is provided with image information outputted from imaging regions (103a, 103b) corresponding to first wavelength selecting regions (102a, 102b); a three dimensional coordinate calculating means (110) for calculating the three dimensional coordinates of an object, based on a positional relationship between optical systems (101a, 101b) which correspond to the first wavelength selecting regions (102a, 102b) and the imaging regions (103a, 103b); and a three dimensional coordinate color information calculating means (110) for calculating color information at each coordinate of the three dimensional coordinates of the object, based on the image information outputted from the sections, which correspond to each coordinate of the three dimensional coordinates of the object in at least three imaging regions (103a, 103c, 103d) receiving light of different wavelength ranges, respectively.

Description

Specification

Imaging device and integrated circuit

Technical field

TECHNICAL FIELD [0001] The present invention relates to a three-dimensional imaging apparatus that captures a three-dimensional shape and color of a subject and creates a color arbitrary viewpoint image or a three-dimensional image based thereon, and an integrated circuit used therefor. Background art

In recent years, there has been an increasing demand for measuring the three-dimensional coordinates of an object and displaying a color image of the object, and there has been an increasing demand for downsizing the imaging device. Such an imaging device can be used, for example, for supporting the visibility of automobiles. By observing the situation around the car from various viewpoints (free viewpoints), it becomes possible to recognize the situation more accurately and to realize safe driving.

[0003] Further, in assisting the visibility of a car, by displaying distance information between a surrounding object and the car, it is possible to avoid collision with the surrounding object and to realize safe driving. Also, in devices that input and output video, such as mobile phones and televisions, 3D video input / output is desired in order to more accurately reproduce the realism of the video. In order to realize these functions, it is necessary to obtain the three-dimensional coordinates and color information of the input object of the imaging device, and to reduce the size of the imaging device.

[0004] As an effective technique for downsizing, in particular, thinning of an imaging apparatus, a compound eye type imaging apparatus using a single lens with a short focal length has been proposed (for example, see Patent Document 1). In general, since the refractive index differs depending on the wavelength of light, the focal length is different, and it is impossible to image a scene color including information on all wavelengths on a photographing surface with a single lens. For this reason, the optical system of a normal image pickup apparatus has a configuration in which a plurality of lenses are overlapped in order to image light of red, green, and blue wavelengths on the same image pickup surface. In other words, the optical length of the imaging apparatus is inevitably increased and thickened.

[0005] On the other hand, in a compound eye type color image capturing apparatus, an imaging optical system includes a lens that receives light of red wavelength, a lens that receives light of green wavelength, and a lens that receives light of blue wavelength arranged in a plane. The imaging surface of the imaging device is provided for each lens. Each lens has a limited wavelength of light, so the focal length of each lens By making the same, it becomes possible to form a subject image on the imaging surface with a single lens, and the thickness of the imaging device can be greatly reduced.

FIG. 20 shows a perspective view of an example of a conventional compound eye type imaging apparatus. 500 is a lens array, and the four lenses 501a, 501b, 501c, and 501d forces are molded together! 501a and 50 lb are lenses that handle light of green wavelength, and convert the formed subject image into image information in the imaging areas 502a and 502b in which a green wavelength separation filter (color filter) is pasted on the light receiving unit. .

[0007] Similarly, 501c is a lens that handles light of red wavelength, and is converted into red image information in the imaging region 502c. 501d is a lens that corresponds to light of blue wavelength, and in the imaging region 502d. Convert to blue image information. A color image can be obtained by superimposing and synthesizing these images. The number of lenses need not be limited to four.

[0008] However, the conventional compound-eye color image capturing apparatus as described above has the following problems. FIG. 21 is a diagram showing a basic configuration of a conventional compound-eye imaging device. This figure shows the subject image formed by two lenses with different optical axes, and the positional relationship between the optical axis and the image sensor. FIG. 21 (a) is a diagram viewed from the cross-sectional side of the lens, and FIG. 21 (b) is a plan view of the imaging region.

[0009] Reference numerals 600a and 600b denote optical axes of the lenses 601a and 601b, and reference numerals 602a and 602b denote positions intersecting with the respective optical axial force imaging areas 603. 605a and 605b are images formed through the subject 604 force S lens 601a and 601b on the optical axis 600a.

[0010] In a compound-eye imaging device, the optical axes of the lenses are different from each other. Therefore, the position of the subject image is connected to the optical axes of the lenses on the imaging device according to the subject distance (direction B in the figure). ) This phenomenon is called parallax. The pixel displacement S of the subject image 605b from the position 602b where the lens optical axis 600b intersects the imaging area is expressed by the following equation, where Zp is the subject distance, t is the distance between the optical axes of the lenses, and f is the imaging distance. It is represented by (1).

[0011] Equation (1) S = f -t / Zp

As can be seen from Equation (1), the positional relationship between the subject image and the pixel in the direction connecting the optical axes of the lenses varies depending on the subject distance Zp. Therefore, area-based match as in Patent Document 1 The image shift amount is adjusted by detecting the shift amount of the subject image in the direction connecting the optical axes that changes depending on the subject distance. In the example of FIG. 21, 601a in FIG. 21 (a) corresponds to the lens 501a designed for green in FIG. 20, and 601b corresponds to 501b in FIG. The red and blue image shift amounts can be calculated based on the known inter-lens distance and focal length from the shift amount of the green imaging region obtained by the equation (1).

In the configuration of FIG. 20, the parallax amount deriving unit 503 calculates red, green, and blue parallax amounts as described above, and the image synthesizing unit 504 has a green imaging region, a red imaging region, and a blue imaging unit. Creates a high-definition color image by synthesizing the subject image of the area.

[0013] Since this configuration uses a compound eye system, the thickness can be reduced. Also, the pixel shift amount S force and the subject distance Zp can be obtained. However, since it is premised on the creation of a two-dimensional image, there is a problem that the three-dimensional shape and color of the color of the subject cannot be estimated.

Patent Document 1: Japanese Patent Laid-Open No. 2002-204462

Disclosure of the invention

The present invention solves the above-described conventional problems, and an object thereof is to provide an imaging device that is thin and capable of obtaining the three-dimensional shape and color of a subject and an integrated circuit used therefor. And

In order to achieve the above object, an imaging apparatus of the present invention inputs a plurality of optical systems and a plurality of wavelength selection regions that selectively transmit light in a specific wavelength region among light from a subject. A plurality of imaging regions that output image information according to the light, and the wavelength selection region and the imaging region are arranged in a one-to-one correspondence on each optical axis of the plurality of optical systems. In the apparatus, the plurality of wavelength selection regions selectively select a first wavelength selection region that selectively transmits light in the same wavelength region, and light in a wavelength region different from the first wavelength selection region. A second wavelength selection region to be transmitted, wherein the first wavelength selection region includes two or more wavelength selection regions, and the second wavelength selection region selectively transmits light in different wavelength regions. The wavelength selection region is included and corresponds to the first wavelength selection region. And image information at least two of the respective imaging regions is outputted, based on the positional relationship between the each optical system and the respective imaging areas corresponding to the first wavelength selection 択領 region said that Image information output by the three-dimensional coordinate calculating means for calculating the three-dimensional coordinates of the subject and the portions corresponding to the coordinates of the three-dimensional coordinates of the subject in at least three of the imaging regions that receive light in different wavelength regions, respectively. 3D coordinate color information calculating means for calculating color information at each coordinate of the three-dimensional coordinate of the subject.

The integrated circuit of the present invention is an integrated circuit that performs an operation based on image information output from an imaging region that receives light from a subject, and the imaging region receives light of a first wavelength. Including a first imaging region and a second imaging region that receives light in a wavelength region different from the light of the first wavelength, wherein the first imaging region includes two or more imaging regions, The second imaging region includes two or more imaging regions that selectively receive light in different wavelength regions, and the integrated circuit includes at least two of the first imaging regions. 3D coordinate calculation means for calculating 3D coordinates of the subject based on image information output from the area, and at least three of the 3D coordinates of the subject in each of the imaging areas that receive light in different wavelength ranges The part corresponding to each coordinate is output. That based on the image information, characterized by comprising a three-dimensional coordinate color information calculating means for calculating color information at each coordinate of the three-dimensional coordinates of the object.

Brief Description of Drawings

FIG. 1 is a configuration diagram of a three-dimensional imaging device and a display according to an embodiment of the present invention.

FIG. 2 is a diagram illustrating the principle of a three-dimensional coordinate calculation method for a three-dimensional imaging device according to an embodiment of the present invention.

FIG. 3 is a diagram showing an imaging region of the three-dimensional imaging device according to the embodiment of the present invention.

[FIG. 4] A view of the subject and the imaging unit 100 in FIG.

FIG. 5 is a view of the subject and the imaging unit 100 viewed from the positive direction side of the X axis of the three-dimensional coordinates in FIG.

FIG. 6 is a view of the subject and the imaging unit 100 in FIG.

[FIG. 7] In FIG. 2, the subject and the imaging unit 100 are viewed from the positive direction side of the X axis of the three-dimensional coordinates. Figure.

FIG. 8 is a view of the subject and the imaging unit 100 in FIG.

[FIG. 9] A view of the subject and the imaging unit 100 in FIG.

FIG. 10 is a diagram for explaining the principle of the viewpoint image calculation method of the three-dimensional imaging apparatus according to the embodiment of the invention.

FIG. 11 is a view of the subject and the virtual lens viewed from the positive direction of the axis of the three-dimensional coordinate in FIG.

FIG. 12 is a view of the subject and the virtual lens as viewed from the positive direction of the X axis of the three-dimensional coordinates in FIG.

FIG. 13 is a diagram showing a display according to an embodiment of the present invention, (a) a perspective view,

(b) The figure which also looked at the upper side force of the display shown to (a) figure.

14 is a diagram for explaining the display principle of the stereoscopic image of the stereoscopic display shown in FIG.

FIG. 15 shows a viewpoint image according to an embodiment of the present invention, (a) the figure shows one viewpoint video, and (b) the figure shows another viewpoint video. Figure.

FIG. 16 is an explanatory diagram of coordinate conversion according to an embodiment of the present invention.

FIG. 17 is an explanatory diagram of shape conversion according to the embodiment of the present invention.

FIG. 18 is a diagram showing an example of a lens arrangement according to an embodiment of the present invention.

FIG. 19 is a diagram showing an example of a lens arrangement according to an embodiment of the present invention.

FIG. 20 is a schematic configuration diagram of an example of a conventional compound eye imaging apparatus.

FIG. 21 is a diagram illustrating the principle of an example of a parallax calculation method for a compound-eye imaging apparatus of a conventional example.

BEST MODE FOR CARRYING OUT THE INVENTION

[0018] According to the present invention, the three-dimensional shape of a subject can be obtained with a thin shape.

In the imaging apparatus of the present invention, a portion corresponding to each coordinate of the three-dimensional coordinates of the subject in each imaging region corresponding to the second wavelength selection region is the three-dimensional calculated subject. It is preferable to calculate based on the coordinates and the positional relationship between each optical system corresponding to the second wavelength selection region and each imaging region. [0020] Further, it is preferable to further include coordinate conversion means for performing coordinate conversion on part or all of the three-dimensional coordinates of the subject. According to this configuration, it is possible to output free viewpoint video or stereoscopic video image data of a subject with higher presence.

[0021] Further, it is preferable to further include a shape conversion means for enlarging a part or the whole of the three-dimensional shape based on the three-dimensional coordinates of the subject. According to this configuration, it is possible to output image data for free viewpoint video and stereoscopic video of a more natural subject.

[0022] Further, it is preferable to further include arbitrary viewpoint image output means for outputting an image of the subject at an arbitrary viewpoint based on the three-dimensional coordinates of the subject and color information at each coordinate. According to this configuration, it is possible to output image data for a free viewpoint video or a stereoscopic video of a subject.

[0023] The image processing apparatus further includes arbitrary viewpoint creation means for creating the subject image having an arbitrary viewpoint power based on the three-dimensional coordinates of the subject by interpolation, and the arbitrary viewpoint image output means is configured to generate the subject image by the interpolation. It is preferable to output an image at an arbitrary viewpoint based on the subject image. According to this configuration, it is possible to output free viewpoint video and stereoscopic image data of a more natural subject.

[0024] Preferably, the arbitrary viewpoint image output means outputs an image at an arbitrary viewpoint for stereoscopic video output.

[0025] Preferably, the arbitrary viewpoint image output means outputs images at a larger number of viewpoints than the number of the optical systems. According to this configuration, it is possible to output image data for stereoscopic video of a more natural subject.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

(Embodiment 1)

First, the outline of the three-dimensional imaging apparatus according to the present embodiment will be described with reference to FIG. FIG. 1 shows a configuration diagram of a three-dimensional imaging apparatus and a display according to the present embodiment. This embodiment includes a plurality of optical systems that form a subject image. Specifically, 101a and 10 lb are single lenses designed primarily for photographing green wavelengths of light. 1 Olc is a single lens designed mainly for photographing red wavelength light. lOld is a single lens designed mainly for photographing blue wavelength light. The lens array 101 forms single lenses 101a, 101b, 101c, and 101d by integral molding. Thereby, a lens array can be produced inexpensively and in a small size.

[0029] A wavelength selection region is provided on the optical axis of each single lens. The wavelength selection region is a region that selectively transmits light in the same wavelength region. In the present embodiment, the wavelength selection region is constituted by a color filter. Specifically, 102a and 102b are color filters that mainly transmit light having a green wavelength. 102c is a color filter that mainly transmits light of red wavelength. Reference numeral 102d denotes a color filter that mainly transmits light having a blue wavelength. Note that the color filter 102b is disposed between the short lens 101b and the image sensor 103 shown by a broken line for convenience of illustration.

[0030] Reference numeral 103 denotes an image sensor realized by a CMOS, a CCD, or the like. Here, the image sensor 103 is composed of a single image sensor, but may be composed of a plurality of image sensors. The image capturing unit 100 outputs image data of a subject image captured by the single lenses 101a, 101b, 101c, 101d, the color filters 102a, 102b, 102c, 102d, and the image sensor 103 to the three-dimensional coordinate calculation unit 110.

[0031] The three-dimensional coordinate calculation unit 110 includes three-dimensional coordinate calculation means and a three-dimensional coordinate color information calculation unit. Although details will be described later, the three-dimensional coordinate calculation means calculates the three-dimensional coordinates of the subject based on the image data from the imaging unit 100. The three-dimensional coordinate color information calculating means calculates color information at each three-dimensional coordinate of the subject obtained by the three-dimensional coordinate calculating means.

A viewpoint image output unit 111 outputs a viewpoint image when the subject is viewed from an arbitrary predetermined viewpoint based on the three-dimensional coordinates of the subject calculated by the three-dimensional coordinate calculation unit 110. The viewpoint image output unit 111 includes arbitrary viewpoint image output means, and can simultaneously output viewpoint images viewed from a plurality of viewpoints, that is, a plurality of viewpoint images having different viewpoints. Reference numeral 112 denotes a display, which also displays a stereoscopic image with a plurality of viewpoint image forces with different viewpoints created by the viewpoint image output unit 111.

Next, the configuration of the imaging unit 100 in FIG. 1 and the setting of the coordinate system will be described with reference to FIGS. 1 to 3. FIG. 2 is a diagram for explaining the principle of calculating the three-dimensional coordinates of one point P of the subject when shooting the subject (star). Each principal point A, B, C of single lens 101a, 101b, 101c, 101d D is configured on the same plane, and the imaging surface of the imaging element 103 is arranged in parallel with the plane.

[0034] On the imaging surface of the image sensor 103, the imaging region 103a with the optical axis of the single lens 101a as a reference, the imaging region 103b with the optical axis of the single lens 101b as a reference, and the optical axis of the single lens 101c as a reference An imaging region 103c that is based on the optical axis of the single lens 101d is configured.

Next, the coordinate system in FIG. 2 will be described. FIG. 3 is a configuration diagram of the image sensor 103 viewed from the single lens side. Each imaging area 103a, 103b, 103c, 103d sets a two-dimensional coordinate system with the origin (0, 0) at the lower left of the imaging area. As shown in FIG. 3, in the imaging region 103a, a horizontal line that is positive in the right direction is set as the xa axis, and a vertical line that is positive in the upward direction is set as the ya axis. In the imaging region 103b, a horizontal line whose right direction is positive is set on the xb axis, and a vertical line whose upper direction is positive is set on the yb axis. In the imaging region 103c, the horizontal line with the right direction being positive is set on the xc axis, and the vertical line with the upward direction being positive is set on the yc axis. In the imaging region 103d, the horizontal line with the right direction being positive is set on the xd axis, and the vertical line with the upward direction being positive is set on the yd axis.

In FIG. 3, 104a on the imaging area 103a is the optical axis of the single lens 101a, 104b on the imaging area 103b is the optical axis of the single lens 101b, 104c on the imaging area 103c is the optical axis of the single lens 101c, and imaging 104d on the region 103d is the optical axis of the single lens 101d. The coordinates of the optical axes 104a, 104b, 104c, and 104d on the imaging regions 103a, 103b, 103c, and 103d are configured to be the same.

[0037] In addition, the three-dimensional coordinate system having the principal point A of the single lens 101a in Fig. 2 as the origin (0, 0, 0) has a positive and negative sign parallel to the xa axis of the imaging region 103a as shown in Fig. 2. The X axis is the opposite axis, the Y axis is parallel to the ya axis of the imaging area 103a, and the Y axis is the same, and the optical axis of the single lens 101a is parallel to the optical axis of the single lens 101a. The positive axis is set to the Z axis. The three-dimensional coordinate calculation unit 110 in FIG. 1 calculates the coordinates of the subject in the three-dimensional coordinates.

Next, a method for calculating the three-dimensional coordinates of the subject by the three-dimensional coordinate calculation means included in the three-dimensional coordinate calculation unit 110 of FIG. 1 will be described in detail with reference to FIGS. As shown in Fig. 3, the image of one point P (Xp, Yp, Ζρ) of the subject is affected by the parallax when Ζρ is finite. As a result, images are formed at different coordinates pa, pb, pc, and pd of each imaging region. The calculation of the three-dimensional coordinates of the subject is performed by comparing the subject images in the imaging region where the light of the same color forms an image, that is, here the imaging regions 103a and 103b of green light.

The positions of the single lenses 101a and 101b differ only in the X coordinate in the three-dimensional coordinates. Therefore, the image coordinates pa (xpa, ypa) corresponding to the point P of the subject in the imaging region 103a,

Comparing the coordinate pb (xpb, ypb) of the image corresponding to the point P of the object at 103b, the values of xpa and xpb have a shift amount of S due to the parallax, and the values of ypa and ypb are equal.

That is, when an image at the coordinate pa (xpa, ypa) of the imaging region 103a is selected, the coordinate pb (xpb, ypb) of the image of the imaging region 103b corresponding to this image is obtained by obtaining xpb, and xpb is shifted. If the quantity S is obtained, it will be obtained. The deviation amount S is the imaging area 103a and the imaging area.

Based on the image information output by 103b, it can be obtained by a conventional technique such as an area-based matching method.

[0041] Further, when the deviation amount S is obtained, Ζρ at one point P (Xp, Yp, Ζρ) of the subject is obtained by the following equation (2) using the relationship between the subject distance and the parallax in equation (1). be able to.

[0042] Equation (2) Zp = f -tab / S

S is parallax, that is, the amount of deviation of the subject image, tab is the distance between the optical axes of the single lenses 101a and 101b

, F is the imaging distance, that is, the principal point A of the single lens 101a in FIG.

The distance between 04a.

FIG. 4 is a view of the subject and the imaging area 103a of the imaging unit 100 from the positive direction side of the Y axis of the three-dimensional coordinates in FIG. In the illustration of Fig. 4, the relationship of the following formula (3) can be derived from the geometric calculation.

[0044] Equation (3) Xp = (xpa-xOa)-| Zp | / f

Therefore, by substituting the relational expression of Zp in equation (2) into equation (3), Xp can be expressed as

It is expressed by 4).

[0045] Equation (4) Xp = (xpa-xOa) -tab / S

The distance between the optical axes tab and the deviation S are known. xOa is known and xpa is the X coordinate of the selected position. For this reason, xpa-xOa can also be calculated. Therefore, the value of Xp can be obtained. [0046] FIG. 5 is a view of the subject and the imaging unit 100 imaging region 103a in the positive direction side force of the X axis of the three-dimensional coordinate in FIG. In the illustration of Fig. 5, the relationship of the following formula (5) can be derived from the geometric calculation.

[0047] Formula (5) Yp =-(ypa-yOa)-| Zp | / f

Therefore, by substituting the relational expression of Zp in equation (2) into equation (5), Yp becomes the following equation (6

).

[0048] Formula (6) Yp =-(ypa-yOa) -tab / S

As described above, the distance tab between the optical axes and the shift amount S are known. yOa is known and ypa is the y coordinate of the selected position. For this reason, ypa-yOa can also be calculated. Therefore, the value of Y p can be obtained.

[0049] In this way, the three-dimensional coordinates of the point P (Xp, Yp, Zp) of the subject can be calculated.

. Similar calculations can be used to determine the three-dimensional coordinates of the entire subject. As a result, the three-dimensional shape of the subject can be obtained.

[0050] Next, the three-dimensional coordinate color information calculation means will be described. As described above, the three-dimensional coordinate color information calculation means calculates color information at each three-dimensional coordinate of the subject obtained by the three-dimensional coordinate calculation means. As described above, the three-dimensional coordinates of the subject, that is, the three-dimensional shape, can be obtained using the subject image of green light. Based on this, it is also possible to calculate the coordinates of the subject images of red and blue light that are imaged in the imaging regions 103c and 103d. Based on the coordinates of these colors, the colors of the subject are synthesized.

FIG. 6 is a view of the subject and the imaging region 103c of the imaging unit 100 as viewed from the positive direction side of the Y axis of the three-dimensional coordinates in FIG. In the illustration of Fig. 6, the relationship of the following equation (7) can be derived from the geometric calculation, and the value of xpc can be obtained.

[0052] Equation (7) xpc = xOc + Xp -f / | Zp |

FIG. 7 is a diagram in which the positive side force of the X axis of the three-dimensional coordinates in FIG. In the illustration of Fig. 7, the relationship of the following equation (8) can be derived from the geometric calculation, and the value of ypc can be obtained.

[0053] Equation (8) ypc = yOc-(Yp + tac) -f / | Zp |

Where tac is the Y-axis direction component of the distance between the optical axes of the single lenses 101a and 101c. From the above, in the imaging region 103c, the red component at one point P of the subject is imaged on pc (xpc, ypc) obtained by the equations (7) and (8).

FIG. 8 is a diagram of the subject and the imaging region 103d of the imaging unit 100 viewed from the positive direction side of the Y axis of the three-dimensional coordinates in FIG. In the illustration of Fig. 8, the relationship of the following formula (9) can be derived from the geometric calculation. From Equation (9), it is possible to obtain xpd of the coordinates pd (xpd, ypd) of the blue light emitted from P on the imaging region 103d.

[0056] Equation (9) xpd = xOd + (Xp -tadx) -f / | Zp |

Where tadx is the distance in the X-axis direction between the optical axes of the single lenses 101a and 101d.

FIG. 9 is a diagram in which the positive side force of the X axis of the three-dimensional coordinates in FIG. 2 also looks at the subject and the imaging region 103 d of the imaging unit 100. In the illustration of FIG. 9, the relationship of the following formula (10) can be derived from the geometric calculation. From equation (10), the ypd of the coordinates pd (xpd, ypd) of the blue light emitted from P on the imaging region 103d can be obtained.

[0058] Equation (10) ypd = yOd- (Yp + tady) -f / | Zp |

Where tady is the distance in the Y-axis direction between the optical axes of the single lenses 101a and 101d.

As described above, in the imaging region 103d, the blue component at the point P of the subject is imaged on pc (xpd, ypd) obtained by the equations (9) and (10).

[0060] In this way, it is possible to calculate a position where an image is formed on the one-point repulsive force imaging areas 103c and 103d represented by the three-dimensional coordinates of (Xp, Yp, Zp). By the same calculation, it is possible to calculate the position where the entire object is imaged on the imaging regions 103c and 103d.

[0061] Accordingly, since the coordinates of the images of the green, red, and blue imaging regions corresponding to each point of the subject can be obtained, by combining the information of the subject image at each of these coordinates, The color of each point can be obtained. By performing the same calculation for the entire subject, the overall color of the subject can be obtained. That is, in addition to obtaining the 3D shape of the entire subject by calculation by the 3D coordinate calculation means, the color of the entire subject can also be obtained by calculation by the 3D coordinate color information calculation means.

[0062] As described above, according to the present embodiment, the three-dimensional shape and color of a subject can be obtained with a thin shape. A configuration using multiple normal imaging systems with multiple lenses stacked, When acquiring the dimensional shape and color, the color is acquired from a single imaging system. In such a configuration, the apparatus becomes large. In the present embodiment, colors corresponding to the three-dimensional shape are extracted from the red, blue, and green optical systems that do not overlap each other, which is advantageous for thinning.

In FIG. 1, the information obtained by the 3D image calculation unit 110 is output to the viewpoint image output unit 111 of FIG. 1 to obtain a stereoscopic image, as described in the second embodiment below. You may force it to output directly to the display 112.

[0064] If there is no pixel corresponding to the calculated value of Equation (7) to Equation (10) in the imaging region, the subject image is calculated by interpolation or extrapolation of adjacent pixels. You can.

[0065] In the present embodiment, an optical system corresponding to three colors of red, blue, and green is used for color acquisition. However, complementary colors of cyan, magenta, and yellow can be added, The color may be reproduced by other methods. When a complementary color system is added for color acquisition, an optical system (lens), a wavelength selection region (color filter) corresponding to this, and an imaging region are added according to the number of added colors.

[0066] (Embodiment 2)

The second embodiment of the present invention will be described below. The present embodiment is premised on the first embodiment. The three-dimensional image calculation unit 110 in FIG. 1 calculates the three-dimensional shape of the subject, that is, the three-dimensional coordinates, and the color of the subject through the processing described in the first embodiment, and outputs it to the viewpoint image output unit 111. .

Next, an arbitrary viewpoint image output unit included in the viewpoint image output unit 111 of FIG. 1 will be described. The arbitrary viewpoint image output means outputs the subject image at an arbitrary viewpoint based on the three-dimensional coordinates of the subject obtained by the three-dimensional coordinate calculation means and the color information at each three-dimensional coordinate obtained by the three-dimensional coordinate color information calculation means. Create and output.

FIG. 10 is a diagram for explaining the principle of the viewpoint image calculation method according to the present embodiment. As shown in Fig. 10, a virtual lens with Rl (vxl, vyl, vzl) as the principal point at an arbitrary position in 3D coordinates and a plane perpendicular to the optical axis, and virtual imaging with an imaging distance fr Set area 103rl. Here, the optical axis of the virtual lens is set parallel to the Z axis of the three-dimensional coordinates. Provisional When viewed from the opposite side of the virtual lens, the virtual imaging area 103rl is the origin (0, 0) at the lower right of the imaging area, the horizontal line that is positive in the left direction is the xrl axis, and the upward direction is positive A two-dimensional coordinate system with a vertical line as the yrl axis is set.

The coordinates 104rl (xOrl, yOrl) are two-dimensional coordinate points on the virtual imaging area 103rl, and are intersection points between the virtual imaging area 103rl and the optical axis of the virtual lens.

[0070] At this time, the coordinates prl (xprl, yprl) on the virtual image area 103rl where the point P (Xp, Yp, Ζρ) of the subject is imaged through the virtual lens are obtained. FIG. 11 shows a view of the subject, the virtual lens, and the virtual imaging region 103rl from the positive direction of the Y axis of the three-dimensional coordinates in FIG. In the illustration of FIG. 11, the relationship of the following formula (11) can be derived from the geometric calculation, and xprl can be obtained.

[0071] Formula (11) xprl = xOrl + (Xp-vxl) -fr / | Zp-vzl |

FIG. 12 shows the X-axis positive direction force of the three-dimensional coordinates in FIG. In the illustration of FIG. 12, the relationship of the following formula (12) can be derived from the geometric calculation, and yprl can be obtained.

[0072] Equation (12) yprl = yOrl-(Yp-vyl) -fr / | Zp-vzl |

In this way, the imaging point prKxprl, yprl) on the virtual imaging region 103rl of one point P (Xp, Yp, Zp) of the subject can be calculated. By the same calculation, the imaging point on the virtual imaging area 103rl of the entire subject can be obtained. Hereinafter, an image formed on the virtual imaging region 103r 1 is referred to as a viewpoint image rl. Similarly, an image formed on the virtual imaging area 103r2 described later is referred to as a viewpoint image r2, and an image formed on the virtual imaging area 103r3 is referred to as a viewpoint image r3, and is formed on the virtual imaging area 103r4. The resulting image is called viewpoint image r4.

[0073] Next, the principal point of the virtual lens is R2 (vx2, vyl, vzl), that is, the viewpoint image r2 when Rl is translated in the X-axis direction of the three-dimensional coordinates is obtained. Similar to the viewpoint image rl, the virtual imaging region 103r2 is arranged at the imaging distance fr on a plane perpendicular to the optical axis. When viewed from the opposite side of the virtual lens, the virtual imaging area 103r2 has the lower right of the imaging area as the origin (0, 0), the horizontal line that is positive in the left direction is the xr2 axis, and the upward direction is A two-dimensional coordinate system with a positive vertical line ¾ yr2 is set. Coordinate 104r2 (x0r2, y0r2) is the virtual imaging area 103r2 It is the point of the upper two-dimensional coordinates and the intersection with the optical axis of the virtual lens.

[0074] From the same calculation as the viewpoint image rl, the imaging point pr2 (xpr2, ypr2) on the virtual imaging region 10 3r2 of one point P (Xp, Yp, Ζρ) of the subject is expressed by Equations (13) and (14). Is required. In other words, the expression (13) ί or the expression (11) is replaced by xOrl, x0r2, and vxl, vx2. Equation (14) is obtained by replacing yOrl with y0r2 in equation (12).

[0075] Formula (13) xpr2 = x0r2 + (Xp-vx2) -fr / | Zp-vzl |

Formula (14) ypr2 = y0r2- (Yp-vyl) -fr / | Zp-vzl |

In this way, the imaging point pr2 (xpr2, ypr2) on the virtual imaging region 103r2 of the point P (Xp, Yp, Zp) of the subject can be calculated. By the same calculation, the imaging point on the virtual imaging area 103r2 of the entire subject can also be obtained.

Next, the principal point of the virtual lens is R3 (vx3, vyl, vzl), that is, the viewpoint image r3 when R2 is translated in the X-axis direction of the three-dimensional coordinates is obtained. Similar to the viewpoint image r2, the virtual imaging area 103r3 is arranged at the imaging distance fr on a plane perpendicular to the optical axis. The virtual imaging area 103r3 has an origin (0, 0) in the lower right corner of the imaging area when the force opposite to the virtual lens is viewed, the horizontal line that is positive in the left direction is the xr3 axis, and the upward direction is positive A two-dimensional coordinate system with the vertical line yr3 is set. Coordinates 104r3 (x0r3, _y 0r3) is the point of the two-dimensional coordinates on the virtual imaging region 103R3, which is an intersection between the optical axis of the virtual lens. From the same calculation as the viewpoint image r2, the image point pr3 (x pr3, ypr3) on the virtual imaging area 103r3 of the point P (Xp, Yp, Zp) of the subject is obtained by the equations (15) and (16). . That is, Expression (15) is obtained by replacing xOrl with x0r3 and vxl with vx3 in Expression (11). Equation (16) is obtained by replacing y0rl¾y0r3 in equation (12).

[0077] (Formula 15) xpr3 = x0r3 + (Xp-vx3) -fr / | Zp-vzl |

(Formula 16) ypr3 = y0r3- (Yp-vyl)-fr / | Zp-vzl |

In this way, the imaging point pr3 (xpr3, ypr3) on the virtual imaging area 103r3 of the point P (Xp, Yp, Zp) of the subject can be calculated. Further, the image formation point on the virtual imaging region 103r3 of the entire subject can be obtained by the same calculation.

Next, the principal point of the virtual lens is R4 (vx4, vyl, vzl), that is, the viewpoint image r4 when R3 is translated in the X-axis direction of the three-dimensional coordinates is obtained. As with the viewpoint image r3, its virtual imaging area The region 103r4 is arranged at a position of the imaging distance fr on a plane perpendicular to the optical axis. In the virtual imaging area 103r4, when viewing the force opposite to the virtual lens, the lower right of the imaging area is the origin (0, 0), the horizontal line that is positive in the left direction is the xr4 axis, and the upward direction is positive A two-dimensional coordinate system is set with the vertical line as the yr4 axis. Coordinates 104r4 (x0r4, _y 0r4) is the point of the two-dimensional coordinates on the virtual imaging region 103R4, which is an intersection between the optical axis of the virtual lens.

[0079] From the same calculation as the viewpoint image r3, the imaging point pr4 (xpr4, ypr4) on the virtual imaging region 10 3r4 of one point P (Xp, Yp, Ζρ) of the subject is expressed by equations (17) and (18). Is required. In other words, equation (17) ί or equation (11) [koo! /,, XOrl x0r4, vxl vx4, respectively, are replaced. Equation (18) is obtained by replacing yOrl with y0r4 in equation (12).

[0080] (Equation 17) xpr4 = x0r4 + (Xp-vx4) -fr / | Zp-vzl |

(Formula 18) ypr4 = y0r4- (Yp-vyl)-fr / | Zp-vzl |

In this way, the image point p r4 (xpr4, ypr4) on the virtual imaging region 103r4 of the point P (Xp, Yp, Zp) of the subject can be calculated.

[0081] Further, the image formation point on the virtual imaging region 103r4 of the entire subject can also be obtained by the same calculation.

As described above, for each point in the entire subject, an imaging point in each of the virtual imaging regions 103rl to 103r4 can be obtained. As described above, color information corresponding to each point of the subject has already been obtained. For this reason, the color information of each point of each viewpoint image uses the red, green, and blue color information of the object obtained by the three-dimensional coordinate color information calculation means of the three-dimensional coordinate calculation unit 110 in FIG. Can do. Therefore, a viewpoint image rl, a viewpoint image r2, a viewpoint image r3, and a viewpoint image r4 that are arbitrary plural viewpoint images are obtained. Note that the point corresponding to the calculated value of equation (11) force equation (18) does not exist in the coordinates on the viewpoint image, such as when the calculated value of equation (11) force equation (18) is decimal. Can apportion points to nearby coordinates by interpolation or extrapolation.

That is, according to the present embodiment, based on the 3D coordinates and color of the subject obtained by the 3D coordinate calculation unit 110, the viewpoint image output unit 111 performs a viewpoint from an arbitrary viewpoint of the subject. An image can be obtained. Convert the calculated multiple viewpoint images into an image format for stereoscopic display and output the image data to the display 112 in FIG. The

[0084] Needless to say, even when the optical axis of the virtual lens is not parallel to the Z axis, the imaging point on the imaging region set by the geometric calculation can be obtained.

The display 112 in FIG. 1 outputs a stereoscopic video based on the image data output from the viewpoint image output unit 111. Here, a general image display method for a stereoscopic display using four-viewpoint images will be described. Figure 13 shows a 3D display using a lenticular lens. FIG. 13 (a) is a perspective view, and FIG. 13 (b) shows a view of the display shown in FIG. 13 (a) as viewed from above.

The configuration shown in FIG. 13 is a configuration in which a lenticular lens 121 is arranged on the front surface of a two-dimensional display 120 such as a general LCD method, plasma method, SED method, FED method, or projection method. Yes. In Fig. 13 (b), in order to explain the image display method of a three-dimensional display using four-viewpoint images, a single lens-shaped lens width Wr of the lenticular lens 121 and a four-pixel width of the two-dimensional display 120 (Fig. 14). (Ref.) Shows a figure that is almost the same. The lenticular lens 121 has a one-row kamaboko-shaped lens width Wr that is set in accordance with the number of viewpoints of the stereoscopic image.

FIG. 14 shows the principal ray directions of the light emitted by the four sets of pixels in the two-dimensional display corresponding to the lens of one line of the lenticular lens in FIG. 13 (b). Light travels from pixel 125 in the direction of chief ray 129. Light travels from the pixel 126 in the direction of the principal ray 130. Light travels from the pixel 127 in the direction of the principal ray 131. Light travels from pixel 128 in the direction of chief ray 132. Since the light traveling directions of the four sets of pixels differ in this way, the human eye, which is generally about 6.5 cm away, forms images with different viewpoints between the right eye and the left eye. Become. For this reason, a stereoscopic image can be displayed.

[0088] Since four viewpoint images rl, r2, r3, r4 are used here, even if the eye position (both right eye and left eye) is moved to the left and right in FIG. In addition, since the right eye and the left eye still form different images with different viewpoints, in general, stereoscopic images are displayed with a wider viewing angle than when stereoscopic images are displayed based on two viewpoint images or three viewpoint images. Can be displayed.

[0089] In general, the more a stereoscopic image is displayed based on many viewpoint images, the wider the viewing angle force is. It becomes possible to see the video, and a more natural stereoscopic video can be obtained. Therefore, the more viewpoint images are created based on the three-dimensional coordinates of the subject obtained by the three-dimensional image calculation unit 110, it is possible to display a natural stereoscopic video with a wider viewing angle. For example, in the above example, it is possible to create five or more viewpoint images that are larger than the number of force optical systems that created the same four viewpoint images as the number of optical systems.

FIG. 15 shows two different viewpoint images. FIG. 15 (a) shows the viewpoint image rl in FIG. 10, and FIG. 15 (b) shows the viewpoint image r4 in FIG. Since the subject image viewed from the lens side is upside down in the imaging region, the yrl axis and yr4 axis are converted upside down from FIG. 10 in FIGS. 15 (a) and 15 (b). As shown in FIG. 15 (a), the subject appears in the lower right of the image in the viewpoint image rl, and the subject appears in the lower left of the image in the viewpoint image r4 as shown in FIG. 15 (b). Needless to say, in the viewpoint images r2 and r3, the subject appears in a position between the positions of the subject in FIGS. 15 (a) and 15 (b).

[0091] By arranging the viewpoint image rl in the pixel 128 in FIG. 14, the viewpoint image r2 in the pixel 127, the viewpoint image r3 in the pixel 126, and the viewpoint image r4 in the pixel 125, the four viewpoints are arranged. It is possible to realize a three-dimensional display using images. That is, the display 112 in FIG. 1 receives the image data arranged as described above from the viewpoint image output unit 111 and displays a natural stereoscopic video based on the image data.

As described above, according to the present invention, it is possible to realize a three-dimensional imaging apparatus that is thin and capable of obtaining a three-dimensional shape of a subject and that outputs natural stereoscopic video image data.

This embodiment may be configured to include coordinate conversion means. Specifically, the three-dimensional coordinates of the subject calculated by the three-dimensional coordinate calculation unit 110 in FIG. 1 are transformed by using affine transformation to obtain a force viewpoint image, thereby obtaining a more realistic three-dimensional image. Video can be realized. For example, as shown in FIG. 16, a part or the whole of the subject is translated by a predetermined distance mz in the negative direction of the Z axis of the three-dimensional coordinates and set to P ′, so that Zp becomes small. Can increase the parallax of the subject (see Equation (1)).

That is, in the example of FIG. 15, the subject image of the viewpoint image rl of FIG. 15 (a) is the subject image of the viewpoint image r4 of FIG. 15 (b) on the right side of the diagram (positive direction of the xrl axis). Is the left side of the figure (the negative of xr4 axis) Direction). This makes it possible to realize a more realistic 3D image because the 3D image appears to jump out from the 3D display.

Further, this embodiment may be configured to include shape conversion means. Specifically, a portion of the three-dimensional shape of the subject calculated by the three-dimensional coordinate calculation unit 110 in FIG. 1 is enlarged as shown in FIG. 17 to reduce the occlusion generated in each viewpoint image. By doing so, more natural 3D images can be obtained.

[0096] Note that the coordinate conversion means and the shape conversion means may include the viewpoint image image output unit 111 in FIG.

If the display 112 in FIG. 1 is a parallax barrier type stereoscopic display, a slit realized by liquid crystal or the like is disposed instead of the lenticular lens 121 in FIG. 13 (a).

[0098] Also, in the case of a depth sample 3D display or a 3D display using a hologram, a 3D image is output based on the 3D coordinates of the object created by the 3D coordinate calculation unit 110 in FIG. Needless to say, you can!

[0099] In the case of a glasses-type stereoscopic display such as a stereo pair method, a polarization method, or a time-division method, a stereoscopic image can be output based on the three-dimensional coordinates of the subject created by the three-dimensional coordinate calculation unit 110 in FIG. Needless to say.

[0100] Since the viewpoint image output unit 111 in FIG. 1 can output a subject image of an arbitrary viewpoint, the display 112 can be used as a general two-dimensional display and can output a free viewpoint image used for in-vehicle peripheral monitoring. Needless to say.

[0101] In Embodiments 1 and 2, it goes without saying that the lenses 101a, 101b, 101c, and lOld in FIG. 1 may be individually molded instead of being integrally molded. In addition, the arrangement of the colors of lenses 101a, 101b, 10 lc, and lOld is not limited to that shown in Fig. 1.As shown in Fig. 18 (a), the green lenses are diagonally arranged, and the red and blue lenses are arranged. It may be arranged at another diagonal, or as shown in Fig. 18 (b), it may be arranged in an arbitrary manner by arranging four lenses on the same line.

[0102] At this time, the position of the principal point of the geometric calculation lens in FIG. 2 of the present embodiment may be set appropriately. At the same time, color filters are appropriately arranged according to the colors. Also, as shown in Fig. 19 (a) and Fig. 19 (b), even if there are 3 or more green lenses, at least 2 Three-dimensional coordinates can be obtained using a single lens. The selection of two of the multiple lenses may be changed according to the subject distance and subject position. In this case, the lens may be arranged arbitrarily. In this embodiment, a plurality of green lenses are arranged to calculate the three-dimensional shape of the subject. However, if the green is not green, a plurality of lenses are arranged to calculate the three-dimensional shape of the subject. Oh, needless to say.

[0103] Even when the imaging regions 103a, 103b, 103c, and 103d in Fig. 1 are not on the same plane, from the positional relationship, by subjecting the subject image to coordinate conversion and mapping on the desired same plane, It can be handled in the same manner as this embodiment.

[Embodiment 3]

The third embodiment relates to an integrated circuit. The three-dimensional coordinate calculation unit 110 and the viewpoint image output unit 111 in FIG. 1 are typically realized as an LSI that is an integrated circuit. These may be individually chipped, or may be chipped to include some or all of them. Here, it is sometimes called IC, system LSI, super LSI, or unroller LSI, depending on the difference in power integration.

[0105] Further, the method of circuit integration may be realized with a dedicated circuit or a general-purpose processor, not limited to LSI. You can use a field programmable gate array (FPGA) that can be programmed after manufacturing the LSI, or a reconfigurable processor that can reconfigure the connection and settings of the circuit cells inside the LSI.

[0106] Furthermore, if integrated circuit technology that replaces LSI emerges as a result of progress in semiconductor technology or other derived technology, it is naturally also possible to perform functional block integration using this technology. For example, biotechnology or the like can be applied.

Industrial applicability

As described above, the imaging apparatus of the present invention is thin, can determine the three-dimensional shape of the subject, and can output image data for free viewpoint video and stereoscopic video of the subject. It is useful as an original imaging device. The integrated circuit of the present invention is useful as an integrated circuit used in a three-dimensional imaging device.

Claims

The scope of the claims

[1] Multiple optical systems,

A plurality of wavelength selection regions that selectively transmit light in a specific wavelength region of light of subject power;

A plurality of imaging regions that output image information according to the input light,

An imaging apparatus in which the wavelength selection region and the imaging region are arranged in a one-to-one correspondence on each optical axis of the plurality of optical systems,

The plurality of wavelength selection regions include a first wavelength selection region that selectively transmits light in the same wavelength region, and a second wavelength selection region that selectively transmits light in a wavelength region different from the first wavelength selection region. Including

The first wavelength selection region includes two or more wavelength selection regions, and the second wavelength selection region includes two or more wavelength selection regions that selectively transmit light in different wavelength regions,

Based on image information output by at least two imaging regions corresponding to the first wavelength selection region and a positional relationship between the optical systems corresponding to the first wavelength selection region and the imaging regions. The three-dimensional coordinate calculation means for calculating the three-dimensional coordinates of the subject and a portion corresponding to each coordinate of the three-dimensional coordinates of the subject in at least three imaging regions that receive light in different wavelength regions are output. An imaging apparatus comprising: three-dimensional coordinate color information calculation means for calculating color information at each coordinate of the three-dimensional coordinates of the subject based on image information.

[2] The portions corresponding to the coordinates of the three-dimensional coordinates of the subject in each imaging region corresponding to the second wavelength selection region are the calculated three-dimensional coordinates of the subject and the second wavelength selection region. 2. The imaging device according to claim 1, wherein the imaging device calculates based on a positional relationship between each corresponding optical system and each imaging region.

[3] The imaging apparatus according to [1], further comprising coordinate conversion means for performing coordinate conversion on part or all of the three-dimensional coordinates of the subject.

[4] A shape change that enlarges a part or the whole of a three-dimensional shape based on the three-dimensional coordinates of the subject. The imaging apparatus according to claim 1, further comprising a replacement unit.

5. The imaging apparatus according to claim 1, further comprising arbitrary viewpoint image output means for outputting an image of the subject at an arbitrary viewpoint based on the three-dimensional coordinates of the subject and color information at each coordinate.

[6] Arbitrary viewpoint creation means for creating the subject image from any viewpoint by interpolation based on the three-dimensional coordinates of the subject is further provided, and the arbitrary viewpoint image output means is the subject image created by the interpolation. 6. The imaging apparatus according to claim 5, wherein an image at an arbitrary viewpoint is output based on.

7. The imaging apparatus according to claim 5, wherein the arbitrary viewpoint image output means outputs an image at an arbitrary viewpoint for stereoscopic video output.

8. The imaging apparatus according to claim 5, wherein the arbitrary viewpoint image output means outputs images at a larger number of viewpoints than the number of the optical systems.

[9] An integrated circuit that performs an operation based on image information output from an imaging region that receives light from a subject,

The imaging region includes a first imaging region that receives light of a first wavelength, and a second imaging region that receives light of a wavelength region different from the light of the first wavelength,

The first imaging region includes two or more imaging regions, and the second imaging region includes two or more imaging regions that selectively receive light in different wavelength regions, respectively. Circuit

Three-dimensional coordinate calculation means for calculating three-dimensional coordinates of the subject based on image information output from at least two of the first imaging areas, and receiving light in different wavelength ranges A tertiary that calculates color information at each coordinate of the three-dimensional coordinates of the subject based on image information output by a portion corresponding to each coordinate of the three-dimensional coordinates of the subject in at least three imaging regions. An integrated circuit comprising: original coordinate color information calculation means.