JP6310320B2 - Image processing apparatus, imaging apparatus, image processing method, and program - Google Patents

Description

  The present technology relates to an image processing device, an imaging device, an image processing method, and a program. Specifically, the present invention relates to an image processing apparatus, an imaging apparatus, an image processing method, and a program for converting an image.

  Conventionally, a fish-eye lens is used in an imaging apparatus in order to image a field of view with a wider viewing angle (for example, 180 degrees) than a general lens. When this fisheye lens is used, the shape of the subject and the shape of the image are not similar because of the optical characteristics, and for example, a fisheye image in which the shape of the image is distorted into a circle is obtained. Since it is difficult for a user to observe if the shape of the image remains distorted, an imaging device that corrects distortion of a fisheye image and converts it into a central projection image without distortion has been proposed (for example, see Patent Document 1). .) In the image conversion, all the coordinates in the central projection image are sequentially input to a predetermined coordinate conversion function in image conversion, and the coordinates in the fisheye image corresponding to the coordinates are calculated.

JP 2010-1118040 A

  However, in the above-described conventional technology, the amount of calculation increases as the number of pixels of the fisheye image increases. On the other hand, if the coordinates calculated in advance by the coordinate conversion function are stored in a storage unit such as a memory and read at the time of conversion, the calculation is not necessary. As a result, the capacity of the storage unit increases. Therefore, it is difficult to reduce both the calculation amount and the capacity of the storage unit.

  The present technology has been created in view of such a situation, and aims to reduce the amount of calculation and the capacity of a storage unit in image processing.

  The present technology has been made to solve the above-described problems, and a first aspect of the present technology is that some specific coordinates among the coordinates in the non-central projection image generated by the projection that does not correspond to the central projection. And a storage unit that stores the corresponding coordinates in the central projection image associated with the specific coordinates, and when one of the specific coordinates and the corresponding coordinates is input, the other coordinates corresponding to the input coordinates Based on the coordinate acquisition unit that acquires from the storage unit, an interpolation unit that interpolates coordinates different from the other coordinate based on the acquired other coordinate, and the other coordinate and the interpolation coordinate An image processing apparatus comprising: an image conversion unit that converts one of the non-central projection image and the central projection image into the other image; and an image processing method and the computer executing the method A program for causing. Thus, the interpolated coordinates are interpolated based on the specific coordinates or the corresponding coordinates, and one of the non-center projected image and the center projected image is converted into the other image based on the coordinates.

  In the first aspect, the coordinate acquisition unit acquires the specific coordinate as the other coordinate, and the interpolation unit rotates the specific coordinate around a predetermined coordinate in the non-centered projection image. Interpolated coordinates may be used as the interpolation coordinates. This brings about the effect that coordinates obtained by rotating specific coordinates are interpolated as interpolation coordinates.

  In the first aspect, the coordinate acquisition unit acquires the specific coordinate as the other coordinate, and the interpolation unit is a line segment connecting the predetermined coordinate and the specific coordinate in the non-centered projection image. The coordinates obtained by interpolating the upper coordinates as the interpolation coordinates and rotating the interpolation coordinates around the predetermined coordinates may be interpolated as the new interpolation coordinates. As a result, the coordinates on the line segment connecting the predetermined coordinates and the specific coordinates are interpolated as interpolation coordinates, and the coordinates obtained by rotating the interpolation coordinates are interpolated as new interpolation coordinates.

  In the first aspect, the coordinate acquisition unit acquires the specific coordinate as the other coordinate, and the interpolation unit interpolates a coordinate on a line segment connecting the two specific coordinates as the interpolation coordinate. May be. This brings about the effect that coordinates on a line segment connecting the two specific coordinates are interpolated as interpolation coordinates.

  In the first aspect, the specific coordinate may be a coordinate of a dividing point obtained by dividing each of the plurality of closed curves having different areas in the non-centered projection image. This brings about the effect that the interpolation coordinates are interpolated based on the coordinates of the dividing points obtained by dividing each of the closed curves.

  In the first aspect, the specific coordinate may be a coordinate of a division point obtained by dividing each of the closed curves by a smaller number of divisions as the area of the closed curve is smaller. As a result, the smaller the area of the closed curve is, the smaller the number of divisions is, and the interpolation coordinates are interpolated based on the coordinates of the division points obtained by dividing each of the closed curves.

  Further, in the first aspect, a perspective projection conversion unit that performs perspective projection conversion on the coordinates in the central projection image and inputs the coordinates obtained by the perspective projection conversion to the coordinate acquisition unit and the interpolation unit. In addition, the interpolation unit may interpolate the interpolated coordinates corresponding to the input coordinates. This brings about the effect | action that the interpolation coordinate corresponding to the coordinate which performed perspective projection transformation is interpolated.

  The first aspect may further include a rotation unit that rotates the coordinates in the central projection image around a predetermined axis, and the perspective projection conversion unit converts the perspective projection conversion with respect to the rotated coordinates. May be done. This brings about the effect that the coordinates obtained by rotating the coordinates in the central projection image around a predetermined axis are subjected to perspective projection transformation.

  Further, in the first aspect, a cylindrical projection conversion unit that performs cylindrical projection conversion on the coordinates in the central projection image and inputs the coordinates obtained by the cylindrical projection conversion to the coordinate acquisition unit and the interpolation unit. In addition, the interpolation unit may interpolate the interpolated coordinates corresponding to the input coordinates. This brings about the effect | action that the interpolation coordinate corresponding to the coordinate which performed cylindrical projection conversion with respect to the coordinate in a center projection image is interpolated.

  Further, in the first aspect, it is determined whether or not the captured image is the one image, and when the image is the one image, the one coordinate is transferred to the coordinate acquisition unit and the interpolation unit. An input determination unit may be further included, and the interpolation unit may interpolate the interpolated coordinates corresponding to the input coordinates. Thereby, when the imaged image is one image, the interpolated coordinates are interpolated.

  In addition, the second aspect of the present technology provides an imaging unit that captures one of a non-central projection image and a central projection image generated by a projection that does not correspond to the central projection, and coordinates of the non-central projection image. A storage unit that stores some specific coordinates and corresponding coordinates in the central projection image that are associated with the specific coordinates, and the input coordinates when one of the specific coordinates and the corresponding coordinates is input. A coordinate acquisition unit that acquires the other coordinate corresponding to the coordinate from the storage unit, an interpolation unit that interpolates a coordinate different from the other coordinate based on the acquired other coordinate as an interpolation coordinate, and the other coordinate An image pickup apparatus comprising: an image conversion unit that converts one of the non-center projected image and the center projected image into the other image based on the interpolation coordinates. Thus, the interpolated coordinates are interpolated based on the specific coordinates or the corresponding coordinates, and one of the non-center projected image and the center projected image is converted into the other image based on the coordinates.

  According to the present technology, it is possible to achieve an excellent effect that the amount of calculation and the capacity of the storage unit can be reduced in image processing. Note that the effects described here are not necessarily limited, and may be any of the effects described in the present disclosure.

1 is a block diagram illustrating a configuration example of an imaging apparatus according to a first embodiment. It is a block diagram which shows one structural example of the coordinate transformation part in 1st Embodiment. It is a figure which shows an example of the fish-eye image and center projection image in 1st Embodiment. It is a figure which shows an example of the fisheye image distortion correction table in 1st Embodiment. It is a block diagram which shows the example of 1 structure of the read-out coordinate output part in 1st Embodiment. It is a block diagram which shows the example of 1 structure of the read coordinate rotation matrix calculating part in 1st Embodiment. It is a figure which shows the example of arrangement | positioning of the mesh intersection in 1st Embodiment. It is a figure which shows an example of the output image plane in 1st Embodiment. It is a figure which shows an example of the output image plane rotated in 1st Embodiment. It is a figure which shows an example of the area | region on the virtual hemisphere which projected the output image plane in 1st Embodiment. It is a figure which shows the read-out coordinate and output coordinate in 1st Embodiment. It is a figure which shows the mesh intersection coordinate and interpolation coordinate in 1st Embodiment. It is a figure which shows an example of the fish-eye image and center projection image in 1st Embodiment. 3 is a flowchart illustrating an example of an operation of the imaging apparatus according to the first embodiment. It is a flowchart which shows an example of the image conversion process in 1st Embodiment. It is a flowchart which shows an example of the coordinate interpolation process in 1st Embodiment. It is a block diagram which shows an example of the coordinate transformation part in 2nd Embodiment. It is a figure which shows an example of the projection range and projection surface in 2nd Embodiment. It is a flowchart which shows an example of the image conversion process in 2nd Embodiment. It is a figure which shows the example of arrangement | positioning of the mesh intersection in 3rd Embodiment. It is a figure which shows the example of arrangement | positioning of the mesh intersection which made the density equal in 3rd Embodiment. It is a block diagram which shows the example of 1 structure of the imaging system in 4th Embodiment. It is a block diagram which shows an example of the coordinate transformation part in 4th Embodiment. It is a block diagram which shows the example of 1 structure of the read-out coordinate output part in 4th Embodiment.

Hereinafter, modes for carrying out the present technology (hereinafter referred to as embodiments) will be described. The description will be made in the following order.
1. First embodiment (an example of converting an image by interpolating coordinates)
2. Second Embodiment (Example in which an image is converted by interpolating coordinates from cylindrically projected coordinates)
3. Third embodiment (an example in which an image is converted by interpolating coordinates from mesh intersections of equal density)
4). Fourth embodiment (example in which an image is converted by interpolating coordinates when a fisheye image is captured)

<1. First Embodiment>
[Configuration example of imaging device]
FIG. 1 is a block diagram illustrating a configuration example of the imaging apparatus 100 according to the embodiment. The imaging apparatus 100 includes a fisheye lens 110, an imaging element 120, an input signal processing unit 130, a frame memory 140, a control unit 150, an image conversion unit 160, a pixel interpolation unit 170, an output signal processing unit 180, and a recording unit 190. In addition, the imaging apparatus 100 includes a coordinate conversion unit 200.

  The fisheye lens 110 condenses light by a projection other than the central projection (for example, equidistant projection) and guides it to the image sensor 120. Note that the projection method of the fisheye lens 110 is not limited to equidistant projection as long as it is other than the central projection. For example, orthographic projection or equisolid angle projection may be used.

  The image sensor 120 converts the light from the fisheye lens 110 into an electrical signal under the control of the control unit 150 to generate an image. For example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor is used as the image sensor 120. The image sensor 120 supplies the generated image as an input image to the input signal processing unit 130 via the signal line 129. This input image is composed of a plurality of pixels arranged in a two-dimensional grid and includes a circular fisheye image. The imaging element 120 is an example of an imaging unit described in the claims.

  The input signal processing unit 130 performs various signal processing such as noise removal processing, demosaic processing, white balance processing, and YC conversion processing on the input image as necessary. The input signal processing unit 130 supplies the input image subjected to signal processing to the frame memory 140 via the signal line 139. The frame memory 140 holds an input image.

  The control unit 150 controls the entire imaging apparatus 100. The control unit 150 controls the image sensor 120 according to a user operation or the like to capture an image. Further, the control unit 150 sequentially supplies a plurality of output coordinates to the image conversion unit 160 and the coordinate conversion unit 200 via the signal line 159, and performs coordinate conversion of the zoom magnification, the output image size, the pan angle, the tilt angle, and the roll angle. Supplied to the unit 200.

  Here, the output coordinates are coordinates in the central projection image generated from the fisheye image. This central projection image is composed of a plurality of pixels arranged on a two-dimensional grid. In the central projection image, an array composed of pixels arranged in a predetermined direction (for example, the horizontal direction) is called a row. In supplying the output coordinates, each of the rows is sequentially selected, and each of the coordinates of the pixels in the selected row is sequentially supplied as an output coordinate.

  The output image size is the size of the central projection image. The zoom magnification indicates the ratio of the output image size to the output image plane. The output image plane is a rectangular projection plane on which at least a part of the fisheye image is projected by perspective projection, and an image obtained by enlarging the output coordinate plane with the zoom magnification is generated as a central projection image. The pan angle, tilt angle, and roll angle will be described later.

  The coordinate conversion unit 200 converts output coordinates into readout coordinates. Here, the readout coordinates indicate coordinates in the fisheye image. Each time the output coordinate is supplied, the coordinate conversion unit 200 converts the output coordinate into a corresponding readout coordinate and supplies it to the image conversion unit 160 via the signal line 209.

  The image conversion unit 160 converts the fisheye image into a central projection image. Each time the image conversion unit 160 receives output coordinates from the control unit 150, the image conversion unit 160 reads the pixel values of the read coordinates corresponding to the output coordinates from the frame memory 140. Then, the image conversion unit 160 supplies the read pixel value to the pixel interpolation unit 170 via the signal line 169 as the pixel value of the output coordinates in the central projection image. As a result, the fisheye image is converted into a central projected image.

  The pixel interpolation unit 170 interpolates pixels in the central projection image as necessary. For example, when part or all of a fisheye image is enlarged, the pixel interpolation unit 170 obtains necessary pixels with subpixel accuracy and performs interpolation. In this interpolation, an algorithm such as a bilinear interpolation algorithm, a bicubic interpolation algorithm, or a Lanzos interpolation algorithm is used. The pixel interpolation unit 170 supplies the central projection image obtained by interpolating the pixels to the output signal processing unit 180 via the signal line 179.

  The output signal processing unit 180 performs OSD (On Screen Display) processing, mask processing, image format conversion processing, and the like on the central projection image as necessary. The output signal processing unit 180 supplies the processed central projection image to the recording unit 190 via the signal line 189. The recording unit 190 records a central projected image.

  Note that the imaging apparatus 100 converts the fisheye image generated by the fisheye lens 110 into a central projection image as a conversion target. However, if the conversion target is an image generated by a projection other than the central projection, the fisheye image It is not limited to. For example, the imaging apparatus 100 may convert an image obtained by imaging a subject projected on a curved mirror with a central projection lens.

  Further, the imaging apparatus 100 converts the fisheye image into the central projection image, but conversely, the central projection image may be converted into the fisheye image. In this case, for example, a central projection lens is provided instead of the fisheye lens 110, and the control unit 150 sequentially supplies read coordinates in the central projection image to the coordinate conversion unit 200 and the image conversion unit 160. In addition, the coordinate conversion unit 200 converts the read coordinates into output coordinates in the fisheye image.

  Moreover, although each part, such as the fisheye lens 110 and the image pick-up element 120, is set as the structure provided in one apparatus (image pick-up device 100), you may distribute and provide these in several apparatuses. For example, the fisheye lens 110 may be provided in the interchangeable lens unit, the image sensor 120 or the like may be provided in the image pickup apparatus 100, and the coordinate conversion unit 200 or the image conversion unit 160 may be provided in the image processing apparatus.

  In addition, the imaging apparatus 100 performs noise removal processing, demosaic processing, white balance processing, YC conversion processing, and the like in the input signal processing unit 130 before image conversion. At least one of these is performed after image conversion. May be. For example, a fish-eye image in RAW data format that has not been demosaiced may be converted into a central projected image and then demosaiced.

  Further, the imaging apparatus 100 may output the central projection image to the outside, or may further include a display unit and display the central projection image on the display unit. In addition, the imaging apparatus 100 records the central projection image in the recording unit 190. However, the image capturing apparatus 100 may perform output to the outside or display on the display unit without recording.

[Configuration example of coordinate conversion unit]
FIG. 2 is a block diagram illustrating a configuration example of the coordinate conversion unit 200 according to the first embodiment. The coordinate conversion unit 200 includes a coordinate normalization unit 210, an output coordinate rotation matrix calculation unit 220, a perspective projection conversion unit 230, a fisheye image distortion correction table 240, and a read coordinate output unit 300.

  Here, a predetermined axis parallel to the fisheye image is defined as the x axis, and an axis parallel to the fisheye image and orthogonal to the x axis is defined as the y axis. Further, an axis orthogonal to these x-axis and y-axis is taken as a z-axis. The origin of the x-axis, y-axis, and z-axis is, for example, the center of the fisheye image. And the surface of the hemisphere centering on the origin is called a virtual spherical surface. The virtual spherical surface represents the visual field range imaged by the fisheye lens 110. In the initial state, the output image plane (that is, the projection plane) is arranged, for example, at a position where the center coincides with the center of the fisheye image. In addition, the output image plane is arranged at a position where the center thereof is in contact with the virtual spherical surface in the initial state.

ｚ_norm＝１ ・・・式４ The coordinate normalization unit 210 normalizes output coordinates based on the zoom magnification and the output image size. For example, when the horizontal coordinate of the central projection image is 0 to out_h and the vertical coordinate is 0 to out_v, out_h and out_v are supplied as output image sizes. For example, the coordinate normalization unit 210 normalizes the output coordinates by the following expression.
r = min (out_h, out_v) / 2 Equation 1

z _norm = 1 Formula 4

In Equation 1, min (A, B) is a function that returns the smaller one of A and B. Further, in the expressions 2 and 3, “zoom” is obtained when the diameter of the fisheye image and the short side of the output image plane coincide with each other and the output image plane (that is, the projection plane) is arranged so as to touch the virtual sphere. The zoom magnification is set to “1”. In equations 2 to 4, x _norm , y _norm and z _norm are normalized x, y and z coordinates. The coordinate normalization unit 210 supplies the normalized output coordinates (x _norm , y _norm , z _norm ) to the output coordinate rotation matrix calculation unit 220. The output coordinates are normalized to the coordinates on the spherical surface of the hemisphere having the radius 1 by Expressions 1 to 4.

  Note that the imaging apparatus 100 enlarges at least a part of the fisheye image with the zoom magnification, but may reduce at least a part of the fisheye image. When the image is reduced, the control unit 150 supplies a reduction rate instead of the zoom magnification “zoom”. In this case, “zoom” in Expression 2 and Expression 3 is replaced with the reduction ratio.

The output coordinate rotation matrix calculation unit 220 rotates the output image plane by rotation matrix calculation. The output coordinate rotation matrix calculation unit 220 receives the pan angle, tilt angle, and roll angle from the control unit 150. Here, the pan angle is a rotation angle for rotating the output coordinates around the x axis. The tilt angle is a rotation angle for rotating the output coordinates around the y axis, and the roll angle is a rotation angle for rotating around the z axis. And the output coordinate rotation matrix calculating part 220 performs rotation matrix calculation by the following formula, for example.

In Equation 5, R _t is the tilt angle. R _r is the roll angle and R _p is the pan angle. Further, (x _rot , y _rot , z _rot ) are output coordinates after rotation. The output coordinate rotation matrix calculation unit 220 supplies the output coordinates (x _rot , y _rot , z _rot ) to the perspective projection conversion unit 230. The output coordinate rotation matrix calculation unit 220 is an example of a rotation unit described in the claims.

The perspective projection conversion unit 230 performs perspective projection conversion on the output coordinates. For example, the perspective projection conversion unit 230 performs perspective projection conversion according to the following expression.
x _sph = x _rot / (x _rot ² + y _rot ² + z _rot ² ) ^1/2 Equation 6
y _sph = y _rot / (x _rot ² + y _rot ² + z _rot ² ) ^1/2 Equation 7
z _sph = z _rot / (x _rot ² + y _rot ² + z _rot ² ) ^1/2 Equation 8
R _x = arctan2 (y _sph , x _sph ) Equation 9
R _z = arccos (z _sph ) Equation 10

In Expressions 6 to 8, x _sph , y _sph and z _sph are coordinates obtained by projecting the output coordinates to the coordinates on the surface of the virtual celestial sphere. In Expressions 9 and 10, arctan2 (y, x) is a function that returns an angle formed by a straight line connecting (y, x) and the origin and the x axis. Arccos represents an inverse function of a sine function. R _x and R _z indicate angles with respect to the x axis and the z axis in the output coordinates in the polar coordinate notation subjected to the projective projection transformation. The perspective projection conversion unit 230 supplies (R _x , R _z ) among the output coordinates (r, R _x , R _z ) subjected to the projection projection conversion to the read coordinate output unit 300. Here, r represents a radius in the polar coordinate system. The reason why r is not supplied is that r is set to a fixed value (for example, “1”).

  The read coordinate output unit 300 converts the output coordinates into read coordinates and outputs the read coordinates to the image conversion unit 160. When the readout coordinates corresponding to the output coordinates are stored in the fisheye image distortion correction table 240, the readout coordinate output unit 300 acquires the readout coordinates from the fisheye image distortion correction table 240 and outputs them. On the other hand, when the readout coordinates corresponding to the output coordinates are not stored in the fisheye image distortion correction table 240, the readout coordinate output unit 300 interpolates and outputs the readout coordinates corresponding to the output coordinates by a predetermined interpolation process. To do. Details of the interpolation processing will be described later.

  The fisheye image distortion correction table 240 stores some specific coordinates among the read coordinates in the fisheye image in association with the output coordinates in the central projection image.

  As described above, since the imaging apparatus 100 interpolates and obtains some coordinates in the fisheye image, the amount of calculation can be reduced compared to a configuration that calculates all coordinates in the fisheye image.

  The coordinate conversion unit 200 is provided with a normalization unit 210 and an output coordinate rotation matrix calculation unit 220 to perform panning, tilting, and rolling. However, if panning, tilting, and rolling are not performed, an output coordinate rotation matrix is used. The calculation unit 220 may not be provided.

  FIG. 3 is a diagram illustrating an example of a fish-eye image and a central projection image in the first embodiment. A in the same figure is a figure which shows an example of the input image 500 in 1st Embodiment. As illustrated in FIG. 5A, the input image 500 is, for example, a rectangular image composed of a plurality of pixels arranged in a two-dimensional lattice pattern. The input image 500 includes a circular image circle 501. The image circle 501 is an area obtained by converting the light collected by the fisheye lens 110, and an image composed of pixels in the circle is treated as a fisheye image. As described above, since the fish-eye image (501) is generated by projection other than the central projection, the actual shape of the subject does not resemble the shape of the image obtained by capturing the subject. For example, the shape of the image is distorted into a circle.

  Further, b in FIG. 3 is a diagram illustrating an example of the central projection image 600 generated from the fisheye image (501). As shown in b in the figure, the distortion of the fisheye image is corrected, and the shape of the image is similar to the actual shape of the subject.

  FIG. 4 is a diagram illustrating an example of the fisheye image distortion correction table 240 according to the first embodiment. In this fisheye image distortion correction table 240, some specific coordinates among the readout coordinates in the fisheye image are stored in association with the output coordinates in the central projection image. For example, read coordinates obtained by projecting a division point obtained by dividing a virtual spherical surface into a mesh shape onto a fisheye image are stored. The output coordinates corresponding to the read coordinates are, for example, coordinates calculated in advance by a coordinate conversion function described in paragraph 0018 of Patent Document 1. Instead of theoretically using a coordinate conversion function or the like, a hemispherical object on which mesh intersections are plotted may be actually prepared, and the object may be imaged and calibrated for each lens. Specifically, for example, the hemispherical object is imaged by each of the fisheye lens and the central projection lens, and the correspondence relationship between the mesh intersection coordinates and the mesh corresponding coordinates is obtained from the obtained fisheye image and the central projection image.

  Hereinafter, the readout coordinates obtained by projecting the mesh intersection point onto the fisheye image in parallel with the z axis are referred to as “mesh intersection point coordinates”, and the output coordinates in the central projection image corresponding to the mesh intersection point coordinates are referred to as “mesh correspondence coordinates”. The mesh corresponding coordinates are represented by, for example, a polar coordinate system, while the mesh intersection coordinates are represented by, for example, an orthogonal coordinate system. Note that the mesh intersection coordinates are an example of specific coordinates described in the claims, and the mesh corresponding coordinates are an example of corresponding coordinates described in the claims.

When the mesh intersection coordinates are (x ₁ , y ₁ ) and the mesh corresponding coordinates in the central projection image corresponding to the coordinates are (R _x1 , R _z1 ), the mesh corresponding coordinates (R _x1 , R _z1 ) The mesh intersection coordinates (x ₁ , y ₁ ) are stored in association with.

  FIG. 5 is a block diagram illustrating a configuration example of the read coordinate output unit 300 according to the first embodiment. The read coordinate output unit 300 includes an orthogonal coordinate acquisition unit 320 and a coordinate interpolation unit 330.

  The orthogonal coordinate acquisition unit 320 acquires readout coordinates corresponding to the output coordinates from the fisheye image distortion correction table 240. The orthogonal coordinate acquisition unit 320 determines whether or not read coordinates corresponding to the output coordinates supplied from the perspective projection conversion unit 230 are stored in the fisheye image distortion correction table 240, in other words, the output coordinates are mesh-corresponding coordinates. Judge whether there is. If the coordinates are mesh-corresponding coordinates, the orthogonal coordinate acquisition unit 320 acquires the mesh intersection coordinates (that is, read coordinates) corresponding to the mesh-corresponding coordinates from the fisheye image distortion correction table 240 and supplies them to the image conversion unit 160. .

On the other hand, when the output coordinates are not mesh-corresponding coordinates, the orthogonal coordinate acquisition unit 320 preferentially selects four meshes that are close to the output coordinates from among the mesh-corresponding coordinates around the supplied output coordinates. Select the corresponding coordinates. These four mesh-corresponding coordinates are (R _xn , R _zn ), (R _{xn + 1} , R _zn ), (R _xn , R _{zn + 1} ) and (R _{xn + 1} , R _{zn + 1} ), respectively. The orthogonal coordinate acquisition unit 320 acquires the mesh corresponding coordinates and the mesh intersection coordinates corresponding to the mesh corresponding coordinates from the fisheye image distortion correction table 240 and supplies them to the coordinate interpolation unit 330. The orthogonal coordinate acquisition unit 320 is an example of a coordinate acquisition unit described in the claims.

  The coordinate interpolation unit 330 interpolates read coordinates corresponding to the supplied output coordinates based on the mesh intersection coordinates acquired by the orthogonal coordinate acquisition unit 320. The coordinate interpolation unit 330 includes a post-rotation linear interpolation unit 331, a post-rotation interpolation coefficient calculation unit 332, a pre-rotation linear interpolation unit 333, a pre-rotation interpolation coefficient calculation unit 334, a rotation angle calculation unit 335, and a read coordinate rotation matrix calculation unit 340. Is provided. The coordinate interpolation unit 330 is an example of an interpolation unit described in the claims.

The pre-rotation interpolation coefficient calculation unit 334 calculates the interpolation coefficient _az used in the pre-rotation linear interpolation unit 333 and supplies it to the pre-rotation linear interpolation unit 333. The interpolation coefficient _az is calculated by the following equation, for example.
_az = _Rz / ( _{Rzn + 1- Rzn} ) ... Formula 11

The pre-rotation linear interpolation unit 333 interpolates coordinates between two mesh-corresponding coordinates adjacent in the radial direction as linear interpolation coordinates by linear interpolation and supplies them to the read coordinate rotation matrix calculation unit 340. The linear interpolation coordinates are obtained by the following equation, for example.
x _a ′ = f _x (R _xn , R _zn ) · (1−a _z ) + f _x (R _xn , R _{zn + 1} ) · a _z (12)
y _a ′ = f _y (R _xn , R _zn ) · (1−a _z ) + f _y (R _xn , R _{zn + 1} ) · a _z Equation 13
_{x b '= f x (R} xn + 1, R zn) · (1-a z) + f x (R xn + 1, R zn + 1) · a z ... Equation 14
y _b ′ = _fy (R _{xn + 1} , R _zn ) · (1−a _z ) + _fy (R _{xn + 1} , R _{zn + 1} ) · a _z Equation 15

In Expressions 12 to 15, f _x (R _x , R _z ) is a function that returns the x coordinate of the mesh intersection coordinates (x, y) corresponding to the mesh corresponding coordinates (R _x , R _z ). F _y (R _x , R _z ) is a function that returns the y coordinate of the mesh intersection coordinates (x, y) corresponding to the mesh corresponding coordinates (R _x , R _z ). From the equations 12 to 15, the coordinates on the line segment connecting the two mesh intersection coordinates are obtained as linear interpolation coordinates. In Expressions 12 to 15, (x _a ′, y _a ′) and (x _b ′, y _b ′) are linear interpolation coordinates.

The rotation angle calculation unit 335 calculates rotation angles R _xa and R _xb that rotate the linear interpolation coordinates. The rotation angles R _xa and R _xb are obtained by the following equations, for example.
R _xa = R _x −R _xn Equation 16
R _xb = R _x −R _{xn + 1} Equation 17

The rotation angle calculation unit 335 supplies, for example, the sine and cosine of the rotation angle R _xa to the read coordinate rotation matrix calculation unit 340, and then supplies the sine and cosine of the rotation angle R _xb to the read coordinate rotation matrix calculation unit 340.

The read coordinate rotation matrix calculation unit 340 rotates each of the linear interpolation coordinates (x _a ′, y _a ′) and (x _b ′, y _b ′) by rotation matrix calculation, and uses the rotated coordinates as rotation interpolation coordinates. The signals are sequentially supplied to the linear interpolation unit 331. For example, the coordinates after rotation are obtained by the following expression.

In formula _{18 (x rot_a ', y rot_a} ') is linear interpolation coordinate _{(x a ', y a'} ) a rotation interpolation coordinates rotate. In Expression 19, (x _{rot_b} ′, y _{rot_b} ′) are rotation interpolation coordinates obtained by rotating the linear interpolation coordinates (x _b ′, y _b ′).

Rotation after interpolation coefficient calculation unit 332, and supplies to the rotating front linear interpolation unit 333 calculates the interpolation coefficients a _x used in the post-rotation linear interpolation unit 331. Interpolation coefficients a _x, for example, is calculated by the following equation.
a _x = R _x / (R _{xn + 1} −R _xn ) Equation 20

The post-rotation linear interpolation unit 331 interpolates coordinates between two rotation interpolation coordinates as final interpolation coordinates by linear interpolation, and supplies the interpolated coordinates to the image conversion unit 160. The interpolation coordinates are obtained by the following formula, for example.
x ′ = x _rot — a ′ · (1−a _x ) + x _rot — b ′ · a _x Expression 21
y ′ = y _rot — a ′ · (1−a _x ) + y _rot — b ′ · a _x.

  The coordinate interpolation unit 330 performs linear interpolation and interpolation based on coordinate rotation, but may perform only linear interpolation. However, in a fish-eye image, since the image is distorted in a circular shape, when performing interpolation in the circumferential direction, it is desirable to perform interpolation by rotation using Equation 18 or Equation 19.

  Also, the coordinate interpolation unit 330 performs linear interpolation in the radial direction, but if interpolation between two adjacent coordinates in the radial direction can be appropriately interpolated, interpolation other than linear interpolation (spline interpolation) Etc.).

[Configuration example of readout coordinate rotation matrix calculation unit]
FIG. 6 is a block diagram illustrating a configuration example of the read coordinate rotation matrix calculation unit 340 according to the first embodiment. The read coordinate rotation matrix calculation unit 340 includes multipliers 341, 342, 343, and 344, a subtracter 345, and an adder 346.

The multiplier 341 multiplies the coordinate x _a ′ or x _b ′ by the sine of the angles R _xa and R _xb and supplies the multiplication result to the subtractor 345. The multiplier 342 multiplies the coordinate y _a ′ or y _b ′ by the cosine of the angle R _xa or R _xb and supplies the multiplication result to the subtractor 345. The subtracter 345 subtracts the multiplication result of the multiplier 342 from the multiplication result of the multiplier 341 and supplies the result to the post-rotation linear interpolation unit 331 as coordinates x _{rot_a} ′ or x _{rot_b} ′.

The multiplier 343 multiplies the coordinate x _a ′ or x _b ′ by the cosine of the angles R _xa and R _xb and supplies the multiplication result to the adder 346. The multiplier 344 multiplies the coordinate y _a ′ or y _b ′ by the sine of the angle R _xa or R _xb and supplies the multiplication result to the adder 346. The adder 346 adds the multiplication result of the multiplier 343 and the multiplication result of the multiplier 344 and supplies the result to the linear interpolation unit 331 after rotation as coordinates y _{rot_a} ′ or y _{rot_b} ′.

  FIG. 7 is a diagram illustrating an arrangement example of mesh intersections according to the first embodiment. The rectangular input image 500 includes a circular image circle 501 (that is, a fisheye image). A virtual spherical surface 502 whose origin is the center of the fisheye image is divided into meshes. In the division, for example, the latitude and longitude of the virtual spherical surface 502 are divided at equal intervals. The coordinates of the points obtained by projecting the divided division points (mesh intersections) 503 and 504 onto the fisheye image parallel to the z axis are held in the fisheye image distortion correction table 240 as mesh intersection coordinates 505 and 506.

  In FIG. 7, the mesh intersection coordinates are plotted on the premise of a fish-eye image in which the image is distorted into a circle by ideal projection. In this ideal state, the coordinates of a point obtained by equally dividing each of a plurality of concentric circles into a predetermined number are held as mesh intersection coordinates. However, the actual fisheye lens 110 is often not an ideal projection method due to product variations and the like, and not only the incident angle of the light beam with respect to the optical axis but also the theoretical value of the rotation angle around the optical axis. It may be out of place. Due to these factors, for example, the image may be distorted in an elliptical shape instead of a circular shape. In this case, as the mesh intersection coordinates, coordinates of points obtained by dividing each of a plurality of ellipses having the same center are held as mesh intersection coordinates. When a part of the image is distorted into an ellipse, mesh intersection coordinates are provided on the ellipse at that part. In this way, the coordinates of the dividing points obtained by equally dividing a plurality of closed curves (circles, ellipses, etc.) having different areas are held as mesh intersection coordinates.

  In this way, since only a part of the coordinates (for example, mesh intersection coordinates) of the fisheye image is stored in the fisheye image distortion correction table 240, it is compared with a configuration in which all the coordinates in the fisheye image are stored. The table size can be reduced. Therefore, the capacity of the storage unit (memory or the like) that holds the fisheye image distortion correction table 240 can be reduced.

  FIG. 8 is a diagram illustrating an example of the output image plane 510 according to the first embodiment. For example, a rectangular output image plane 510 is set in the image circle 501 (fisheye image). In the initial state, the output image plane 510 is arranged, for example, at a position where the center coincides with the center of the fisheye image. In addition, the output image plane 510 is arranged at a position where the center thereof is in contact with the virtual spherical surface 502 in the initial state.

  FIG. 9 is a diagram illustrating an example of the rotated output image plane 510 according to the first embodiment. For example, the output image plane 510 is rotated about the x, y, and z axes due to the pan angle, tilt angle, and roll angle.

FIG. 10 is a diagram illustrating an example of a region 511 obtained by perspectively projecting the output image plane 510 to the virtual spherical surface 502 (that is, center projection) in the first embodiment. The Cartesian coordinates (x _sph , y _sph , z _sph ) on this region 511 are calculated by the above formulas 6 to 8. The orthogonal coordinates are converted into polar coordinates (R _x , R _z ) by Expressions 9 and 10, and the polar coordinates are converted into readout coordinates on the fisheye image.

  FIG. 11 is a diagram showing read coordinates and output coordinates in the first embodiment. A in the same figure is a figure which shows an example of the read-out coordinate in the fish-eye image in which ideal projection was performed. Black circle pixels 601, 603, 607, and 609 in a in FIG. 6 indicate pixels at mesh intersection coordinates. Also, white circle pixels 602, 604, 605, 606, and 608 indicate interpolation coordinate pixels.

  Coordinates between mesh intersection coordinates adjacent in the circumferential direction are interpolated by rotation of at least one of the mesh intersection coordinates.

For example, when the coordinate conversion unit 200 interpolates coordinates between adjacent pixels 601 and 603 in the circumferential direction, first, the mesh intersection coordinates of the pixel 601 are set to (R _xn , R _zn ) and (R _xn , R _{zn + 1).} ) Set both. Also, the coordinate conversion unit 200 sets the mesh intersection coordinates of the pixel 603 to both (R _{xn + 1} , R _zn ) and (R _{xn + 1} , R _{zn + 1} ). With this setting, substantially no linear interpolation is performed before rotation.

The coordinate conversion unit 200 rotates the coordinates of the pixel 601 by the rotation angle R _xa to obtain rotation interpolation coordinates (x _{rot_a} ′, y _{rot_a} ′), and rotates the coordinates of the pixel 603 by the rotation angle R _xb. Interpolated coordinates (x _{rot_b} ', y _{rot_b} ') are obtained. In an ideal fisheye image, these rotational interpolation coordinates are the same coordinates. For this reason, substantially no linear interpolation is performed after rotation. The coordinate conversion unit 200 rotates both the coordinates of the pixels 601 and 603, but may perform interpolation by rotating only one of the coordinates.

  In addition, the coordinates between two mesh intersection coordinates adjacent in the radial direction are obtained by linear interpolation. For example, when the coordinate conversion unit 200 interpolates the coordinates between the adjacent pixels 601 and 607 in the radial direction, the coordinates of the pixel 604 are interpolated by linear interpolation without performing interpolation by rotation. For example, four mesh intersection coordinates around the pixel 604 are set, and the mixing ratio of interpolation coordinates by rotation is set to 0%.

Further, when interpolating the coordinates inside the region surrounded by the four mesh intersection coordinates, linear interpolation and interpolation by rotation are performed. For example, consider a case where the coordinates of the pixel 605 inside the four pixels 601, 603, 607 and 609 are interpolated. In this case, the mesh intersection coordinates of the pixels 601, 603, 607, and 609 are set to (R _xn , R _zn ), (R _{xn + 1} , R _zn ), (R _xn , R _{zn + 1} ) and (R _{xn + 1} , R _{zn + 1} ). The With this setting, the coordinates of the pixels 604 and 606 are interpolated by linear interpolation, and the interpolation coordinates of the pixel 605 are obtained by rotating these coordinates.

  B in FIG. 11 is a diagram illustrating output coordinates in the first embodiment. In FIG. 5b, black circle pixels 701, 703, 707, and 709 indicate mesh coordinate coordinates. Also, white circle pixels 702, 704, 705, 706, and 708 indicate pixels having coordinates corresponding to the interpolation coordinates. When the image conversion unit 160 outputs the pixel value of each pixel a in the figure as the pixel value of the output coordinate corresponding to the readout coordinate, a central projection image with corrected distortion is generated.

FIG. 12 is a diagram illustrating mesh intersection coordinates and interpolation coordinates according to the first embodiment. In the figure, it is assumed that the fish-eye image is not ideal and at least a part of the image is distorted into an ellipse. In this case, the coordinate conversion unit 200 performs linear interpolation along the radial direction, performs interpolation that rotates along the circumferential direction, and further performs linear interpolation to obtain final interpolation coordinates. For example, when the coordinates of the pixel 605 are interpolated, the mesh intersection coordinates of four pixels 601, 603, 607 and 609 around the pixel 605 are (R _xn , R _zn ), (R _{xn + 1} , R _zn ), (R _xn , R _{zn + 1} ) and (R _{xn + 1} , R _{zn + 1} ). With this setting, the coordinates of the pixels 604 and 606 are interpolated by linear interpolation, and the coordinates of the pixels 608 and 609 are interpolated by interpolation that rotates these linear interpolation coordinates. When the image is distorted into an ellipse, the coordinates of these pixels 608 and 609 are different. Therefore, the coordinate conversion unit 200 interpolates the coordinates of the pixel 605 from the coordinates of the pixels 608 and 609 by linear interpolation. Since an actual fisheye image is not necessarily in an ideal state, it is desirable to perform linear interpolation after rotation. However, as illustrated in FIG. In the case where the decrease in the value is allowed, the coordinate conversion unit 200 may be configured not to perform the linear interpolation after the rotation.

  FIG. 13 is a diagram illustrating an example of a fisheye image and a central projection image according to the first embodiment. A in the same figure is an example of the input image 500 containing a fish-eye image. In FIG. 9A, an area surrounded by a thick solid line indicates an output image plane.

  B in FIG. 13 is a diagram illustrating an example of a central projection image in a comparative example in which only linear interpolation is performed. As indicated by b in the figure, if only linear interpolation is performed without interpolation by rotation, distortion is not sufficiently interpolated, particularly in the circumferential direction, and distortion remains.

  FIG. 13C is a diagram illustrating an example of a central projection image in the first embodiment in which interpolation by rotation is performed in addition to linear interpolation. As indicated by c in the figure, when interpolation by rotation is performed, distortion is appropriately interpolated in the circumferential direction, and a natural central projection image is obtained.

  FIG. 14 is a flowchart illustrating an example of the operation of the imaging apparatus 100 according to the first embodiment. This operation is executed every time an image is captured.

  The imaging apparatus 100 performs predetermined signal processing on the input image including the fisheye image (step S901). Then, the imaging apparatus 100 performs image conversion processing that converts the fisheye image into a central projection image (step S910). The imaging apparatus 100 interpolates pixels in the central projection image (step S902) and performs predetermined signal processing (step S903). After step S903, the imaging apparatus 100 ends the operation.

  FIG. 15 is a flowchart illustrating an example of image conversion processing according to the first embodiment. The imaging apparatus 100 acquires the output coordinates (step S911) and normalizes the coordinates (step S912). Then, the imaging apparatus 100 performs a rotation matrix calculation on the normalized output coordinates (step S913), and performs perspective projection (center projection) conversion (step S914).

  The imaging apparatus 100 determines whether or not the readout coordinates corresponding to the output coordinates subjected to the perspective projection conversion are in the fisheye image distortion correction table 240 (step S915). When there is a read coordinate (step S915: Yes), the imaging apparatus 100 acquires a read coordinate corresponding to the output coordinate from the fisheye image distortion correction table 240 (step S916). On the other hand, when there is no readout coordinate (step S915: No), the imaging apparatus 100 performs a coordinate interpolation process for interpolating the interpolation coordinate (step S920).

  After step S920 or S916, the imaging apparatus 100 reads the pixel value of the pixel of the read coordinate in the fisheye image and outputs it as the pixel value of the corresponding output coordinate (step S917). Then, the imaging apparatus 100 determines whether or not conversion of all output coordinates has been completed (step S918). If the conversion has not ended (step S918: No), the imaging apparatus 100 returns to step S911. On the other hand, when the conversion ends (step S918: Yes), the imaging apparatus 100 ends the image conversion process.

  FIG. 16 is a flowchart illustrating an example of the coordinate interpolation process in the first embodiment. The imaging apparatus 100 acquires mesh intersection coordinates (that is, read coordinates) corresponding to the four mesh-corresponding coordinates around the output coordinates from the fisheye image distortion correction table 240 (step S921).

  The imaging apparatus 100 performs linear interpolation in the radial direction as necessary, and acquires two linear interpolation coordinates (step S922). In addition, the imaging apparatus 100 rotates the linear interpolation coordinates as necessary to obtain two rotation interpolation coordinates (step S923). Then, the imaging apparatus 100 performs linear interpolation as necessary, and acquires readout coordinates from the two rotational interpolation coordinates (step S924). After step S924, the imaging apparatus 100 ends the coordinate interpolation process.

  As described above, according to the first embodiment of the present technology, some specific coordinates and corresponding coordinates are stored in the storage unit in association with each other, and interpolation coordinates are interpolated based on the coordinates acquired from the storage unit. Therefore, the capacity and calculation amount of the storage unit can be reduced.

<2. Second Embodiment>
In the first embodiment, the fisheye image is converted into the central projection image by the perspective projection conversion. However, the cylindrical projection conversion may be performed instead of the perspective projection conversion. When cylindrical projection conversion is performed, it is possible to convert a fisheye image into a panoramic image having a wider field of view than in the case of perspective projection conversion. The imaging apparatus 100 according to the second embodiment is different from the first embodiment in that cylindrical projection conversion is performed instead of perspective projection conversion.

  FIG. 17 is a block diagram illustrating an example of the coordinate conversion unit 200 according to the second embodiment. The coordinate conversion unit 200 according to the second embodiment includes a coordinate normalization unit 210, an output coordinate rotation matrix calculation unit 220, and a coordinate normalization unit 211 and a panoramic coordinate conversion unit 250 instead of the perspective projection conversion unit 230. This is different from the first embodiment.

The coordinate normalization unit 211 normalizes the coordinates based on the viewing angle of the projection range in the fisheye image and the output image size. This viewing angle is set by a combination of dg_offset and dg_apl. dg_offset is a fixed angle, and a negative angle with respect to a reference axis (for example, the x axis) is set in the xy plane. dg_apl is a variable angle that can be changed by an application or the like, and a positive angle with respect to the reference axis is set in the xy plane. When performing Mercator projection conversion as cylindrical projection conversion, the coordinate normalization part 211 performs normalization, for example using the following formula | equation.

In Expression 23 and Expression 24, x _pano and y _pano are the normalized x coordinate and y coordinate. In Equation 25, rad () is a function that returns a value expressed in units of “degrees” and returns a value expressed in “radians”. Further, tan ⁻¹ () is an inverse function of the tangent function. ln A is the natural logarithm of A.

  According to Expression 24 and Expression 25, the range excluding the vicinity of the zenith where the angle with respect to the z axis exceeds a certain allowable angle (for example, 85 degrees) is panorama developed. This is because when the panorama is developed near the zenith, the closer to the zenith, the larger the enlargement ratio and the lower the visibility.

The coordinate normalization unit 211 supplies the normalized output coordinates (x _pano , y _pano ) to the panorama coordinate conversion unit 250.

上式においてｇｄは、グーデルマン関数である。 The panorama coordinate conversion unit 250 converts output coordinates (x _pano , y _pano ) into polar coordinates. The panorama coordinate conversion unit 250 obtains polar coordinates using, for example, the following expression.
R _x = x _pano ... Formula 26

In the above equation, gd is a Gudelman function.

The output coordinates are normalized and converted into a cylindrical projection by Expressions 23 to 27. The panorama coordinate conversion unit 250 supplies the converted polar coordinates (R _x , R _y ) to the read coordinate output unit 300.

  The coordinate normalization unit 211 and the panorama coordinate conversion unit 250 are an example of a cylindrical projection conversion unit in the claims.

The panorama coordinate conversion unit 250 calculates polar coordinates, but for each orthogonal coordinate (x _pano , y _pano ), further provides a polar coordinate table that holds the polar coordinates calculated in advance, and _obtains polar coordinates from the table. It may be configured to read. Further, the normalization unit 211 and the panorama coordinate conversion unit 250 perform Mercator projection conversion, but may perform cylindrical projection conversion other than Mercator projection conversion such as equirectangular projection conversion. When performing a conversion other than the Mercator projection conversion, the panorama coordinate conversion unit 250 may perform the conversion using polar coordinates stored in advance in the polar coordinate table.

  FIG. 18 is a diagram illustrating an example of a projection range and a projection surface according to the second embodiment. A projection range 512 shown in the figure is a range to be subjected to cylindrical projection conversion in the fisheye image. A fan-shaped range having an angle obtained by adding dg_apl to dg_offset is set as the projection range 512. The projection plane 513 is a plane on which the projection range 512 is cylindrically projected. In the figure, one side of a rectangular plane is curved along the outer periphery of the fisheye image. An image projected on the projection plane 513 on the plane is generated as a central projection image.

  FIG. 19 is a flowchart illustrating an example of image conversion processing according to the second embodiment. The image conversion process of the second embodiment is different from the first embodiment in that step S919 is executed instead of steps S913 and S914.

  The imaging apparatus 100 normalizes the coordinates based on the viewing angle and the output image size (step S912), and converts the normalized output coordinates to polar coordinates (step S919).

  Thus, according to the second embodiment, the cylindrical projection conversion is performed on the output coordinates, and the readout coordinates and the interpolation coordinates corresponding to the coordinates subjected to the cylindrical projection conversion are obtained. It can be converted into a panoramic image of the projection method.

<3. Third Embodiment>
In the first embodiment, the number of mesh intersections of a plurality of closed curves (for example, circles) is the same. However, the smaller the radius of the closed curve, the shorter the outer circumference. For this reason, if the number of divisions is the same, the density of mesh intersections increases as the distance from the center is approached. The higher the mesh intersection density, the higher the correction accuracy. Therefore, from the viewpoint of equalizing the correction accuracy, it is desirable to reduce the number of mesh intersections as the radius is smaller and to make the mesh intersection density closer to a constant value. The imaging apparatus 100 of the third embodiment is different from the first embodiment in that the number of mesh intersections is reduced as the radius is smaller (in other words, the area is smaller).

  FIG. 20 is a diagram illustrating an arrangement example 521 of mesh intersections according to the third embodiment. In the figure, a is an arrangement example when the mesh intersection point is doubled in the closed curve having the largest radius and the closed curve having the next largest radius. A black circle surrounded by a dotted line in a in the figure indicates an added mesh intersection. B in the same figure is the example 522 of arrangement | positioning when a mesh intersection is reduced in half in the closed curve with the smallest radius. The dotted white circles in b in FIG. 6 indicate the reduced mesh intersections. Thus, increasing the outer mesh intersection or reducing the inner mesh intersection makes the density of the mesh intersections nearly constant.

  FIG. 21 is a diagram illustrating a mesh intersection arrangement example 523 when the densities in the third embodiment are equalized. In the figure, mesh intersections are added to the closed curve having the largest radius and the closed curve having the next largest radius, and the mesh intersections are reduced to the closed curve having the smallest radius.

  The coordinate interpolation unit 330 according to the third embodiment obtains the coordinates of the reduced mesh intersection from the surrounding mesh intersection coordinates. For example, interpolation for rotating two adjacent mesh intersections in the circumferential direction, linear interpolation from two adjacent mesh intersections in the radial direction, or the like is performed.

  If a mesh intersection is added to the outer closed curve and not added to the inner closed curve, five mesh intersections including four mesh intersections and the added mesh intersection are located around the coordinates between the closed curves. Will be placed.

  Here, the dotted frame 524 is a polygon having apexes at four mesh intersections that do not include the added mesh intersection. On the other hand, the solid line frame 525 is a polygon whose interpolation coordinates are interpolated between the added mesh intersection and the center, and the mesh intersections adjacent in the radial direction, the added mesh intersection, and the interpolation coordinates are vertices. is there. In this case, the coordinates inside the frame 524 are interpolated from the mesh intersections of those four vertices. Alternatively, interpolation coordinates may be interpolated at the vertices of the frame 525, and the internal coordinates may be interpolated from the mesh intersections and vertices. Alternatively, the imaging apparatus 100 may mix coordinates interpolated from the vertex of the frame 524 and coordinates interpolated from the vertex of the frame 525, and interpolate the mixed coordinates as final interpolation coordinates. Considering the continuity of distortion correction, a method of gradually mixing as the center or the outer circumference is approached is desirable. When gradually mixing, for example, the closer to the center, the higher the mixing ratio of coordinates interpolated from the vertex of the frame 524 is set.

  As described above, according to the third embodiment of the present technology, the smaller the closed curve area, the smaller the number of divisions, the division points obtained by dividing the closed curve are set as mesh intersections, so that the density of mesh intersections is evenly approximated. be able to.

<4. Fourth Embodiment>
In the first embodiment, the imaging apparatus 100 corrects the distortion of the fisheye image. However, when the central projection image is input, the imaging apparatus 100 can also correct the distortion of the central projection image. The imaging apparatus 100 according to the fourth embodiment is different from the first embodiment in that it determines which of the fisheye image and the central projection image is input and changes the distortion correction method.

  FIG. 22 is a block diagram illustrating a configuration example of an imaging system according to the fourth embodiment. This imaging system includes an interchangeable lens unit 400 and an imaging device 100.

  The interchangeable lens unit 400 is a component to which a fisheye lens or a central projection lens is attached. When the interchangeable lens unit 400 is attached, the interchangeable lens unit 400 notifies the imaging apparatus 100 of the type of the lens.

  The configuration of the imaging device 100 of the fourth embodiment is the same as that of the first embodiment except that the fisheye lens 110 is not provided. The control unit 150 according to the fourth embodiment acquires the type of lens and determines whether or not the attached lens is a fisheye lens. Then, control unit 150 generates and transmits a switching signal based on the determination result. The switching signal is a signal for instructing switching of the conversion method. The control unit 150 is an example of a determination unit described in the claims.

  FIG. 23 is a block diagram illustrating an example of the coordinate conversion unit 200 according to the fourth embodiment. The coordinate conversion unit 200 according to the fourth embodiment is different from the first embodiment in that it further includes a central projection image distortion correction table 260. The center projected image distortion correction table 260 is a table in which the output coordinates of the center projected image after distortion correction are associated with some of the readout coordinates of the center projected image before distortion correction.

  FIG. 24 is a block diagram illustrating a configuration example of the read coordinate output unit 300 according to the fourth embodiment. The read coordinate output unit 300 of the fourth embodiment differs from the first embodiment in that it further includes a switch 310.

  The switch 310 supplies read coordinates in the fisheye image distortion correction table 240 or the central projection image distortion correction table 260 to the orthogonal coordinate acquisition unit 320 according to the switching signal. For example, when the lens type is a fisheye lens, read coordinates from the fisheye image distortion correction table 240 are supplied. Otherwise, read coordinates from the central projection image distortion correction table 260 are supplied.

  Also, the pre-rotation interpolation coefficient calculation unit 334 of the fourth embodiment changes the interpolation coefficient in accordance with the switching signal. For example, when the lens is a fish-eye lens, the interpolation coefficient calculated by Expression 11 is supplied, and when not, an interpolation coefficient of “0” is supplied.

  In addition, the rotation angle calculation unit 335 of the fourth embodiment changes the rotation angle according to the switching signal. For example, when the lens is a fisheye lens, the sine and cosine of the rotation angle calculated by Expression 16 or 17 is supplied. On the other hand, when the lens is not a fish-eye lens, an angle of “0” is set, and “1” is supplied as the sine and “0” is supplied as the cosine.

  The readout coordinate output unit 300 performs distortion correction on the central projection image. However, when the central projection image is input, the readout coordinates may be output as an output image without performing distortion correction. . In this case, the central projection image distortion correction table 260 is not provided, and the image conversion unit 160 does not substantially perform image conversion.

  As described above, according to the fourth embodiment, the imaging apparatus 100 determines whether or not the input image is a fisheye image, and if it is a fisheye image, converts it into a central projection image. Image processing can also be performed on input images other than eye images.

  The above-described embodiment shows an example for embodying the present technology, and the matters in the embodiment and the invention-specific matters in the claims have a corresponding relationship. Similarly, the invention specific matter in the claims and the matter in the embodiment of the present technology having the same name as this have a corresponding relationship. However, the present technology is not limited to the embodiment, and can be embodied by making various modifications to the embodiment without departing from the gist thereof.

  Further, the processing procedure described in the above embodiment may be regarded as a method having a series of these procedures, and a program for causing a computer to execute these series of procedures or a recording medium storing the program. You may catch it. As this recording medium, for example, a CD (Compact Disc), an MD (MiniDisc), a DVD (Digital Versatile Disc), a memory card, a Blu-ray disc (Blu-ray (registered trademark) Disc), or the like can be used.

  Note that the effects described here are not necessarily limited, and may be any of the effects described in the present disclosure.

In addition, this technique can also take the following structures.
(1) a storage unit that stores some specific coordinates among the coordinates in the non-centered projection image generated by the projection not corresponding to the central projection and the corresponding coordinates in the central projection image associated with the specific coordinates;
When one of the specific coordinates and the corresponding coordinates is input, a coordinate acquisition unit that acquires the other coordinates corresponding to the input coordinates from the storage unit;
An interpolation unit that interpolates, as interpolation coordinates, coordinates different from the other coordinates based on the obtained other coordinates;
An image processing apparatus comprising: an image conversion unit that converts one of the non-center projected image and the center projected image into the other image based on the other coordinate and the interpolated coordinate.
(2) The coordinate acquisition unit acquires the specific coordinate as the other coordinate,
The image processing apparatus according to (1), wherein the interpolation unit interpolates, as the interpolation coordinates, coordinates obtained by rotating the specific coordinates around a predetermined coordinate in the non-center projected image.
(3) The coordinate acquisition unit acquires the specific coordinate as the other coordinate,
The interpolation unit interpolates coordinates on a line segment connecting predetermined coordinates and the specific coordinates in the non-centered projection image as the interpolation coordinates, and coordinates obtained by rotating the interpolation coordinates around the predetermined coordinates. The image processing apparatus according to (1) or (2), wherein interpolation is performed as new interpolation coordinates.
(4) The coordinate acquisition unit acquires the specific coordinate as the other coordinate,
The image processing apparatus according to (1), wherein the interpolation unit interpolates coordinates on a line segment connecting two specific coordinates as the interpolation coordinates.
(5) The image processing device according to any one of (1) to (4), wherein the specific coordinate is a coordinate of a dividing point obtained by dividing each of the plurality of closed curves having different areas in the non-center projected image.
(6) The image processing apparatus according to (5), wherein the specific coordinate is a coordinate of a division point obtained by dividing each of the closed curves by a smaller number of divisions as the area of the closed curve is smaller.
(7) further comprising a perspective projection conversion unit that performs perspective projection conversion on the coordinates in the central projection image and inputs the coordinates obtained by the perspective projection conversion to the coordinate acquisition unit and the interpolation unit;
The image processing apparatus according to any one of (1) to (6), wherein the interpolation unit interpolates the interpolation coordinates corresponding to the input coordinates.
(8) It further comprises a rotating unit that rotates the coordinates in the central projection image around a predetermined axis,
The image processing apparatus according to any one of (1) to (7), wherein the perspective projection conversion unit performs the perspective projection conversion on the rotated coordinates.
(9) It further includes a cylindrical projection conversion unit that performs cylindrical projection conversion on the coordinates in the central projection image and inputs the coordinates obtained by the cylindrical projection conversion to the coordinate acquisition unit and the interpolation unit,
The image processing apparatus according to (1), wherein the interpolation unit interpolates the interpolation coordinates corresponding to the input coordinates.
(10) A determination unit that determines whether or not the captured image is the one image and inputs the one coordinate to the coordinate acquisition unit and the interpolation unit when the image is the one image; Equipped,
The image processing apparatus according to any one of (1) to (9), wherein the interpolation unit interpolates the interpolation coordinates corresponding to the input coordinates.
(11) an imaging unit that captures one of a non-centered projection image and a center projected image generated by a projection that does not correspond to the center projection;
A storage unit that stores some specific coordinates among the coordinates in the non-center projected image and the corresponding coordinates in the central projected image associated with the specific coordinates;
When one of the specific coordinates and the corresponding coordinates is input, a coordinate acquisition unit that acquires the other coordinates corresponding to the input coordinates from the storage unit;
An interpolation unit that interpolates, as interpolation coordinates, coordinates different from the other coordinates based on the obtained other coordinates;
An image pickup apparatus comprising: an image conversion unit that converts one of the non-center projected image and the center projected image into the other image based on the other coordinate and the interpolated coordinate.
(12) When one of the specific coordinates and the corresponding coordinates is input, the coordinate acquisition unit and a part of the specific coordinates among the coordinates in the non-centered projection image generated by the projection that does not correspond to the central projection A coordinate acquisition procedure for acquiring the other coordinate corresponding to the input coordinate from the storage unit storing the corresponding coordinate in the central projection image associated with the specific coordinate;
An interpolation procedure in which an interpolation unit interpolates coordinates different from the other coordinates based on the acquired other coordinates as interpolation coordinates;
An image processing method comprising: a conversion procedure in which an image conversion unit converts one of the non-center projected image and the center projected image into the other image based on the other coordinate and the interpolated coordinate.
(13) When one of the specific coordinates and the corresponding coordinates is input, the coordinate acquisition unit receives a part of the coordinates in the non-centered projection image generated by the projection that does not correspond to the central projection and the specific coordinates A coordinate acquisition procedure for acquiring the other coordinate corresponding to the input coordinate from the storage unit storing the corresponding coordinate in the central projection image associated with the specific coordinate;
An interpolation procedure in which an interpolation unit interpolates coordinates different from the other coordinates based on the acquired other coordinates as interpolation coordinates;
A program for causing a computer to execute a conversion procedure in which an image conversion unit converts one of the non-center projected image and the center projected image into the other image based on the other coordinate and the interpolated coordinate. .

DESCRIPTION OF SYMBOLS 100 Imaging device 110 Fisheye lens 120 Image sensor 130 Input signal processing part 140 Frame memory 150 Control part 160 Image conversion part 170 Pixel interpolation part 180 Output signal processing part 190 Recording part 200 Coordinate conversion part 210, 211 Coordinate normalization part 220 Output coordinate rotation Matrix calculation unit 230 perspective projection conversion unit 240 fisheye image distortion correction table 250 panorama coordinate conversion unit 260 central projection image distortion correction table 300 readout coordinate output unit 310 switch 320 orthogonal coordinate acquisition unit 330 coordinate interpolation unit 331 post-rotation linear interpolation unit 332 Post-rotation interpolation coefficient calculation unit 333 Pre-rotation linear interpolation unit 334 Pre-rotation interpolation coefficient calculation unit 335 Rotation angle calculation unit 340 Read coordinate rotation matrix calculation unit 341, 342, 343, 344 Multiplier 345 Subtractor 346 Adder 400 Interchangeable lens uni The

Claims (12)

A storage unit for storing the corresponding coordinate in the central projection image associated with some specific coordinates and the specific coordinates of the coordinates in the non-central projection image generated by the projection does not correspond to centered projection,
A coordinate acquisition unit that acquires the specific coordinates corresponding to the corresponding coordinates from the storage unit when the corresponding coordinates are input ;
An interpolation unit that interpolates, as rotational interpolation coordinates, coordinates obtained by rotating a coordinate on a line segment connecting a predetermined center coordinate and the acquired specific coordinate in the non-center projected image with the center coordinate as a center ;
An image processing apparatus comprising: an image conversion unit that converts the non-center projected image into the center projected image based on the specific coordinates and the rotation interpolation coordinates. Before SL interpolation unit, the image processing apparatus according to claim 1, wherein the coordinate obtained by rotating a specific coordinate about said center coordinates in the non-central projection image to interpolate as the rotation interpolation coordinates. Before SL interpolation unit, said interpolation to the linear interpolation coordinates the coordinates on the line connecting the said specific coordinates and the center coordinates in the non-central projection image as linear interpolation coordinate is rotated about said center coordinates The image processing apparatus according to claim 1, wherein coordinates are interpolated as the rotation interpolation coordinates. A storage unit that stores some specific coordinates among the coordinates in the non-central projection image generated by the projection not corresponding to the central projection and the corresponding coordinates in the central projection image associated with the specific coordinates;
When one of the specific coordinates and the corresponding coordinates is input, a coordinate acquisition unit that acquires the other coordinates corresponding to the input coordinates from the storage unit;
An interpolation unit that interpolates, as interpolation coordinates, coordinates different from the other coordinates based on the obtained other coordinates;
An image conversion unit that converts one of the non-center projected image and the center projected image into the other image based on the other coordinate and the interpolated coordinate;
Comprising
The specific coordinates, the non-central Ru coordinates der division points obtained by dividing each of the plurality of closed curves having different areas in the projection image <br/> image processing apparatus. The image processing apparatus according to claim 4 , wherein the specific coordinates are coordinates of a dividing point obtained by dividing each of the closed curves by a smaller number of divisions as the area of the closed curve is smaller. Further comprising a perspective projection conversion unit that performs perspective projection conversion on the coordinates in the central projection image and inputs the coordinates obtained by the perspective projection conversion to the coordinate acquisition unit and the interpolation unit;
The image processing apparatus according to claim 4 , wherein the interpolation unit interpolates the interpolation coordinates corresponding to the input coordinates. A rotation unit that rotates the coordinates in the central projection image around a predetermined axis;
The image processing apparatus according to claim 6 , wherein the perspective projection conversion unit performs the perspective projection conversion on the rotated coordinates. A cylindrical projection conversion unit that performs cylindrical projection conversion on the coordinates in the central projection image and inputs the coordinates obtained by the cylindrical projection conversion to the coordinate acquisition unit and the interpolation unit;
The image processing apparatus according to claim 4 , wherein the interpolation unit interpolates the interpolation coordinates corresponding to the input coordinates. If the captured image is the one image, and if the image is the one image, further comprises a determination unit that inputs the one coordinate to the coordinate acquisition unit and the interpolation unit,
The image processing apparatus according to claim 4 , wherein the interpolation unit interpolates the interpolation coordinates corresponding to the input coordinates. An imaging unit that captures one of a non-central projection image and a central projection image generated by a projection that does not correspond to the central projection;
A storage unit that stores some specific coordinates among the coordinates in the non-center projected image and the corresponding coordinates in the central projected image associated with the specific coordinates;
When one of the specific coordinates and the corresponding coordinates is input, a coordinate acquisition unit that acquires the other coordinates corresponding to the input coordinates from the storage unit;
An interpolation unit that interpolates, as interpolation coordinates, coordinates different from the other coordinates based on the acquired other coordinates;
An image conversion unit that converts one of the non-center projected image and the center projected image into the other image based on the other coordinate and the interpolation coordinate ;
The specific device is a coordinate of a dividing point obtained by dividing each of a plurality of closed curves having different areas in the non-center projected image . One of the coordinates in the non-central projection image generated by the projection not corresponding to the central projection and the corresponding coordinates in the central projection image associated with the specific coordinate is obtained by the coordinate acquisition unit. When input, a coordinate acquisition procedure for acquiring the other coordinates corresponding to the input coordinates from the storage unit storing the specific coordinates and the corresponding coordinates ;
An interpolation procedure in which an interpolation unit interpolates coordinates different from the other coordinates based on the acquired other coordinates as interpolation coordinates;
An image conversion unit comprising: a conversion procedure for converting one of the non-center projected image and the center projected image into the other image based on the other coordinate and the interpolated coordinate ;
In the image processing method , the specific coordinate is a coordinate of a dividing point obtained by dividing each of a plurality of closed curves having different areas in the non-center projected image . One of the coordinates in the non-central projection image generated by the projection not corresponding to the central projection and the corresponding coordinates in the central projection image associated with the specific coordinate is obtained by the coordinate acquisition unit. When input, a coordinate acquisition procedure for acquiring the other coordinates corresponding to the input coordinates from the storage unit storing the specific coordinates and the corresponding coordinates ;
An interpolation procedure interpolation unit interpolates the coordinates different from the other coordinates based on the obtained other coordinate as the interpolation coordinates,
Image conversion unit, a program for executing a conversion procedure for converting one of the images of the non-central projection image and the central projection image based on the other coordinate and said interpolation coordinates to other image to the computer Because
The specific coordinates are coordinates of a dividing point obtained by dividing each of a plurality of closed curves having different areas in the non-centered projection image.
program.

Images