KR100755450B1 - 3d reconstruction apparatus and method using the planar homography - Google Patents

Abstract

A 3D reconstruction apparatus using planar homography and a method therefor are provided to simplify camera calibration in comparison to a Z. Zang method by using the homography for 3 planar patterns photographed in a single image and minimize an error by configuring a disparity map on the basis of the variance of a correspondence point candidate and removing the disparity of a threshold value or less. An image loading device(101) receives uncalibrated left and right images photographed to include 3 planar patterns. A feature point extracting device(103) extracts feature points of the planar patterns from the received left and right images on the basis of a Harris feature point search method. A homography calculating device(107) calculates homography among the planar patterns by using the extracted feature points. A camera calibrating device(109) calculates and calibrates intrinsic and extrinsic parameters of a camera on the basis of the calculated homography and a camera model. A fundamental matrix calculating unit(111) calculates a fundamental matrix through epipolar geometry on the basis of matching information among the feature points of the planar patterns. An image calibrating device(113) horizontally calibrates epipolar lines of the left and right images and calibrates the images. A disparity map configuring device(115) searches all matched correspondence points in the calibrated two images, and configures a disparity map. A 3D coordinate calculating device(117) calculates a 3D coordinate by a back projection method on the basis of the configured disparity map. A polyhedron configuring device(119) configures the calculated 3D coordinate as a polyhedron according to a Delauny triangulation method. A texture mapping device(121) maps a texture to the configured polyhedron.

Description

3D RECONSTRUCTION APPARATUS AND METHOD USING THE PLANAR HOMOGRAPHY}

1 is a block diagram of a three-dimensional reconstruction device of the present invention,

2 is an exemplary diagram of a coordinate system of a planar pattern of the present invention;

3 is an exemplary view showing a concept for epipolar,

4A and 4B show an input image for an experimental example of camera calibration;

4c is a result table according to an experimental example of a camera calibration;

5a to 5c are exemplary views showing an example of the configuration of the image correction and the parallax map according to the present invention,

6a to 6c is an exemplary view showing a three-dimensional reconstruction experiment example in the present invention.

** Description of symbols for the main parts of the drawing **

101: image loading means 103: feature point extraction means

105: XY alignment means 107: homography calculation means

109: camera calibration means 111: basic matrix calculation means

113: image correction means 115: parallax map construction means

117: three-dimensional coordinate calculation means 119: polyhedron construction means

121: texture mapping means

The present invention belongs to a three-dimensional reconstruction technique using planar homography, and more particularly, an apparatus and method for easily and simply implementing three-dimensional reconstruction by calculating homography between three pattern images existing in a single image. will be.

When producing digital image contents such as computer graphics (CG), animation, virtual reality, and games, reconstructing and representing three-dimensional images facilitates the implementation of a three-dimensional virtual world. Therefore, 3D image reconstruction technology is widely applied in entertainment related fields such as augmented reality [1] [2] [3].

The technique for 3D image representation is a rendering technique mainly used in CG, which requires relatively long time to model and render an object because it needs to calculate object surface information.

Unlike this general rendering technique, there is Image Based Rendering which restores a 3D object based on a plurality of images (2D) captured by a calibrated camera [3]. This technique is effective for modeling and rendering a 3D real world, such as a building, and has the advantage of being able to use a variety of primitive models. If the target image is a natural scene, it is not easy to express.

On the other hand, there is a technique for reconstructing a three-dimensional image using a continuous image taken by the camera. As disclosed in the following documents [4], [5], and [6] below, camera calibration should be preceded, which has a problem that the calculation necessary for camera calibration thereof is difficult, and thus the reconstruction itself is complicated.

According to the recently introduced method, three-dimensional reconstruction is possible from a single image, and a camera calibration is performed based on the vanishing point in the target image. However, the application of these methods is limited to buildings that can calculate or estimate vanishing points [7] [8] [9].

Prior literature information

[1] RT Azuma, "A Survey of Augmented Reality", Presence: Teleoperators and Virtual Environments , vol. 6, no. 4, pp. 355-385, 1997.

[2] Y. Ohta and H. Tamura, "Mixed Reality-Merging Real and Virtual Worlds", Ohmsha Ltd. & Springer-Verlag , 1999.

[3] R PE Debevec, CJ Taylor, and J. Malik, "Modeling and Rendering Architecture from Photographs: A Hybrid Geometry and Image-Based Approach", In Proceedings of ACM SIGGRAPH 1996, ACM Press / ACM SIGGRAPH , pp. 11- 21, Aug. 1996.

[4] P. Beardsley, P. Torr and A. Zisserman, "3D Model Acquisition from Extended Image Sequences", In Proceeding 4th European Coference on Computer Vision, Cambridge, LNCS 1065, Volume II , page 683-695, Springer-Verlag , 1996.

[5] M. Pollefeys, R. Koch and L. Van Gool, "Self-Calibration and Metric Reconstruction in spite of Varying and Unknown Internal Camera Parameters", International Journal of Computer Vision , 32 (1), 7-25, 1999 .

[6] C. Tomasi and T. Kanade, "Shape and motion from image streams under orthography: a factorization method", International Journal of Computer Vision , 9 (2), 137-154, 1990.

[7] D. Liebowitz, A. Criminisi, and A. Zisserman, "Creating Architectural Models from Images", pp. 39-50, EUROGRAPHICS , 1999.

[8] Beardsley, PA, Torr, PHS and Zisserman, AP, "3D model acquisition from extended image sequence" OUEL Report 2089/96, Department of Engineering Science, University of Oxford . 1996.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-described problems, and provides a technical problem to provide an apparatus and method that can be easily and simply implemented in three-dimensional reconstruction compared to the above-mentioned conventional technology.

The technical gist of the present invention is to take three plane pattern images from a non-correction image as one sheet, calculate homography between these plane pattern images, and perform camera calibration.

In general, camera calibration in non-corrected images introduces additional problems due to the difficulty of finding a corresponding point between the two images. In particular, accuracy or errors should be minimized at the matched points so that correct results can be expected.

As a part of minimizing errors in camera calibration, Z. Zhang's prior literature proposes a series of methods for correcting three consecutive planar pattern images [Z. Zhang, "A Flexible New Techinque for Camera Calibration", IEEE Transaction on Pattern Analysis and Machine Intelligence , vol. 22, no. 11, pp. 1-20, 1998].

In the present invention, based on the correction method proposed by Z. Zhang, it is simplified by using a single image photographed to include three planar patterns instead of three consecutive planar pattern images. In this case, the user can easily set the coordinate system in the planar pattern image, and since the error elimination of the three-dimensional coordinates is used as a constraint, the error can be significantly reduced as compared with the conventional art.

Specific features and advantages of the above technical problem will become more apparent from the following detailed description based on the accompanying drawings.

1 is a view showing the configuration of the three-dimensional reconstruction device 100 of the present invention, the image loading means 101, feature point extraction means 103, XY alignment means 105, homography calculation means 107 , Camera correction means 109, basic matrix calculation means 111, image correction means 113, parallax map construction means 115, three-dimensional coordinate calculation means 117, polyhedron construction means 119 and texture mapping means And 121, which are understood to be implemented as some function of computer-based three-dimensional graphics software or tools.

In detail, the image loading means 101 receives an uncalibrated image photographed by the camera (two sets for the stereo image) including three plane patterns. At this time, the planar pattern is a pattern of repeated check patterns as shown in FIG. 1. For the pattern of check patterns, see Z. Zhang's preceding literature. Zhang, "A Flexible New Techinque for Camera Calibration", IEEE Transaction on Pattern Analysis and Machine Intelligence , vol. 22, no.11, pp.1-20, 1998. .

와, 이에 대응되는 2차원 카메라 평면 영상 좌표계의 한 점

가 주어졌을 때, 아래의 카메라 사영 행렬에 의해 모든 점 i에 대하여

를 만족한다.If we look at the general camera model, a point in three-dimensional space with a general pin-hole camera

And a point of the corresponding 2D camera plane image coordinate system.

Is given, for all points i by the camera projection matrix below

Satisfies.

....................... [수학식 1]

....................... Equation 1

In the above equation, the projection matrix has 11 degrees of freedom. Equation 2 below expresses the relationship between the real world coordinate system and the camera origin.

...................................... [수학식 2]

............ Equation (2)

이며,

,

는 카메라 영상의 x, y축에 대한 초점거리이고,

는 x, y 두 축의 기울기(또는 비틀림 정도)이며,

,

는 카메라 중심과 카메라 투영 평면이 직교하는 주점(principle point)을 나타낸다. 그리고

과

는 외부 변수로서 카메라의 회전과 이동을 의미한다.Internal variables

Is,

,

Is the focal length for the x and y axes of the camera image,

Is the slope (or twist) of the two axes x, y,

,

Denotes a principal point at which the camera center and the camera projection plane are orthogonal to each other. And

and

Is an external variable that means the rotation and movement of the camera.

The feature point extraction unit 103 extracts the feature points of the planar pattern based on the Harris feature point search method. Harris feature point retrieval method is described in the following equation [C. Harris and M. Stephens, "A combined corner and edge detector", 4th Alvey Vision Conference, pp. 147-151, 1988]. 3].

...................... [수학식 3]

.......... [Equation 3]

이며, 밝기 영상

에서

와

는

와

방향으로의 미분을 나타낸다.here,

Brightness image

in

Wow

Is

Wow

The derivative in the direction is shown.

,

방향으로 정렬되어야 하는데, 이는 XY 정렬수단(105)에 의해 수행된다.The feature points extracted in this way are

,

Direction, which is performed by the XY alignment means 105.

,

,

, 이동 벡터를

라 하면, 아래의 관계가 성립한다.The homography calculation unit 107 calculates homography between the three planar patterns included in the image photographed by each camera using the aligned feature points. The homography represents a projective transformation relationship between two planes. Assuming that the horizontal and vertical planes of the planar pattern are the x, y axis, the planar pattern and the normal direction in the three-dimensional coordinate system as shown in FIG. The z coordinate in the pattern is zero. Each column of the rotation matrix

,

,

Move vector

Then, the following relationship holds.

..................... [수학식 4]

........... [ Equation 4 ]

과 설정된 3차원 좌표

사이에는

의 호모그래피 관계가 성립한다.Thus coordinates in the image

And set three-dimensional coordinates

In between

The homography relationship is established.

..................................... [수학식 5]

........... [ Equation 5 ]

이다.here,

to be.

, [수학식 1], [수학식 2]로부터 회전행렬의

,

,

의 성질을 이용하면, 내부 변수는 아래의 [수학식 6]과 같다.The camera calibration means 109 calculates and corrects the internal and external variables of the camera required for three-dimensional reconstruction, which are implemented based on the homography and camera model calculated above. According to a characteristic aspect of the invention, a camera matrix

, From [Equation 1], [Equation 2]

,

,

Using the property of, the internal variable is shown in [Equation 6] below.

..................................... [수학식 6]

[ Equation 6 ]

,

이며, 위 [수학식 6]은 다음과 같이 정리될 수 있다. here,

,

[Equation 6] can be summarized as follows.

..................... [수학식 7]

..... [ Equation 7 ]

The left matrix of Equation 7 is expressed by Equation 8 below.

.......... [수학식 8]

.......... [ Equation 8 ]

From the above [Equation 7] and [Equation 8], the following relation holds.

. ..... [수학식 9]

. ..... [ Equation 9 ]

의 선형 방정식은 다음과 같다.In Equation 9 above

The linear equation of is

.................. [수학식 10]

.................. [Equation 10]

.................. [수학식 11]

[ Equation 11 ]

은 5 자유도의 코닉(conic)의 계수이다. 코닉은 일반적인 위치에서 다섯 점들로 결정된다. 위 선형 방정식에서

을

라 정의하면,

가 된다. 따라서

로 표현될 수 있다. 여기서

은 대칭이므로

로 표현된다.procession

Is the coefficient of conic of 5 degrees of freedom. Conic is determined by five points in the general position. In the linear equation above

of

If you define

Becomes therefore

It can be expressed as. here

Is symmetric

It is expressed as

의 관계에서 [수학식 2]에서 정의한

를 대입하면 다음의 [수학식 13]이 유도된다.remind

Defined in Equation 2 in relation to

Substituting Equation leads to Equation 13 below.

................. [수학식 13]

[ Equation 13 ]

The intrinsic parameter of the camera is calculated by Equation 13 as follows.

,

,

,

,

,

,

,

,

Therefore, the above-mentioned equations of the rotation matrix and the movement matrix are derived based on Equation 6, each of which is described below.

............................... [수학식 14]

......................... [ Equation 14 ]

................................ [수학식 15]

......................... [Equation 15]

는 상기

의 네 번째 열벡터이다.However, in [Equation 15]

Above

Is the fourth column vector of.

On the other hand, the basic matrix calculation means 111 calculates a fundamental matrix through the epipolar geometry based on the matching information between the feature points of the planar pattern.

As is well known, epipolar geometry is illustrated in the accompanying FIG. 3, the relationship between images obtained from two cameras may be defined as a point-to-line correspondence, not a point-to-point correspondence. That is the point.

을 서로 다른 위치의 카메라(

,

)에 의한 이미지 평면(

,

)에 투영하면, 각각의 영상에 점

과

로 맺히며, 이러한 두 점을 대응점이라고 한다. 공간상의 점

과 두 카메라가 이루는 평면은 에피폴라 평면이며, 에피폴라 평면과 영상 평면이 교차하는 선을 에피폴라 선(

,

)이라고 하고, 두 카 메라를 연결하는 선분과 각 영상 평면의 교점을 에피폴(

,

)이라고 한다. 여기서 공간상의 점 M은 투영된 점

,

는 에피폴라 선상에 존재하며, 이를 에피폴라 구속 조건이라고 한다. 이에 대한 대수적 표현이 앞서 언급한 기본행렬(F)이며, 아래의 [수학식 16]과 같은 조건을 만족한다.Random point in space

At different locations

,

Image plane by

,

), A dot on each image

and

These two points are called correspondence points. Point in space

The plane formed by the two cameras is the epipolar plane, and the epipolar line (the line where the epipolar plane and the image plane intersect)

,

) And the intersection of the two cameras and the intersection of each image plane is called epipole (

,

It is called). Where point M in space is the projected point

,

Is on the epipolar line and is called epipolar constraint. Thus for the default matrix (F) an algebraic expression is mentioned above, and satisfies the condition shown in [Equation 16] below.

................................. [수학식 16]

[ Equation 16 ]

Next, the image correcting means 113 corrects the image by making the epipolar lines of the two images parallel to each other. Conventional image correction is disclosed in the following precedent documents [O. Faugeras, "Three-Dimensional Computer Vision: a Geometric Viewpoint", MIT press , 1993], [N. Ayache and C. Hansen, "Rectification of images for binocular and trinocular stereovision", Proc. International Conference on Pattern Recognition , pp. 11-16, 1988, [D. Papadimitriou and T. Dennis, "Epipolar line estimation and rectification for stereo image pairs", IEEE Trans. Image Processing , 5 (4): 672-676, 1996].

The gist of the above-mentioned prior documents can be said to transform the three-dimensional space of two images into one plane image. However, the present invention uses a transformation method using epipole, coordinate distance and angle in the x, y reference coordinate system of the image. This deformation method has the advantage that the coordinate system is intuitively transformed without unnecessary pixels. As it is introduced in detail in Pollefeys' prior literature, "A simple and efficient rectification method for general motion", Computer Vision , 1999, the present invention refers only to the effects accordingly.

Next, the disparity map constructing unit 115 constructs a compact disparity map by finding all matched corresponding points in the two corrected images. According to the present invention, a window having a predetermined size is set for pixels of an image in a model configuration by minimizing an error rate. In the image, the variance in the window is calculated, and pixels of the image larger than the variance threshold are extracted as the first corresponding candidate points, and then constitute a disparity map.

More specifically, after searching for pixels having a variance (see Equation 17) that is greater than or equal to a threshold in image 1 (left image), pixels corresponding to the searched pixels are searched in image 2 (right image). To match.

................ [수학식 17]

[ Equation 17 ]

이다.here,

to be.

The matching method described above is as follows.

을 중심점으로 상관 윈도우 크기를

로 정의한다. 따라서 영상2에서

에 대응되는 점은 영상2의 에피폴라 선을 검색해서 영상1에서

에 대응되는

를 찾는다. 이를 위한 상관함수는 [수학식 18]로 표현된다.Point in Image 1

The correlation window size

Defined as So in image 2

The point corresponding to is to search the epipolar line of image 2

Corresponding to

Find it. The correlation function for this is expressed by Equation 18.

[ Equation 18 ]

로서,

는 점 (

,

)에서의 평균을 나타내며,

는 (

,

)의 이웃하는

에서 영상

의 표준 편차를 나타내고, 상기

는 아래와 같이 정의된다.here,

as,

Points (

,

) Mean,

Is (

,

Neighbors

Video from

Represents the standard deviation of

Is defined as

Since the range of the correlation has been normalized, it has a value between -1 and 1, and is -1 if the correlation window is completely inaccurate, and 1 if it is correct.

,

)에 대응되는 영상2의 점 (

,

)가 검색되었다고 가정하면, 영상1의 (

,

)에서 영상2의 최대 상관도의 점이 (

,

)라 할지라도, 영상2의 (

,

)를 기준으로 영상1에서 검색 시, 정합점은 (

,

)가 아닐 가능성이 있다. 따라서 본 발명에서는 영상1의 한 점을 기준으로 영상2에서 검색한 후, 정합된 점을 기준으로 다시 영상1에서 검색하여 영상1에서 정합이 되었을 경우에만 최종 정합된 것으로 한다.In order to consider [Equation 17] above,

,

Of the image2 corresponding to)

,

) Is found, the

,

), The point of maximum correlation of Image 2 is (

,

, Even if the

,

When searching in Image 1 based on), the matching point is (

,

May not be). Therefore, in the present invention, the search is performed on the image 2 based on one point of the image 1, and then searched on the image 1 again based on the matched point.

The three-dimensional coordinate calculation unit 117 calculates the three-dimensional coordinates by back projection based on the constructed disparity map.

The polyhedron construction means 119 configures the calculated three-dimensional coordinates into a polyhedron according to the Delauny triangulation method. The texture mapping unit 121 maps a texture to the constructed polyhedron.

Three-dimensional reconstruction is performed from the previously loaded image through the series of means described above.

Experimental example

One.  Experiment on Camera Calibration of Pattern Image.

As an experimental example of the above-described camera calibration, three pattern planes were taken from the left and right images using two cameras, and then experimented with the input images. The input image is shown in FIGS. 4A and 4B. 4C is the result of internal and external variables of the camera according to the present invention.

2.  Experiments on Image Correction and Parallax Map Construction.

For the sample left and right images, the base matrix and epipole were calculated. As shown in FIG. 5A, the epipolar lines on the left and right sides are aligned in parallel, and the images of FIG. 5B are corrected. Since accurate matching must be performed on the corrected image, a matching candidate point is calculated as mentioned above. The calculated matching candidate point is shown in FIG. 5C.

The calculated matching candidate point is the parallax of the left and right images, and after the parallax map is configured with an edge in the corrected left and right images, the calculated parallax distribution is shown in the table of FIG. 5D. In the present experimental example, parallax maps below the threshold were removed as an error, and as shown in FIG. 5D, it can be seen that they are distributed between approximately 108 and 156.

3.  3D reconstruction experiment.

The calculated three-dimensional coordinates are projected by the reverse projection method. The back projected three-dimensional coordinates form a polygon by Delaunay triangulation.

6A is an image reconstructed based on 3D coordinates, and FIG. 6B shows a polygon formed by 3D coordinates. FIG. 6C is an implementation of the polygon constructed in FIG. 6B in OpenGL.

As described above and described with reference to a preferred embodiment for illustrating the technical idea of the present invention, the present invention is not limited to the configuration and operation as shown and described as described above, it is a deviation from the scope of the technical idea It will be understood by those skilled in the art that many modifications and variations can be made to the invention without departing from the scope of the invention. Accordingly, all such suitable changes and modifications and equivalents should be considered to be within the scope of the present invention.

In the case of image-based modeling, camera calibration, basic matrix calculation, parallax map construction, and inverse projection method include a lot of errors in performance. In view of this point, the present invention can simplify camera calibration compared to Z. Zhang's method by using homography for three plane patterns captured on a single image, and disparity map based on the dispersion of corresponding candidate candidates. To configure the, it is possible to minimize the error by removing the parallax below the threshold.

Claims (4)

A three-dimensional reconstruction device using planar homography, Image loading means for receiving a comparative left and right image photographed to include three plane patterns; Feature point extracting means for extracting feature points of the planar pattern from an input left and right image based on a Harris feature point searching method; Homography calculation means for calculating homography between the planar patterns using the extracted feature points; Camera calibration means for calculating and calibrating internal and external parameters of the camera based on the calculated homography and camera model; Basic matrix calculation means for calculating a basic matrix through epipolar geometry based on matching information between feature points of a plane pattern; Image correction means for correcting the image by parallelizing the epipolar lines of the left and right images; Disparity map constructing means for constructing a compact disparity map by finding all matched corresponding points in the two corrected images; Three-dimensional coordinate calculation means for calculating three-dimensional coordinates by a reverse projection method based on the constructed parallax map; Polyhedron construction means for constructing the calculated three-dimensional coordinates into a polyhedron according to the Delaunay triangulation method; And Texture mapping means for mapping the texture to the constructed polyhedron; 3D reconstruction apparatus using planar homography, comprising a. delete The method according to claim 1, The disparity map constituting means, After setting a window of a predetermined size for the pixels of the image, calculating the variance in the window in the left image, searching for pixels of the image larger than a predetermined threshold for variance, and searching for pixels matching the found pixels in the right image. 3D reconstruction apparatus using planar homography, characterized in that to construct a parallax map from the matched corresponding point. A first process of receiving a comparative left and right image photographed to include three plane patterns, a second process of extracting feature points of the planar pattern based on a Harris feature point search method, and an extracted feature point A third process of calculating a homography between the planar patterns using the first and second processes of calculating and correcting internal and external variables of the camera based on the calculated homography and a camera model, and matching between feature points of the planar pattern A fifth process of calculating the basic matrix through the epipolar geometry based on the information, a sixth process of correcting the image by parallelizing the epipolar lines of the left and right images, and finding and finding all matched correspondence points in the two corrected images. A seventh process of constructing a parallax map, an eighth process of calculating three-dimensional coordinates by an inverse projection method based on the constructed disparity map, and calculation And a tenth process of constructing the three-dimensional coordinates into polyhedrons according to the Delaunay triangulation method, and the tenth process of mapping a texture to the constructed polyhedron.

Images