KR20060056050A - Creating method of automated 360 degrees panoramic image - Google Patents

Abstract

The present invention is a method of generating a wide-angle panoramic image including 360 ° by joining a plurality of pictures taken while rotating the body while fixing the camera in one place. The generated panoramic image is viewed through the panoramic VR viewer, giving the user the same effect as standing around the shooting location and looking around. The present invention has the following important features.

1) High-quality 360 ° panoramic images can be generated using only a tripod without special equipment such as rotators.

2) No information about the focal length of the lens is needed.

3) Automatically corrects image distortion that occurs when using wide-angle lenses.

4) It is possible to paste pictures taken when the camera is not level with the ground.

5) Photo paste is done automatically.

6) Automatically corrects differences in brightness between input pictures.

7) Generate cylinder and sphere projection panoramic images.

Template matching, 360 ° panorama images, homography, LM, 8-parameters, projection.

Description

CREATING METHOD OF AUTOMATED 360 DEGREES PANORAMIC IMAGE}             

1 is a flowchart illustrating a method of generating a 360 ° panoramic image according to an embodiment of the present invention.

2 is a flowchart illustrating a process of calculating parameters in more detail.

3 is a flow chart illustrating in more detail the process of calculating relative camera rotation angles between images.

4 is a flowchart specifically illustrating a process of generating a 360 ° panoramic image.

The present invention relates to a method for generating an image, and in particular, to an automated 360 ° panoramic image generation method.

To build a Panoramic Virtual Reality (VR) requires two major steps. The first step is to get a panoramic image, and the second step is to display it through a panoramic VR viewer. The panoramic VR viewer receives a panoramic image as input, and if a user specifies a specific gaze direction, the panoramic VR viewer generates a real-time view of the panorama image when viewed in the gaze direction.

Recently, as panoramic VR became popular on the Internet, many panoramic VR viewers appeared. Currently used viewers include Apple-QuickTime Player, Macromedia-Shockwave Player, PTViewer, Anything3D-Viewer, iSeeMedia-Viewer, Zoomify-Viewer, iPIX-Viewer, MG System-Viewer.

The important of these two steps is the process of obtaining a panoramic image. To create a panoramic image from successive photographs while rotating the body with the camera fixed, the following three parts must be solved.

First, you need to know the correspondence of pixels between neighboring pictures. (Image Registration)

Second, the photographs should be projected to the proper position of the cylinder or sphere. (Cylindrical / Spherical Projection)

Third, when projecting photos into a cylinder or sphere, the overlapping parts of the photos should be smoothly connected. (Image Blending)

The method used in the early days used an algorithm that achieved only when the camera was fully panned to the ground while the picture was taken (Pure Panning) and the focal length of the camera lens was known.

Knowing the focal length of the lens allows you to project the original photograph onto the cylinder. When the camera is rotated while being completely horizontal to the ground, the correspondence between neighboring cylindrical projection images is defined only by translation. Thus, knowing only the amount of translation, it is possible to join neighboring cylindrical projection images to create a panoramic image. The amount of translation between the images can be easily obtained through a method called template matching.

However, there are a number of limitations to this approach. First, you need to know the focal length of the lens accurately. In addition, since the camera is allowed to rotate only when the camera is completely level with the ground, it is difficult to shoot without special equipment such as a rotator.

In addition, since tilting of the camera is not allowed, only a panorama image photographed in a single row may be obtained. In order to overcome these problems, they have not developed the focal length of the lens, and have developed methods for attaching pictures even when there is tilting as well as the camera's panning.

With the camera's position fixed, there is a Planar Projective Transformation relationship defined by a 3x3 matrix between the two images obtained through rotation only. Since Projective Transformation is defined regardless of scale, the Projection Transformation Matrix consists of eight parameters, which is called 8-Parameter Homography.

The methods for obtaining homographies developed so far are divided into two categories.

The first is the Pixel Intensity Based Method, which uses a method of nonlinear minimization that minimizes homography, which minimizes the average difference in brightness between pixels in overlapping images. Obtain by using This method uses all the pixels in the overlapping area, so it works well even if there are few textures or feature points, but it has a disadvantage in that a large amount of calculation is required.

The second is the Feature Based Method. In theory, 8-parameter homography can be obtained given four or more image point correspondence pairs. After extracting feature points such as corner points from the images, and obtaining a corresponding relationship between the feature points between the images, a homograph is obtained using this. This method has the advantage of being fast but lacks accuracy when there are few feature points.

Once the homography between the images has been obtained, it is possible to find the focal length and the relative rotation angle of the camera from it. This is called Camera Auto-Calibration. Knowing the focal length and the angle of rotation of the camera, it becomes possible to project the pictures in the proper position of the cylinder or sphere.

When a panoramic image is generated from photographs taken in multi-row rather than a single row, an area where two or more images overlap is generated. In this case, image registration errors accumulate and it is difficult to obtain proper results. A global registration method has been developed to reduce such accumulation errors. Similar problems arise when generating 360 ° panoramic images. Even if registration is relatively accurate between neighboring images, if an error accumulates, the first and last images may not be correctly registered. In this case, the global registration method is used to solve the problem.

The present invention is proposed to solve the above problems of the prior art, it is possible to generate a high-quality 360-degree panoramic image using only a tripod without special equipment such as rotator, does not require information about the focal length of the lens It can automatically correct image distortion that occurs when using a wide-angle lens.It is also possible to paste pictures taken when the camera is not level with the ground, and to detect differences in brightness between input pictures. It is an object of the present invention to provide an automated 360 ° panoramic image generation method that can automatically correct and generate cylindrical and spherical projection panoramic images.

In order to achieve the above object, the present invention comprises the steps of taking a plurality of pictures in a 360 ° direction; Calculating a parameter consisting of a focal length and a distortion ratio of a camera and a rotation angle between two images using any pair of neighboring photographs selected from the plurality of photographed photographs; Calculating relative camera rotation angles between images from each neighboring photo pair using the calculated parameter; Calculating camera parameters corresponding to each image using relative rotation angles between the images; Correcting the difference in brightness between neighboring images using the camera parameter; Projecting each image whose brightness difference is corrected to a single cylinder; As a result of the projection, performing image blending in an overlapping region to form an unfinished panoramic image; Generating a cylindrical projection image using the first image of the photographic column; And generating a 360 ° panoramic image using the cylindrical projection image and the unfinished panoramic image.

The present invention considers lens distortion, considers camera tilt, and performs image registration using a pair wire national method. It also automatically corrects for differences in brightness between input pictures and generates 360 ° panoramic images.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can more easily implement the present invention.

1 is a flowchart illustrating a 360 ° panoramic image generating method according to an exemplary embodiment of the present invention, and looks at the panoramic image generating process with reference to the flowchart.

First, in order to generate a panoramic image, a plurality of pictures are taken in a 360 ° direction (S101). After selecting a random pair of neighboring pictures from the photographed pictures, a parameter is calculated using the selected random pair of neighboring pictures (S102). The parameters include the focal length and distortion of the camera and the angle of rotation between the two images.

Using the calculated parameters, relative camera rotation angles between images from each neighboring photo pair are calculated (S103). The camera parameter corresponding to each image is calculated using the relative rotation angles between the images (S104).

The difference in brightness between neighboring images is corrected using the camera parameter (S105). Subsequently, each image whose brightness difference is corrected is projected onto a single cylinder (S106).

As a result of the projection, image blending is performed in an overlapping area to form an unfinished panorama image (S107). After generating a cylindrical projection image using the first image of the photo row (S108), a 360 ° panoramic image is generated using the cylindrical projection image and the unfinished panoramic image (S109).

2 is a flowchart illustrating a process of calculating a parameter in more detail.

Referring to FIG. 2, the calculating of the parameter may include calculating an amount of translation between two images through template matching with respect to any pair of neighboring pictures (S201) and LM (Levenberg-Marquardt) minimizing the angle of rotation between the two images through the minimization (S202), and calculating the distortion of the lens through the LM minimization and 8-parameter Homography between the two images (S203) And calculating a focal length of the camera through camera auto-calibration (S204).

3 is a flowchart illustrating in more detail a process of calculating relative camera rotation angles between images.

Referring to FIG. 2, calculating relative camera rotation angles between images may include generating an image in which lens distortion is corrected from each neighboring pair of photographs using the calculated parameters (S301), Generating a cylindrical projection image using the corrected image (S302), and after image rotation, calculating the amount of translation between the two images through template matching (S303), and between the two images through LM minimization Computing the three-parameter homography (S304).

4 is a flowchart specifically illustrating a process of generating a 360 ° panoramic image.

Referring to FIG. 4, the step of generating a 360 ° panoramic image includes calculating a translation amount between two images through template matching using a cylindrical projection image and an unfinished panoramic image (S401), and cutting after merging the images. It consists of a step (S402).

On the other hand, the image pyramid is used to reduce the computation amount when performing template matching and LM minimization.

Hereinafter, the 360 ° panorama image generation process of the present invention will be described in detail by using an equation.

Speaking of points X = (X, Y, Z) points obtained are projected onto the image plane of the two-dimensional x = (x, y) on the three-dimensional space by a standard camera without distortion caused by the lens, with the X and x The relationship may be modeled as in Equations 1 and 2 below.

Where u is a vector representing x in Homogeneous Coordinates, K is a Camera Calibration Matrix, which is a 3D (three-dimensional) point on the camera coordinate system and the 2D ( 2D) represents a relationship with a point, and R and t are a rotation matrix and a translation vector of a camera, and represent a transformation relationship between a world coordinate system and a camera coordinate system. Parameters contained in matrix K This parameter is called an internal parameter, where f is the focal length, a is the aspect ratio of the pixels, s is the skew, and (x _c , y _c ) is the principal point. .

Now let's look at the correspondence of the pixels between the image I ₁ obtained after fixing the projection center of the camera and rotating the camera by a certain amount and the image I ₀ obtained before the rotation.

The rotation matrix and the translation of the camera before the rotation are called R ₀ and t ₀ , respectively, and the one after the rotation is called R ₁ , t ₁ . For convenience, if the projection center of the camera coincides with the origin of the world coordinate system, t ₀ = t ₁ = 0. If a single point X in a three-dimensional space is projected onto I ₀ and I ₁ , and a single vector (Homogeneous Vector) is u ₀ and u ₁ , respectively, it can be seen that they have a relationship as shown in Equation 3 below. .

That is, the correspondence of the pixels between two images obtained by rotating the camera is represented by a matrix H of 3 × 3, which is called planar homomography. Considering that even if a single vector is multiplied by any real number other than zero, it can be seen that H is also defined regardless of scale. Thus, planar homography is defined by eight parameters rather than nine. In fact, since H = KR ₀₁ K ^-1 , the number of parameters included in K is 5, and the number of parameters included in R ₀₁ is 3 (see Equation 4 below), so the number of independent parameters constituting H is It can be seen that there are eight. Such homography is called 8-parameter homography.

Given two images, finding the correspondence H of the pixels between the two images is called image registration. H can usually be obtained if four or more correspondences are given, but care must be taken to obtain an accurate H.

Now let's look at a more simplified model. Most cameras on sale meet the following conditions.

1) Most of the pixels are square pixels, so the aspect ratio of the pixels satisfies a = 1.

2) Since most of the CCD (Charge Coupled Device) cells forming the image plane are arranged at right angles, the skew component satisfies s = 0.

3) The optical axis of the camera can be assumed to pass through the center of the CCD array, so the main point (x _c , y _c ) coincides with the center of the image.

Thus, in the simplified model, the number of independent parameters included in H is four (the focal length f and the three rotation angles included in R ₀₁ ), which is called 4-parameter homography.

For reference, the general rotation matrix R is decomposed as shown in Equation 4 below.

Here, R (Ψ), R (Θ), and R (Φ) are matrices representing rotational transformations around the X, Y, and Z axes, respectively, and Ψ, Θ, and Φ are referred to as Euler Angles.

Given Planar Homography H in a simplified model, K and R can be expressed by the following equations. The matrix K may be decomposed as shown in Equation 5 below.

Substituting Equation 5 into Equation 3 results in Equation 6.

Therefore, R is expressed as in Equation 7 below.

Since the matrix R is an orthogonal matrix, the following equation 8 is satisfied.

After f is obtained from Equation 8, R can be obtained by substituting Equation 7.

On the other hand, in obtaining f and R from H, the following caution is required.

1) Since H is given regardless of scale, after calculating R from Equation 7, the scale value of R should be adjusted so that RR ^T = I.

2) If the given H does not satisfy the four-parameter homography model, R obtained through Equation 7 is not an orthogonal matrix. Therefore, an orthogonal matrix approximated using Singular Value Decomposition (SVD) must be obtained from the first obtained R.

The camera model mentioned above does not take into account the distortion caused by the lens. Since a wide angle lens is often used to capture a 360 ° panoramic image in reality, it is necessary to consider the lens distortion phenomenon to apply the algorithm in a wider range. (x _d , y _d ) is called the coordinate of the pixel in the original image (ie, the image with lens distortion), and (x _d , y _d ) is called the corresponding pixel coordinate in the image from which the lens distortion has been removed. If (x _c , y _c ) is the center coordinate of the original image, (x _d , y _d ) is given by (x _u , y _u ) as shown in Equation 9 below.

Here, K ₁ and K ₂ are referred to as radial distortion coefficients. In most cases, considering only the first radiation distortion coefficient term, it is sufficient to assume that K = K ₁ , and Equation 9 is simplified as in Equation 10 below.

On the contrary, when (x _u , y _u ) is given, (x _d , y _d ) is represented by Equation 11 below.

Here, r _d is equal to K and Equation 12 below,

Since h (K) and the same as Equation 13 below,

r _u has a relationship with (x _c , y _c ) and (x _u , y _u ) as shown in Equation 14 below.

Hereinafter, camera parameter extraction (Camera Parameter Extraction).

Given the photographs taken by the user as input, the panoramic image generation tool performs image registration with any neighboring pair of photographs to find the 8-parameter homography H and the radiation distortion coefficient K, and then Find the focal length f using 8.

Assuming that no lens is changed or zoomed in the middle of taking a picture, the f and K values are fixed. Therefore, the f and K obtained from any pair of images may be applied to all images in common. Do.

The image registration method used in the present invention is as follows.

In a given pair of images, the first _one is called I ₀ and the second _one is called I ₁ . Further, as any pixel x = (x, y) the pixel coordinates of the _{I 1 x ^ = (x ^} , y ^) corresponding to belonging to I _0. The planar homograph H that converts x into x ^ is expressed by Equation 15 below.

Since H is defined regardless of scale, the last element of H is set to 1. If it is assumed that there is no lens distortion, and a single coordinate system of x is set to (x, y, 1), the corresponding relationship between x and x ^ may be expressed by Equation 16 using Equation 3 below.

In order to find H, a cost function e defined as in Equation 17 below is considered.

Where R represents an overlapping region between I ₀ and I ₁ , N _R represents the number of pixels belonging to R, and I (x, y) is the number of pixels present at the position (x, y) in image I Indicates the brightness value.

Then, we consider the optimal H to find H that minimizes the cost function e. Assuming that the pixels obtained by projecting a point in three-dimensional space into different images all have the same brightness, it is reasonable that H, which minimizes the cost function e presented above, becomes the optimal H to find.

Given an initial initial value of H, we can find H to minimize e using a numerical nonlinear minimization method, typically Levenberg-Marquardt minimization. There is a way.

For reference, Levenberg-Marquardt (hereinafter referred to as LM) minimization method is a slight modification of the Newton Minimization method.

In the following, we briefly look at the Newton minimization method.

It is assumed that the cost function e to be minimized is given in the following formula (18).

Where m = (m ₁ , ...., m _M ) is a parameter vector. Given the m ₀ value of the currently given parameter vector,

Assuming that we find a vector Δm that satisfies Eq. (19), if we update m ₀ to m ₀ ← m ₀ + Δm and then repeat looking for Δm that satisfies (19), we end up with a local minimum of e ( Local Minimum) will be reached. There are many ways to find Δm, but the Newton minimization method looks like this:

If e (m ₀ + Δm) is expanded only to the first term in the Taylor Series from around m ₀ , it can be approximated by Equation 20 below.

Here, the matrix J (m) is called a Jacobian matrix and is given by Equation 21 below.

The Newton minimization method finds Δm which minimizes || e (m ₀ ) + J (m ₀ ) Δm ||, where Δm is given by Equation 22 below.

Here J = J (m ₀ ), J ⁺ is called Pseudo-Inverse of J.

If A = J ^T J in Equation 22 and a _ij is an element located in (i, j) of the matrix A, the LM minimization method replaces the matrix A with the matrix A ^ given by Equation 23 below. do.

The initial value of λ is usually set to 10 ⁻³ . If Δm is calculated using the currently set λ value, the value of e is decreased to λ ← 0.1λ, and if the value of e is increased, it is updated to λ ← 10λ. When the lambda value becomes very small, the LM minimization method becomes similar to the Newton minimization method, and when the lambda value is very large, it can be observed that the LM minimization method becomes similar to the gradient decent minimization method. In general, the LM minimization method is known to operate faster and more stably than the Newton minimization method.

In order to minimize the equation 17 using the LM minimization method, the Jacobian matrix of the equation 21 should be obtained. _Suppose J _ij located at (i, j) of J, the element is given by the following equation (24) by a chain rule.

는

의 위치에서 I₁의 구배(gradient)가 된다.From here

Is

It is a gradient of I ₁ at the position of.

The present invention provides an image registration method considering lens distortion. To consider the lens distortion phenomenon, Equations 16 and 17 should be changed to Equations 25 and 26, respectively.

Where (x _d , y _d ) ↔ (x _u , y _u ) is the conversion relationship (or (x ^ _d , y ^ _d ) ↔ (x ^ _u , y ^ _u )) As given in

(또는

)를 구하는데 있어서, 분석적인(Analytic) 방법에 의하여

의 일반적인 형태를 얻기 힘들면, 수식적인(Numerical) 방법에 의하여 구해야만 한다. For reference, in Equation 24

(or

), By an analytical method

If the general form of is difficult to obtain, it must be obtained by a numerical method.

In general, given a function f (x), df / dx at x = x ₀ can be approximated as shown in Equation 27 below.

Here, Δx is a relatively small positive real number compared to | x ₀ |, and it is known that a good result can be obtained by setting the following equation (28).

In order to minimize the cost function e of Equation 17 by the LM minimization method, and to properly obtain H corresponding to the desired answer, the initial value of H must be properly given. Most formal minimization methods, including the LM minimization method, do not look for a global minimum, but instead find a local minimum. If the initial values are not given properly, you will find the wrong answer.

To obtain the initial value of H to know the approximate correspondence between the two images I ₀ and I _1, so that the two images are appropriately overlapped allow the user to secure the position of the I ₀ and to move the I ₁ to the beginning of the I ₁ There is a way to provide a UI to get a location or to specify more than four correspondences between two images, but doing so can be very cumbersome for the user. Such a method was avoided because automation was seen as one of the main points in the present invention.

Fixing the position of the I ₀ was the way to go to I ₁ by measuring the degree of well-stacked to locate the position of the two images I ₁ that overlaps should be well defined first. There are several ways to measure how well the two images overlap, which is expressed as the sum of the squares of the difference in brightness of the corresponding pixel between the two images at the current position, as shown in Equation 17. It is called a sum of squared difference ( SSD ). The smaller the SSD value, the better the two images overlap, but this method has a problem that it may not work properly when the difference between the overall brightness and contrast of the two images is large.

As a method irrespective of the difference in overall brightness and contrast, there is a method of measuring a Normalized Cross-Correlation ( NCC ) value between two images. The NCC is defined as Equation 29 and Equation 30 below.

Here, R denotes an area where two images I ₀ and I ₁ overlap, N _R denotes the number of pixels belonging to R, and (x _t , y _t ) denotes an amount of shift of I ₁ . The larger the value of NCC, the better I ₀ and I ₁ overlap.

Measuring NCC values for all pixel positions to find (x _t , y _t ) where the NCC value is largest is called template matching. In reality, when taking a picture, considering that the direction of rotation of the camera is always constant and horizontal rotation, the search area can be significantly reduced in finding the position having the largest NCC value. Therefore, these points should be taken into account when performing template matching.

If the position found by the template matching is (x ₀ , y ₀ ), the initial value H ₀ of H may be given by Equation 31 below.

Applying the above method directly to the original size image causes the following problems.

That is, because the gap between the initial value H ₀ and H corresponding to the correct answer is still large, it does not converge to the desired answer, and the computational amount becomes too large when performing template matching or LM minimization.

Image Pyramid is used to solve these problems. The image pyramid consists of a given original image and images created by minimizing the image. If the set of images constituting the image pyramid of the original image I is Pyramid (I) = {I ⁽⁰⁾ , I ⁽¹⁾ , ..., I ⁽ⁿ⁾ }, I is expressed as in Equation 32 below. Is expressed.

Here Reduce () is an operator that reduces the image, which is usually reduced by 0.5 times.

To perform efficient template matching and LM minimization, we generate image pyramids Pyramid (I ₀ ) and Pyramid (I ₁ ) of I ₀ and I ₁ .

Then, starting with I ₀ ⁽⁰⁾ and I ₁ ^(0), we perform template matching or LM minimization in turn. At this time, utilizing the result from the I ₀ ⁽ⁱ⁾ and I ₁ ⁽ⁱ⁾ template matching, or LM minimized in performing the, I ₀ ^(i-1) and I ^{_{1 (i-1)}} as an initial value.

If you do this, you will find a more accurate answer as you get closer to the bottom of the pyramid. When performing template matching, the amount of computation can be significantly reduced by setting the search area to the vicinity (within a few pixels) of the position obtained in the previous step.

In addition, considering the case of performing LM minimization, when applied to I ₀ ⁽⁰⁾ and I ₁ ⁽⁰⁾ , since H ₀ of Equation 31 is used as the initial value of H because the size of the image is quite small, You can observe good convergence.

Also, by using the results from the previous stage of the pyramid as the initial value of the next stage, it can be observed that convergence is very fast. Therefore, using the image pyramid not only converges well to the desired answer when performing LM minimization, but also significantly reduces the amount of computation.

In addition, the present invention provides an image registration method in consideration of camera tilt. Hereinafter, an image registration method in consideration of camera tilt will be described.

When taking a picture, if there is a tilt angle of the camera (an angle formed between the ground and the direction viewed by the camera), a rotation angle exists between neighboring pictures.

According to the experiment, when the rotation angle is relatively large, when H ₀ of Equation 31 is used as the initial value, even if the image pyramid is adopted, it does not converge properly. To solve this problem, the rotation angle existing between the images must be reflected in H ₀ . Given a rotation angle Θ ₀ between I ₀ and I ₁ , H ₀ can be modified as in Equation 33 below.

In order to obtain the rotation angle Θ ₀ , the cost function e is minimized using the LM minimization method as shown in Equations 34 and 35 below.

At this time, the initial values of the parameters are set to (Θ, x _t , y _t ), and an image pyramid is introduced to facilitate convergence.

The present invention also provides a pairwire local image registration method.

Given the focal length f, the camera's internal parameters, and the coefficient of radiation distortion, K, use them to take each image pair {I ₁ , I ₂ , ..., I _n } into the input image column {I _i , I _{i +} Image registration is performed for ₁ } to obtain homography H _{i (i + 1)} therebetween.

 When performing image registration, the image registration method 2 considering the camera tilt may still be applied, but in this case, there is a problem of having different f and K values for each pair of images.

Therefore, the existing f and K should be used in common, and a method of obtaining the values of the remaining parameters should be adopted.

First, an image without lens distortion is obtained using the K value obtained in the previous section. Then, image registration based on 4-parameter homography is performed. The four parameters included in the four-parameter homography consist of the focal length f and the camera's rotation angles Ψ, Θ and Φ, where f is fixed to a given value and only the camera's rotation angle is obtained. In other words, Ψ, Θ, and Φ for minimizing the cost function e shown in equations (36) and (37) are obtained.

Here, the rotation matrix R _{i (i + 1)} is given by Equation 4.

To minimize the cost function given above, the initial values of Ψ, Θ, and Φ should be given appropriately. First, template matching is performed with the image {I ^ _i , I ^ _{i + 1} } obtained by cylindrically mapping a given image pair {I _i , I _{i +1} } to obtain initial values of Ψ and Θ. Then, we obtain the rotation angle between I ^ _i and I ^ _{i + 1} to get the initial value of Φ.

Images obtained by planar projection can be converted to cylindrical projection images or spherical projection images given camera parameters. If the calibration matrix and the rotation matrix of the camera corresponding to the image I obtained by the plane projection are K and R, respectively, the direction vector X of the optical ray passing through any pixel x = (x, y) belonging to I = (X, Y, Z) is given by Equation 38 below.

Suppose the center of the cylinder with radius 1 is positioned to coincide with the projection center of the camera. We can then think of the intersection point v = (θ, v) where the ray X meets the cylinder, which corresponds to the pixel x X I in the cylindrical projection image I. Θ is an angle formed by the plane including the Y axis and the light ray with the YZ plane, and v is the Y axis coordinate of the intersection point, which is given by Equation 39 below.

In the case of projecting onto a sphere having a radius of 1, the coordinates (θ, Φ) of the intersection point where the ray X meets the sphere are given by Equation 40 below.

Is the angle that the ray makes with the XY plane.

If the camera is perfectly horizontally rotated around the Y axis (i.e. when the camera's rotation angles Ψ and θ are 0), the camera's rotation angle is called Δθ, and the cylindrical projection image obtained before the rotation is I ^ ₀ , If the image obtained after the rotation is I ^ ₁ , it can be seen that the relationship shown in Equation 41 is established.

In other words, I ^ ₁ can be obtained by simply shifting I ^ ₀ . Therefore, when the camera takes a picture with only the horizontal rotation about the Y axis, if the camera knows only the focal length f of the camera, it makes a cylindrical projection image of each picture and then calculates the amount of shifting between neighboring images through template matching. Registration is possible. When obtaining the cylindrical projection image of each picture, let R = I in equation (38).

In the present invention, a more general camera rotation is considered, and thus, without performing image registration in the above manner, the result obtained by performing template matching as mentioned above is used as an initial value of the parameter to minimize the cost function of Equation 36. do.

If the shift amount of I ^ _{i + 1} obtained by template matching a cylindrical projection image {I ^ _i , I ^ _{i + 1} } of two neighboring images {I _i , I _{i + 1} } is (Δθ, Δv), The initial values of Ψ and θ, Ψ ₀ and θ ₀ representing the angles of rotation of the camera in the X and Y axes are given by Equation 42 below.

As the initial value of the rotation angle Φ to the Z axis of the camera, a rotation angle between I ^ _i and I ^ _{i + 1} is used. The method of obtaining is obtained by minimizing Equation 34 in the image registration method considering the camera tilt. Is the same as

Given an initial value of the camera rotation angle, the cost function e of Eq. 36 is minimized by using the LM minimization method to obtain an accurate camera rotation angle. The method presented above introduces an image pyramid to improve convergence and speed.

When image registration is performed by the proposed method, more stable and uniform results are obtained for each image pair than the 8-parameter homography based.

In addition, the present invention provides a method for automatically correcting a brightness difference existing between input pictures.

Even if the pictures are taken with the same camera, there is a difference in overall brightness and contrast between the pictures. If a panoramic image is generated without correcting such a difference, an unnatural boundary occurs at the joint.

The overall brightness and contrast of any image can be expressed as the mean and standard deviation of the brightness of the pixels. In order to correct the difference in brightness between two neighboring images I _i and I _{i +1} , the average value and standard deviation of the brightness of pixels in the overlapping area are obtained. Given a homograph between I _i and I _{i +1} , we can find areas where they overlap. The mean value and standard deviation of the brightness of I _{i in} the overlapping region are called m _i ^r and σ _i ^r , and the mean value and standard deviation of the brightness of I _{i +1} are called m ^l _{i +1} and σ ^l _{i +1} . In order to make the average value of the two images equal to the standard deviation, the pixel values of I _{i + 1} must be corrected as shown in Equations 43 and 44.

Given a picture sequence {I ₁ , I ₂ , ..., I _n }, the pixel brightness of I _{i + 1 for} the neighboring image pair {I _i , I _{i +1} } is corrected using Equation 43 Consider the case where things are done sequentially.

Unless you are creating a 360 ° panoramic image, there is no particular problem.However, if you are creating a 360 ° panoramic image, the difference between the brightness of I _n and I ₁ should be corrected. In the case of correction, the difference in brightness between I _n and I ₁ cannot be guaranteed.

When creating a 360 ° panoramic image, brightness correction takes the form of a cyclic chain, so one sequential correction cannot solve the problem. In the present invention, as shown in Equations 45 to 47, the problem is solved by obtaining a ₁ and b ₁ for minimizing the cost function.

The method for obtaining the value of e shown in Equation 45 is as follows. First, a ₁ and b ₁ are obtained by using m _n ^r , σ _n ^r , m ₁ ^l , and σ ₁ ^r as shown in Equation 46.

M ₁ ^l , σ ₁ ^l , m ₁ ^r , and σ ₁ ^r are updated as shown in Equation 47 using the obtained a ₁ and b ₁ .

Using the updated m ₁ ^r , σ ₁ ^r, and the existing m ₂ ^l , σ ₂ ^l , we find a ₂ and b ₂ . This is repeated to find m _n ^r and sigma _n ^r, and then the value of e.

When a ₁ and b ₁ are minimized to minimize the cost function e, the remaining a _i and b _i are obtained in the above-described manner, and the brightness of the image is corrected using Equation 43.

Thus, the present invention can generate a 360 ° panoramic image.

If the picture was taken at 360 ° and the camera parameter values were perfectly obtained, I _n and I ₁ would automatically overlap without error by the cylindrical projection previously performed. Theoretically speaking, when the rotation matrix of the camera obtained from homography between I _n and I ₁ is R _{n 1} , R is expressed by Equation 48 below.

That is, R shown in Equation 48 should be a perfect identity matrix when the rotation matrices are obtained without error. In any case, however, calculation errors are often included, so that a 360 ° panoramic image cannot be obtained simply by fairware local image registration. To solve this problem, we can consider a global image registration problem that minimizes the cost function e as shown in Eq.

However, this method is too computational and does not guarantee accurate 360 ° panorama image generation.

In the present invention, a 360 ° panoramic image is generated in a simpler and faster way. When the panorama image obtained through the above process is referred to as I _p , a method of generating a 360 ° panorama image is as follows.

Step 1: When a region I ₁ is mapped at I _p as R _1, to extract only the pixels belonging to R ₁ from I _p to make the image I ^ _1.

Step 2: When the I _n the mapped area in the I _p as R _n, performs a template matching of I _p and I ^ ₁ in R _n. The position of I _p corresponding to (0,0) of I ^ ₁ is called (x _m , y _m ).

Step 3: After placing the point of (0,0) of I ^ ₁ at (x _m , y _m ) on I _p , image blending is performed to make image I _p '.

Step 4: the center point coordinates of the _{I ^ 1 (x c, y} c) that if, I _p 'from taking the area between the two vertical line x = x _c and x = x _m + x _c images I _p "the Make.

Step 5: Shearing and cropping I _p "in the vertical direction so that both ends of I _p " match exactly.

A tremendous amount of digital cameras is now available, and film cameras are losing their place. In addition, due to the tremendous spread of the Internet, digital cameras and the Internet or contents are complementarily developed. Due to the present invention as described above, it is possible to generate a high quality 360 degree panoramic image using only a tripod without special equipment such as a rotator, does not require information about the focal length of the lens, and appears when using a wide-angle lens Image distortion can be corrected automatically, photos taken without the camera being level with the ground can be pasted, and brightness differences between input pictures can be corrected automatically. It has been seen through the examples that it is possible to generate a) and sphere projection panoramic image.

Although the technical idea of the present invention has been described in detail according to the above preferred embodiment, it should be noted that the above-described embodiment is for the purpose of description and not of limitation. In addition, those skilled in the art will understand that various embodiments are possible within the scope of the technical idea of the present invention.

As described above, the present invention allows an image-based virtual space to be easily and quickly created without a special technology using a digital camera, thereby creating a larger number of image-based virtual space contents, which can be expanded throughout the industry. .

Claims (5)

Taking a plurality of pictures in a 360 ° direction; Calculating a parameter consisting of a focal length and a distortion ratio of a camera and a rotation angle between two images using any pair of neighboring photographs selected from the plurality of photographed photographs; Calculating relative camera rotation angles between images from each neighboring photo pair using the calculated parameter; Calculating camera parameters corresponding to each image using relative rotation angles between the images; Correcting the difference in brightness between neighboring images using the camera parameter; Projecting each image whose brightness difference is corrected to a single cylinder; As a result of the projection, performing image blending in an overlapping region to form an unfinished panoramic image; Generating a cylindrical projection image using the first image of the photographic column; And Generating a 360 ° panoramic image using the cylindrical projection image and the unfinished panoramic image 360 ° panoramic image generation method comprising a. The method of claim 1, Computing the parameter, Calculating the amount of translation between the two images through template matching for any pair of neighboring photos, Calculating the angle of rotation between the two images by minimizing the Levenberg-Marquardt (LM), Calculating the distortion of the lens through LM minimization and the 8-parameter homography between the two images, 360 ° panoramic image generation method comprising the step of calculating the focal length of the camera through camera auto-calibration. The method of claim 1, Computing relative camera rotation angles between the images, Generating an image in which lens distortion is corrected from each neighboring pair of photographs using the calculated parameters; Generating a cylindrical projection image using the image with the lens distortion corrected; After image rotation, calculating the amount of translation between the two images through template matching, Computing a three-parameter homography between two images through LM minimization. The method of claim 1, Generating the 360 ° panoramic image, Calculating the amount of translation between the two images through template matching using the cylindrical projection image and the unfinished panoramic image; 360 ° panoramic image generation method comprising the step of cutting after image coalescing. The method of claim 2 or 3, And an image pyramid to reduce the amount of computation in performing the template matching and the LM minimization.

Images