KR102159134B1 - Method and system for generating real-time high resolution orthogonal map for non-survey using unmanned aerial vehicle - Google Patents

Abstract

Disclosed are a method for generating a real-time high-resolution orthogonal map for non-survey using an unmanned aerial vehicle to enable a user to quickly check a precise map, and an apparatus thereof. According to one embodiment of the present invention, the method comprises the following steps: performing calibration to acquire correction data for a camera; when receiving an image captured by the camera from an unmanned aerial vehicle flying with the camera, using the correction data to perform first correction of correcting a lens and tangent distortion with respect to the image; and performing second correction for performing orthogonal correction with respect to the first corrected image to generate an orthogonal image.

Description

Real-time high resolution orthogonal map generation method for non-surveying using UAV and real-time high resolution orthogonal map generation device for non-surveying {METHOD AND SYSTEM FOR GENERATING REAL-TIME HIGH RESOLUTION ORTHOGONAL MAP FOR NON-SURVEY USING UNMANNED AERIAL VEHICLE}

The present invention is, at the same time as the image capture by an unmanned aerial vehicle such as a drone, by receiving an image captured in real time through a communication network and converting it into a precision map, the user can quickly check a map service reflecting the captured image. It relates to a real-time high-resolution orthogonal map generation method for non-surveying using an airplane and an apparatus for generating real-time high-resolution orthogonal maps for non-surveying.

In general, in order to obtain a high-resolution orthogonal map by drone shooting, a computer running a dedicated orthogonal map generation software by first performing a shooting flight by a drone and then acquiring a number of images captured from the drone after the shooting flight is finished, etc. And use the method of obtaining an orthogonal map through the image processing step through this dedicated orthogonal map generation software.

However, this method of obtaining an orthogonal map has a problem in that it takes time to shoot a drone and also requires resources to merge a plurality of captured images. In addition, in a general method of obtaining an orthogonal map, there is a problem that it takes too much time to generate a final result and provide it as a map service because an image processing step must be performed.

For example, according to the general method of obtaining an orthogonal map, it takes about 50 minutes of flight time to take 1000 photos with a drone, and the captured 1000 photos are finally converted into an orthogonal map through image processing. It is known that it takes about 5 hours more to generate. The processing time for the image processing in the above example is a high-end computer with a relatively high computer specification (Intel(R) Xeon(R) Gold 6154 3.0Ghz and 3.7Ghz (turbo) 18 physical core, 4k (4000x3000 resolution)). It may be the time you need to do it.

Accordingly, there is an urgent need for a new technique for dramatically shortening processing resources by enabling a precise map to check an image captured in real time simultaneously with drone photography.

An embodiment of the present invention is linked to photographing an image from a camera mounted on an unmanned aerial vehicle, by receiving an image captured from an unmanned aerial vehicle and converting the generated orthogonal image into a tile map in real time, allowing the user to An object of the present invention is to provide a method for generating a real-time high-resolution orthogonal map for non-surveying using an unmanned aerial vehicle and an apparatus for generating a real-time high-resolution orthogonal map for non-surveying.

In addition, an embodiment of the present invention aims to rapidly generate a precise orthogonal image by continuously performing lens and tangential distortion correction and orthogonal correction through the gimbal posture information of the image on an image received from an unmanned aerial vehicle. .

According to an embodiment of the present invention, a method of generating a real-time high-resolution orthogonal map for non-surveying using an unmanned aerial vehicle includes: obtaining correction data for a camera by performing calibration; Performing primary correction of correcting lens and tangential distortion on the image by using the correction data as the image captured by the camera is received from the unmanned aerial vehicle in flight with the camera mounted thereon; And generating an orthogonal image by performing a second correction of orthogonal correction on the first corrected image.

In addition, according to an embodiment of the present invention, an apparatus for generating a real-time high-resolution orthogonal map for non-surveying using an unmanned aerial vehicle includes: an acquisition unit configured to perform calibration to obtain correction data for a camera; And performing a first-order correction of correcting lens and tangential distortion on the image using the correction data as an image photographed by the camera is received from an unmanned aerial vehicle in flight with the camera mounted, and the first The corrected image may be configured to include a processing unit for generating orthogonal images by performing secondary correction for orthogonal correction.

According to an embodiment of the present invention, in conjunction with photographing an image from a camera mounted on an unmanned aerial vehicle, by receiving an image taken from an unmanned aerial vehicle and converting the generated orthogonal image into a tile map in real time, It is possible to provide a method for generating a real-time high-resolution orthogonal map for non-surveying using an unmanned aerial vehicle and an apparatus for generating a real-time high-resolution orthogonal map for non-surveying.

In addition, according to an embodiment of the present invention, a precise orthogonal image can be quickly generated by continuously performing lens and tangential distortion correction and orthogonal correction through the gimbal posture information of an image received from an unmanned aerial vehicle. have.

1 is a block diagram showing the configuration of an apparatus for generating real-time high-resolution orthogonal maps for non-surveying using an unmanned aerial vehicle according to an embodiment of the present invention.
2 is a diagram for explaining an example of a camera projection model.
3A is a diagram illustrating a GML camera calibration tool, and FIG. 3B is a diagram illustrating an execution result of a GML camera calibration tool.
4 is a diagram for explaining an example of correction processing for lens distortion and tangential distortion according to the present invention.
FIG. 5A is a view for explaining an example of perspective projection, and FIG. 5B is a diagram showing the relationship of FIG. 5A.
6 is a diagram illustrating an example of an observation frustum in order to explain the orthogonal transformation.
7 is a diagram for explaining derivation of coordinate information according to the present invention.
8 is a diagram showing movement of GPS coordinates before and after image correction according to the present invention.
9 is a diagram illustrating an example of processing according to the present invention when overlapping tiles are stored in GPS coordinates.
10 is a flowchart illustrating a method of generating a real-time high-resolution orthogonal map for non-surveying using an unmanned aerial vehicle according to an embodiment of the present invention.

Hereinafter, exemplary embodiments will be described in detail with reference to the accompanying drawings. However, since various changes may be made to the embodiments, the scope of the rights of the patent application is not limited or limited by these embodiments. It should be understood that all changes, equivalents, or substitutes to the embodiments are included in the scope of the rights.

The terms used in the examples are used for illustrative purposes only and should not be interpreted as limiting. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present specification, terms such as "comprise" or "have" are intended to designate the presence of features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, but one or more other features. It is to be understood that the presence or addition of elements or numbers, steps, actions, components, parts, or combinations thereof, does not preclude in advance.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiment belongs. Terms as defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and should not be interpreted as an ideal or excessively formal meaning unless explicitly defined in this application. Does not.

In addition, in the description with reference to the accompanying drawings, the same reference numerals are assigned to the same components regardless of the reference numerals, and redundant descriptions thereof will be omitted. In describing the embodiments, when it is determined that a detailed description of related known technologies may unnecessarily obscure the subject matter of the embodiments, the detailed description thereof will be omitted.

1 is a block diagram showing the configuration of an apparatus for generating real-time high-resolution orthogonal maps for non-surveying using an unmanned aerial vehicle according to an embodiment of the present invention.

1, a real-time high resolution orthogonal map generating device 100 for non-surveying using an unmanned aerial vehicle according to an embodiment of the present invention (hereinafter, abbreviated as'real-time high-resolution orthogonal map generating device for non-surveying') May be configured to include an acquisition unit 110 and a processing unit 120. In addition, the apparatus 100 for generating a real-time high-resolution orthogonal map for non-surveying may be configured by selectively adding a mapping unit 130 and a receiving unit 140 according to an embodiment.

First, the acquisition unit 110 performs calibration to obtain correction data for a camera. That is, the acquisition unit 110 is mounted on an unmanned aerial vehicle such as a drone, and performs a correction operation for a camera that generates an image by performing a photographing operation while the unmanned aerial vehicle is photographing and flying. , Including the degree of tangential distortion).

The calibration measures the focal length or aperture scale of the camera lens according to the change in the distance between the camera and the subject or the incident light, and displays it in advance to determine in advance what state the image generated by the camera is generated in. It can be a task.

In other words, the acquisition unit 110 may recognize a degree of distortion due to the current state of the camera through calibration, and obtain a result value as the correction data.

In addition, the processing unit 120 performs primary correction for correcting lens and tangential distortion for the image using the correction data as an image photographed by the camera is received from an unmanned aerial vehicle in flight with the camera mounted thereon. Perform. That is, the processor 120 may correct the image by excluding distortion of the camera itself by performing a correction operation for lines and surfaces in consideration of the previously acquired correction data for the image received from the unmanned aerial vehicle.

In addition, the processing unit 120 generates an ortho image by performing a second correction for orthogonal correction on the first corrected image. That is, the processing unit 120 may perform orthogonal correction on the lens and the tangential distortion-corrected image again, thereby calculating an orthogonal image as viewed vertically from the sky.

In order to receive an image from an unmanned aerial vehicle, the apparatus 100 for generating a real-time high resolution orthogonal map for non-surveying according to the present invention may further include a receiving unit 140.

The receiving unit 140 receives the image captured from the unmanned aerial vehicle in real time through a communication network in conjunction with the image being captured by the camera. That is, the receiving unit 140 may serve to perform the aforementioned first and second corrections by the processing unit 120 by receiving the corresponding image whenever the camera generates an image.

Here, the communication network refers to a network that transmits an image generated by connecting the device 100 for generating a real-time high resolution orthogonal map for non-surveying and an unmanned aerial vehicle, and may be illustrated as 4G and 5G LTE.

In the second correction, the processing unit 120 may generate the orthogonal image by reflecting the gimbal posture information of the image and processing orthogonal correction on the first corrected image. The gimbal posture information includes yaw information about the z-axis rotation of the first corrected image, pitch information about the x-axis rotation of the first corrected image, and the first corrected image. It may include roll information about the y-axis rotation.

That is, the processing unit 120 may produce a three-dimensional orthogonal image through rotation correction for each axis in a three-dimensional space with respect to the image corrected by the first correction.

In another embodiment, the processing unit 120 corrects the center coordinate of the orthogonal image by reflecting at least one of the angle of view, the pitch rotation angle, and the altitude value of the camera, and the corrected center coordinate of the image The GPS coordinates of the upper left corner and the lower right corner are derived, converted into a TIFF file, and inserted into the orthogonal image together with the coordinate information (GEO-Reference) information. That is, the processing unit 120 includes coordinate information of a reference object on the ground that can be an indicator in the generated orthogonal image, so that in relation to the orthogonal image generated according to the correction, the position of the ground on which the image was photographed is accurately recognized. You can apply.

The generated orthogonal image can be visualized by being expressed on a map.

To this end, the apparatus 100 for generating a real-time high-resolution orthogonal map for non-surveying may further include a mapping unit 130 that maps an orthogonal image to an online map.

The mapping unit 130 converts the orthogonal image into a first tile map and stores it in a storage associated with TMS (Tile Map Service), so that the first tile map is displayed on the online map when the service is requested by the TMS. Make it overlay. That is, the mapping unit 130 generates an orthogonal image generated for each image as a tile set of a tile sheet texture (2D image), and the tile size of the desired tile, border maring, and tile spacing (Per Tile-Spacing) After adjusting the tileset according to the like, it may be converted into a first tile map. In this case, the tile map can set the size of the entire map and the tile size to be used by setting the number of horizontal and vertical tiles (Map Width and Map Height) and tile size (Tile Width and Tile Height) of the online map according to the tileset to be used.

In addition, the mapping unit 130 may store the first tile map in a storage associated with the TMS by matching the GPS coordinates in which the image is captured. At this time, when a pre-registered second tile map corresponding to the GPS coordinate is stored in the storage, the processing unit 120 compares the image occupancy ratio between the first tile map and the second tile map, and A tile map having a large image occupancy ratio can be stored in the storage.

That is, the mapping unit 130 stores only one relatively high-quality tile map corresponding to one GPS coordinate, so that the tile map provided in the online map service is not duplicated and stored, thereby saving storage resources. .

According to an embodiment of the present invention, in conjunction with photographing an image from a camera mounted on an unmanned aerial vehicle, by receiving an image taken from an unmanned aerial vehicle and converting the generated orthogonal image into a tile map in real time, It is possible to provide a method for generating a real-time high-resolution orthogonal map for non-surveying using an unmanned aerial vehicle and an apparatus for generating a real-time high-resolution orthogonal map for non-surveying.

In addition, according to an embodiment of the present invention, a precise orthogonal image can be quickly generated by continuously performing lens and tangential distortion correction and orthogonal correction through the gimbal posture information of an image received from an unmanned aerial vehicle. have.

The apparatus 100 for generating a real-time high resolution orthogonal map for non-surveying according to the present invention may implement a method of selectively storing a duplicate tile map.

The real-time high-resolution orthogonal map generation device 100 for non-surveying receives in real time a single image taken during a shooting flight from a drone, corrects the lens and tangential distortion (first correction) of the received image, and then orthogonalizes the image again. Correction (second correction) may be performed to generate an orthogonal image. Through this, the apparatus 100 for generating a real-time high-resolution orthogonal map for non-surveying may perform individual processing for each image, thereby significantly reducing processing time.

After generating the orthogonal image, the apparatus 100 for generating a real-time high-resolution orthogonal map for non-surveying may convert the generated orthogonal image into a tile map to display it by overlaying it on an online map, and store it in a storage linked to the TMS.

In converting the tile map of an orthogonal image, the real-time high-resolution orthogonal map generating device 100 for non-surveying performs a separate tiling process for each single orthogonal image, so in the case of overlapping GPS coordinates, the previously converted tile map information is overwritten. Due to (Overwrite), there may be a problem in that previous tile map information is deleted.

For example, when the previously converted tile map information is entirely filled with images, whereas the currently converted tile map information is only half full of images, the real-time high resolution orthogonal map generating apparatus 100 for non-surveying is relatively While deleting previous tile map information, which may be of good quality, there may be a problem of storing the currently converted tile map information in the storage.

To improve this problem, the device 100 for generating a real-time high resolution orthogonal map for non-surveying, when attempting to write tile map information in the same GPS coordinates, checks whether there is tile map information of the existing GPS coordinates, and occupies an image in the tile map. You can select and save the page with more ratio.

When an image is acquired by an unmanned aerial vehicle, a geometric error, which is a spatial distortion of the image, may occur due to various reasons such as the relative motion of the earth and the unmanned aerial vehicle, the sensor's relative motion, sensor characteristics, and limitations of device control. The image in which such a geometric error has occurred is in a state of being distorted not to match the actual spatial distribution of the ground, and this state of distortion can be corrected by geometric correction.

The apparatus 100 for generating a real-time high-resolution orthogonal map for non-surveying may correct an image based on orthogonal correction, unlike the geometric correction described above.

The orthogonal correction may refer to a process of removing the undulation displacement caused by the center projection and the displacement caused by the camera posture so as to have orthogonal projection characteristics like a map.

Orthogonal correction is a technique of correcting the position of an image by geometrically reconstructing the same environment as when the image was taken, taking into account all the causes of distortion of the image, and all points in the image have the same shape as viewed from the vertical direction.

Geometric errors can be caused by the Earth's rotation effect, Panoramic Distortion, Scan Time Skew, changes in the attitude of the UAV, and the curvature of the Earth.

In the Earth rotation effect, since the Earth is rotating, the area photographed by the sensor does not form a rectangular shape as shown in a normal image, and may actually show a square shape distorted in the east-west direction.

Due to the panoramic distortion, the IFOV angle (instantaneous field of view) of the unmanned aerial vehicle's mounted sensor is constant, so the surface area represented by one pixel is larger at the end of the line than in the vertical direction. Can occur.

Due to the distortion of the scan time, in the case of acquiring an image in line units such as Landsat MSSS or TM in the image acquisition process, it takes a certain time to scan a certain width, and during this time, the satellite continuously moves forward and tracks the track. ) Can cause distortion along the direction.

The type of posture change of the UAV can be classified into yaw, pitch, and roll, which can cause rotation of the image and deviations in the vertical and horizontal directions of the track.

By the curvature of the earth, it may cause severe distortion due to the curvature of the earth depending on the altitude. However, since UAVs have a low altitude and narrow viewing width, the effect of the Earth's curvature can be neglected.

In the geometric correction method, a model for the size and characteristics of the cause of geometric distortion is established, and mathematical modeling that establishes a correction equation with this model, and the position and ground coordinates of the pixels in the image using a ground control point (GC). Polynomial modeling, etc. to obtain a mathematical relationship between the two can be illustrated.

The real-time high-resolution orthogonal map generating apparatus 100 for non-surveying performs orthogonal correction, and for real-time high-speed processing, after analyzing the cause of the distortion due to the deviation error due to the change in the attitude of the unmanned aerial vehicle and the panoramic distortion. , Using this, a mathematical modeling technique that corrects distortion can be used to obtain an inverse transformation system that converts the distorted image to its original state.

Hereinafter, a step-by-step process for generating a real-time orthogonal map implemented by the real-time high-resolution orthogonal map generating apparatus 100 for non-surveying according to the present invention will be described.

First, the apparatus 100 for generating a real-time high-resolution orthogonal map for non-surveying may obtain correction data for lens distortion and tangential distortion by calibrating the drone camera before flying the drone.

The world we see with our eyes is three-dimensional, but this three-dimensional image turns into a two-dimensional image as it is captured with a camera. In this case, the geometric position of where the three-dimensional points are formed on the image may be determined by the position and direction of the camera at the time of taking the image. However, the actual image can be greatly affected by the mechanical parts inside the camera, such as the lens used, the distance between the lens and the image sensor, and the angle between the lens and the image sensor.

Therefore, when obtaining the position where the 3D points are projected onto the image or conversely restoring the 3D spatial coordinates from the image coordinates, accurate calculations can be performed only by removing such factors inside the camera. In addition, the process of obtaining the parameter value of such an internal factor may be referred to as camera calibration.

The camera image can be obtained by projecting points in a 3D space onto a 2D image plane, and in a pinhole camera model, this transformation relationship can be modeled by [Equation 1].

Here, (X,Y,Z) is the coordinates of the 3D point on the world coordinate system, [R|t] is the rotation and movement transformation matrix for converting the world coordinate system to the camera coordinate system, and A is the intrinsic camera May refer to a matrix.

Equationally, camera calibration is a transformation relationship between the 3D spatial coordinates of the camera coordinate system (Xc, Yc, Zc) and the 2D plane coordinates of the image coordinate system (X, Y), or a parameter that describes the transformation relationship. It may be a process of finding.

Accordingly, in [Equation 1], [R|t] may be referred to as an external camera parameter, and A may be referred to as an intrinsic parameter. In addition, a combination of A and [R|t] may be referred to as a camera matrix or a projection matrix.

The camera external parameters are parameters related to the geometrical relationship between the camera and the external space, such as the installation height and direction (pan, tilt) of the camera, and the internal parameters of the camera are internal to the camera such as the focal length, aspect ratio, and center point of the camera. It may mean a phosphorus parameter.

The camera internal parameters may include a focal length, a principal point, and a skew coefficient.

The focal length f may refer to a distance between a lens center and an image sensor (CCD, CMOS, etc.). The unit of the focal length f may be expressed as a pixel.

Since the pixels of the image correspond to the cells of the image sensor, the meaning that the focal length f is a pixel unit may mean that the focal length is expressed as a value relative to the cell size of the image sensor. For example, if the size of the image sensor cell is 0.1 mm and the focal length of the camera is f = 500 pixels, the distance from the center of the lens of the camera to the image sensor is 500 times the size of the image sensor cell. That is, it may mean 50 mm.

In the field of computer vision, the reason camera focal length is expressed in pixels rather than physical units (m, cm, mm, ...) is to facilitate geometric analysis in images by expressing the focal length in the same unit as image pixels. It is to do.

However, in the camera model, the focal length is not expressed as f as a single value, but is expressed by dividing it into fx and fy depending on the camera calibration. This is because the physical cell spacing of the image sensor is horizontal and vertical. This is to model that these can be different.

In this case, fx indicates how many times the focal length (the distance from the center of the lens to the image sensor) is the cell size (interval) in the horizontal direction, and fy indicates how the focal length is the cell size (interval) in the vertical direction. . Both units of fx and fy are pixels, and since there is no difference between horizontal cell spacing and vertical cell spacing in modern cameras, f = fx = fy can be set.

For reference, when calibration is performed with the same camera, if the image resolution is reduced to 1/2, the focal length of the calibration result is also reduced to 1/2. Although the actual physical focal length does not change, the focal length in the camera model is a relative concept, so if the resolution is changed, the physical size corresponding to one pixel changes, and thus the focal length also changes. For example, when the image resolution is lowered to 1/2, the 2 x 2 cells of the image sensor are combined to form one image pixel, so the physical size corresponding to one pixel is doubled. Therefore, the focal length should be 1/2.

2 is a diagram for explaining an example of a camera projection model.

In FIG. 2, the lens center (focus) of the camera model may correspond to a pinhole in the pinhole camera model. The pinhole camera model may be a model in which all light passes through a point (focus) in a straight line and is projected onto the image plane (sensor). This pinhole model greatly simplifies the geometric projection relationship between the 3D space and the 2D image plane.

A plane whose distance from the focal point is 1 (unit distance) is called a normalized image plane, and coordinates on this plane can be referred to as normalized image coordinates. The normalized image plane may be a virtual (imaginary) image plane that does not actually exist. When converting a point (Xc, Yc, Zc) on the camera coordinate system to the image coordinate system, first dividing Xc, Yc by Zc (distance from the camera focal point) is to convert it to the coordinates on this normalized image plane. By multiplying the focal length f, the image coordinate (pixel) in the desired image plane can be obtained.

However, since the pixel coordinate in the image is based on the upper left corner of the image, not the center of the image, the actual final image coordinate is a value obtained by adding (cx, cy) to the reference origin.

The main points cx and cy are the image coordinates (units are pixels) of the center of the camera lens, that is, the feet of the repairs lowered from the pinhole to the image sensor, and are different from the image center generally referred to. For example, if the lens and the image sensor are displaced horizontally due to an error in the camera assembly process, the main point and the image center will have different values.

In image geometry, the principal point is much more important than a simple image center, and all geometrical interpretations of the image are performed using this main point.

The asymmetry coefficient may indicate the degree of inclination of the y-axis of the cell array of the image sensor. However, cameras manufactured in recent years rarely have such skew errors, so the asymmetry coefficient is usually not considered in the camera model.

These camera internal parameters can be calculated relatively easily by using a publicly available calibration tool.

The calibration tool used in the present invention is GML C++ Camera Calibration Toolbox.

3A is a diagram illustrating a GML camera calibration tool, and FIG. 3B is a diagram illustrating an execution result of a GML camera calibration tool.

The basic usage of the GML camera calibration tool is to take a paper printed with a calibration pattern (chess board) as shown in FIG. 3A, photograph it with a camera at various positions and angles, and then save it as an image, and calibrate the stored image with a tool. Is doing.

A feature of the GML camera calibration tool is that it is possible to perform calibration by using multiple calibration patterns as shown in FIG. 3A at the same time, and when two or more patterns are used, the calibration effect may be further improved. Depending on the embodiment, it is also possible to use only one pattern.

The result of performing the calibration using the GML camera calibration tool is shown in FIG. 3B.

As shown in FIG. 3B, the calibration result is output in the form of an estimated value ± an estimated error (3*sigma), and the meaning of each item is as follows.

-Focal length: fx = 3497.576, fy = 3501.038

-Principal point: cx = 1058.429, cy = 797.136

-Distortion: k1 = -0.041196, k2 = -0.203893, p1 = 0.006114, p2 = 0.002318 (k1,k2: radial distortion coefficient, p1,p2: tangential distortion coefficient)

After obtaining the correction data, the apparatus 100 for generating a real-time high-resolution orthogonal map for non-surveying may perform surface photographing by flying a low-altitude drone (unmanned aerial vehicle) of 100 m or less. In this case, the apparatus 100 for generating a real-time high-resolution orthogonal map for non-surveying may capture a captured image in real time and receive an image through an LTE network or the like.

In addition, the apparatus 100 for generating a real-time high-resolution orthogonal map for non-surveying may perform correction processing for lens distortion and tangential distortion on the image received in real time, based on correction data obtained through calibration previously.

In performing the correction process, the apparatus 100 for generating real-time high-resolution orthogonal maps for non-surveying may use openCV, which is a general-purpose library.

4 is a diagram for explaining an example of correction processing for lens distortion and tangential distortion according to the present invention.

In the correction process for lens distortion and tangential distortion, focal length (Focal length fx, fy), principal point (Principal point cx, cy), and distortion coefficient (Distortion k1, k2, p1, p2) are used, and these are summarized as follows. Can be.

Camera Matrix: 3 x 3 Matrix => {fx, 0, cx, 0, fy, cy, 0, 0, 1} => {3497.576, 0, 1058.429, 0, 3501.038, 797.136, 0, 0, 1}

Distortion Matrix: 1 x 4Matrix => {k1, k2, p1, p2} => {-0.041196, -0.203893, 0.006114, 0.002318}

Correction processing for lens distortion and tangential distortion requires a Camera Matrix and a Distortion Matrix, and by applying these matrices to the cvUndistort2() function, which is a distortion correction function of OpenCV, as shown in the right drawing of FIG. You can get the corrected image.

Thereafter, the apparatus 100 for generating a real-time high-resolution orthogonal map for non-surveying may perform orthogonal correction processing on the generated image based on the gimbal attitude information (RAW, PITCH, and ROLL) stored in the image.

Perspective transformation is the same principle that our eyes perceive external objects, so this perspective transformation is not separately used in computer graphics to describe objects, but is used as part of the projective projection. I can. In other words, our eyes are applied to the perspective of showing small things that are far away and those that are close to them, and the image passes through the eye and forms an image on the retina, which has a similar feature to a camera (pinhole camera).

Perspective transformation can be expressed as [Equation 2] using matrix multiplication.

The two forms of matrix multiplication in [Equation 2] are due to the difference between placing the transformation matrix on the left and placing the transformation matrix on the right.

When the position of the transformation matrix is arranged to the left of the point P, the point P is expressed in the form of a row, whereas when the position of the transformation matrix is arranged to the right of the point P, the point P can be expressed in the form of a column. In general, the transformation matrix is located on the right, and the point P can be expressed as a column, which makes it relatively easy to overlap the matrix.

FIG. 5A is a view for explaining an example of perspective projection, and FIG. 5B is a diagram showing the relationship of FIG. 5A.

Projecting a three-dimensional object onto a two-dimensional image plane can be important in three-dimensional computer graphics.

In Fig. 5A, perspective projection is illustrated, the center of the projection in Fig. 5A becomes an eyepoint, and the center projection line passes vertically through the image plane.

In Fig. 5B, the relationship between the point P(x,y,z) and the point P'(x',y') of the image plane is explained using a similar triangle.

The relation between (0,0,0) corresponding to the eye point, the arbitrary point P(x,y,z) of the object, and the point P'(x',y') of the image plane is expressed as [Equation 3]. I can explain.

In [Equation 3], dividing by z is called a perspective divide. Through the perspective divide, as the distance increases, the z value increases, so that the distance from the origin becomes closer in the projection plane. [Equation 3] may be a method of mathematically representing perspective.

If the image plane is at the position of z = D as in FIG. 5B, the determinant as in [Equation 4] can express the projection.

Since the projection equation of [Equation 4] is z'= D, there may be a problem that information about the depth expressed by z disappears.

As a solution to this, a factor that does not affect x', y'but affects z'can be introduced. That is, by replacing the Euclit space represented by z ∈ [1, ∞] with the projection space represented by z'∈ [0, D ], a coordinate system transformation equation such as [Equation 5] can be completed.

The matrix expressed in [Equation 5] is a matrix for perspective transformation.

6 is a diagram illustrating an example of an observation frustum in order to explain the orthogonal transformation.

As shown in FIG. 6, the viewing frustum consists of six faces. This observation frustum can be defined as the range in which the human eye can see the object within 6 planes, consisting of the position of the viewpoint, the space that can be projected on the image plane, and the sense of depth that can be expressed in a three-dimensional object. If the object deviates from the observation frustum, it is clipped and cannot be displayed.

In the Euclidean space, the side closer to the field of view has a small image plane, and the far side has a wide image plane, which may be another expression of perspective. Assuming that the size of the image plane is '2*Sx by 2*Sy' and the z at the near plane is D and the z at the far plane is F, the size of the image plane can be changed using the viewing angle. . If this is expressed as a trigonometric function, it is as shown in [Equation 6].

Here, fov may be a field of view, that is, a field of view.

For simplicity of calculation, if the near face is converted to a face with z'= 0 and the far face is converted to a face with z'= 1, the point on the near face is a point (0,0,D), and the far face The point at can be expressed as a point (0,0,F). Each perspective transformation for these points can be expressed by [Equation 7] and [Equation 8].

In [Equation 7] and [Equation 8], [P] may be a perspective matrix.

Here, if the z'value is substituted with an appropriate factor for the third column to become the desired 0 and 1, it can be expressed as [Equation 9].

[Equation 9] may be a perspective transformation (projection transformation) matrix in consideration of the observation frustum.

In [Equation 9], if F is considered to be very large compared to D, since D/F becomes 0, the projection transformation matrix can be finally expressed as [Equation 10].

By processing the perspective transformation using the projection transformation matrix of [Equation 10], the real-time high-resolution orthogonal map generation apparatus 100 for non-surveying may perform orthogonal correction.

In addition, the real-time high-resolution orthogonal map generating device 100 for non-surveying corrects the GPS coordinate information that is the center of the generated orthogonal image by reflecting the camera angle of view, the PITCH rotation angle, and the altitude value, and an image based on the corrected GPS coordinates. It is possible to derive GPS coordinates of the upper left and lower right corners of and convert them into TIFF files and insert the derived coordinate information (GEO-Reference) information into the TIFF image.

7 is a diagram for explaining derivation of coordinate information according to the present invention.

7A is a view for explaining a difference in GPS coordinate points according to a gimbal pitch angle in an unmanned aerial vehicle, and FIG. 7B is a view for explaining FIG. 7A in more detail in a two-dimensional plane.

In FIGS. 7A and 7B, a calculation equation for deriving the corrected GPS coordinate point A'from the real GPS coordinate point A may be expressed as [Equation 11].

The apparatus 100 for generating a real-time high-resolution orthogonal map for non-surveying may also correct altitude information that changes as GPS coordinate points are corrected. The correction of the altitude information may be performed using a trigonometric function of'h' = sin (pitch rotation angle) x h (measured altitude).

In order to display a JPG image on a map, the JPG image must be converted into a Georeferenced Tiff file containing georeference information. Georeferenced Tiff files can be converted by setting the upper left GPS coordinates and the lower right GPS coordinates of the JPG image.

The real-time high-resolution orthogonal map generating device 100 for non-surveying uses A'(corrected GPS coordinates) and the Width, Height, and Yaw rotation angle of the JPG image to calculate GPS coordinate information of the upper left and lower right of the JPG image. I can.

The real-time high-resolution orthogonal map generator 100 for non-surveying obtains the distance D (D = tan (camera angle of view / 2) x h'(corrected elevation)) from A'(corrected GPS coordinates) to the upper left corner, and The coordinate movement distance to (m, n) is obtained as in [Equation 12].

Through the calculation in [Equation 12], the apparatus 100 for generating a real-time high-resolution orthogonal map for non-surveying can obtain the upper left coordinate B(x, y) by adding the derived coordinate movement amounts m and n to the A'coordinate. .

Upper left coordinate B(x, y) = A'+ (m, n) = (XA + m, YA + n)

In addition, the real-time high-resolution orthogonal map generation apparatus 100 for non-surveying calculates using the rotation matrix of [Equation 13] that reflects the Yaw rotation angle to the upper left coordinate B(x, y) of the obtained image, As shown in Equation 14], the final corrected upper left B'(x', y') can be obtained.

Also for the lower left coordinate, the apparatus 100 for generating a real-time high resolution orthogonal map for non-surveying may be calculated and derived in the same manner as described above.

8 is a diagram showing movement of GPS coordinates before and after image correction according to the present invention.

As shown in FIG. 8, the real-time high-resolution orthogonal map generating apparatus 100 for non-surveying rotates the upper left and lower left GPS coordinates of the image before correction by reflecting the Yaw rotation angle in consideration of the direction of drone photography. Can be corrected. At this time, the GPS coordinate point at the center of the image is also corrected and moved in consideration of the pitch rotation angle.

In addition, the apparatus 100 for generating a real-time high-resolution orthogonal map for non-surveying may convert the geo-referenced orthogonal image into a standardized tile map for each zoom level through a general-purpose library and store it in a designated path. Conversion into a tile map for each zoom level corresponds to accumulation in a general paper map, and can indicate the size of the map based on pixels compared to the actual distance. The apparatus 100 for generating real-time high-resolution orthogonal maps for non-surveying may split the map into a tiled map having a size of 256x256 for each map zoom level (accumulation), and display it on a GCS location on the map during a map service.

Tile Map Service may be a service that performs a map service very quickly by using a storage that collects static images. The map image storage may be a directory structure in which map regions are stored as Tile images for each specific scale. Therefore, retrieving images from a directory structure is much shorter than creating a map directly from a database according to user needs. By using such a Tile Map Service, when a user creates a complex map, it is possible to drastically reduce the map creation time and eliminate all efforts to improve performance.

Since each image is individually transformed into a tile map, if there is a tile map with the same GPS coordinate information that was previously saved when saving in a designated path after conversion, the file name (GPS coordinate) is the same, causing a problem of being overwritten. If the previously saved tile map is of higher quality, overwriting the tile map causes a problem that the quality of the tile map deteriorates further. Therefore, the real-time high resolution orthogonal map generator 100 for non-surveying uses a designated path to avoid this problem. It is determined whether or not a tile map with the same GPS coordinates exists before saving in the file, and if there is, the one with the larger area (the one with a larger amount of data) can be selected and saved.

9 is a diagram illustrating an example of processing according to the present invention when overlapping tiles are stored in GPS coordinates.

9 illustrates a case in which tile map images A and B exist and three overlapping tiles exist corresponding to GPS coordinates.

In the case of the redundant tile map A, the apparatus 100 for generating a real-time high resolution orthogonal map for non-surveying may prioritize the tile map image A having a wider map area to store the tile map.

Similarly, the real-time high resolution orthogonal map generating apparatus 100 for non-surveying stores the tile map of the tile map image B in the case of the overlapping tile map B, and stores the tile map of the tile map image A in the case of the overlapping tile map C. I can.

Hereinafter, in FIG. 10, a work flow of the apparatus 100 for generating a real-time high-resolution orthogonal map for non-surveying using an unmanned aerial vehicle according to embodiments of the present invention will be described in detail.

10 is a flowchart illustrating a method of generating a real-time high resolution orthogonal map for non-surveying using an unmanned aerial vehicle according to an embodiment of the present invention.

The method of generating a real-time high-resolution orthogonal map for non-surveying using an unmanned aerial vehicle according to the present embodiment may be performed by the apparatus 100 for generating a real-time high-resolution orthogonal map for non-surveying using an unmanned aerial vehicle.

First, the apparatus 100 for generating a real-time high-resolution orthogonal map for non-surveying obtains correction data for a camera by performing calibration (1010). Step 10101 is mounted on an unmanned aerial vehicle such as a drone, and performs a correction operation on a camera that generates an image by performing a photographing operation while the unmanned aerial vehicle is photographing and flying, and the current state of the camera (lens distortion, tangential distortion) It may be a process of obtaining correction data for (including degree).

The calibration measures the focal length or aperture scale of the camera lens according to the change in the distance between the camera and the subject or the incident light, and displays it in advance to determine in advance what state the image generated by the camera is generated in. It can be a task.

In other words, the apparatus 100 for generating real-time high-resolution orthogonal maps for non-surveying may recognize a degree of distortion due to the current state of the camera through calibration, and obtain a result value as the correction data.

In addition, the real-time high-resolution orthogonal map generating apparatus 100 for non-surveying is equipped with the camera and, as an image photographed by the camera is received from an unmanned aerial vehicle in flight, the lens and the First-order correction for correcting tangential distortion is performed (1020). Step 1020 may be a process of correcting the image by excluding distortion of the camera itself by performing a correction operation for lines and surfaces in consideration of the previously acquired correction data on the image received from the unmanned aerial vehicle.

In addition, the real-time high-resolution orthogonal map generating apparatus 100 for non-surveying performs secondary correction of orthogonal correction on the first corrected image to generate an orthogonal image (1030). Step 1030 may be a process of generating an orthogonal image as viewed vertically from the sky by performing orthogonal correction again on the lens and the tangential distortion-corrected image.

The apparatus 100 for generating real-time high-resolution orthogonal maps for non-surveying may receive an image captured from the unmanned aerial vehicle in real time through a communication network in conjunction with the image being captured by the camera. That is, the apparatus 100 for generating a real-time high-resolution orthogonal map for non-surveying may serve to continuously perform the above-described first and second corrections by receiving the image each time a camera generates an image.

Here, the communication network refers to a network that transmits an image generated by connecting the device 100 for generating a real-time high resolution orthogonal map for non-surveying and an unmanned aerial vehicle, and may be illustrated as 4G and 5G LTE.

In the second correction, the apparatus 100 for generating a real-time high-resolution orthogonal map for non-surveying may generate the orthogonal image by reflecting the gimbal position information of the image and processing orthogonal correction on the first corrected image. have. The gimbal posture information includes yaw information about the z-axis rotation of the first corrected image, pitch information about the x-axis rotation of the first corrected image, and the first corrected image. It may include roll information about the y-axis rotation.

That is, the real-time high-resolution orthogonal map generating apparatus 100 for non-surveying may produce a three-dimensional orthogonal image through rotation correction for each axis in a three-dimensional space with respect to the image corrected by the first correction. .

In another embodiment, the apparatus 100 for generating a real-time high-resolution orthogonal map for non-surveying corrects the center coordinates of the orthogonal image by reflecting at least one of a field of view, a pitch rotation angle, and an altitude value of the camera, and the corrected The GPS coordinates of the upper left corner and the lower right corner of the image are derived based on the center coordinates, converted into a TIFF file, and inserted into the orthogonal image together with GEO-Reference information. That is, the real-time high-resolution orthogonal map generating apparatus 100 for non-surveying includes coordinate information of a reference object on the ground that can be an indicator in the generated orthogonal image, so that an image is captured in relation to the orthogonal image generated according to the correction. It can support to accurately recognize the location of the ground.

The generated orthogonal image can be visualized by being expressed on a map.

To this end, the real-time high-resolution orthogonal map generating device 100 for non-surveying may further include a real-time high-resolution orthogonal map generating device 100 for non-surveying that maps orthogonal images to an online map.

The real-time high-resolution orthogonal map generating apparatus 100 for non-surveying converts the orthogonal image into a first tile map and stores it in a storage associated with a tile map service (TMS), so that when a service request by the TMS is requested, the first Tile maps can be overlaid on an online map. That is, the real-time high-resolution orthogonal map generating apparatus 100 for non-surveying generates an orthogonal image generated for each image as a tile set of a tile sheet texture (2D image), and the tile size of the desired tile, border maring, and tile After adjusting the tileset according to spacing (Per Tile-Spacing), etc., it may be converted into a first tile map. In this case, the tile map can set the size of the entire map and the tile size to be used by setting the number of horizontal and vertical tiles (Map Width and Map Height) and tile size (Tile Width and Tile Height) of the online map according to the tileset to be used.

In addition, the apparatus 100 for generating a real-time high-resolution orthogonal map for non-surveying may store the first tile map in a storage associated with the TMS by matching the GPS coordinates in which the image is captured. In this case, when a pre-registered second tile map corresponding to the GPS coordinates is stored in the storage, the real-time high resolution orthogonal map generating apparatus 100 for non-surveying is an image between the first tile map and the second tile map. It is possible to compare the occupancy ratio and store a tile map having a relatively large image occupancy ratio in the storage.

That is, the real-time high-resolution orthogonal map generating apparatus 100 for non-surveying allows only one relatively high-quality tile map to be stored in correspondence with one GPS coordinate, so that the tile map provided in the online map service is not duplicated and stored. It can save storage resources.

According to an embodiment of the present invention, in conjunction with photographing an image from a camera mounted on an unmanned aerial vehicle, by receiving an image taken from an unmanned aerial vehicle and converting the generated orthogonal image into a tile map in real time, It is possible to provide a method for generating a real-time high-resolution orthogonal map for non-surveying using an unmanned aerial vehicle and an apparatus for generating a real-time high-resolution orthogonal map for non-surveying.

In addition, according to an embodiment of the present invention, a precise orthogonal image can be quickly generated by continuously performing lens and tangential distortion correction and orthogonal correction through the gimbal posture information of an image received from an unmanned aerial vehicle. have.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of the program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

The software may include a computer program, code, instructions, or a combination of one or more of these, configuring the processing unit to behave as desired or processed independently or collectively. You can command the device. Software and/or data may be interpreted by a processing device or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. , Or may be permanently or temporarily embodyed in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

As described above, although the embodiments have been described by the limited drawings, a person of ordinary skill in the art can apply various technical modifications and variations based on the above. For example, the described techniques are performed in a different order from the described method, and/or components such as a system, structure, device, circuit, etc. described are combined or combined in a form different from the described method, or other components Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved.

Therefore, other implementations, other embodiments and claims and equivalents fall within the scope of the following claims.

100: Real-time high-resolution orthogonal map generation device for non-surveying using an unmanned aerial vehicle
110: acquisition unit 120: processing unit
130: mapping unit 140: receiving unit

Claims (14)

Performing calibration to obtain correction data for the camera;
Performing primary correction of correcting lens and tangential distortion on the image by using the correction data as the image captured by the camera is received from the unmanned aerial vehicle in flight with the camera mounted thereon;
Generating an orthogonal image by performing a secondary correction for orthogonal correction on the first corrected image;
Converting the orthogonal image into a tile map image and storing it in a storage associated with a tile map service (TMS); And
Upon requesting the service by the TMS, causing the tile map image to be overlaid on an online map
Including,
The storing step,
When the tile map image A converted from the orthogonal image and the tile map image B converted from another orthogonal image are at least partially overlapped when stored in correspondence with the GPS coordinates, and a duplicate tile map occurs,
Comparing a map area corresponding to the tile map image A and a map area corresponding to the tile map image B with respect to the overlapping tile map; And
As a result of the comparison, if the map area corresponding to the tile map image A is wider, storing the duplicate tile map as a tile map belonging to the tile map image A in the storage
Real-time high resolution orthogonal map generation method for non-surveying using an unmanned aerial vehicle comprising a. The method of claim 1,
The step of generating the orthogonal image,
Reflecting the gimbal posture information of the image, processing orthogonal correction on the first corrected image, and generating the orthogonal image
Real-time high-resolution orthogonal map generation method for non-surveying using an unmanned aerial vehicle comprising a. The method of claim 2,
The above gimbal position information is:
Yaw information related to the z-axis rotation of the first corrected image, pitch information related to the x-axis rotation of the first corrected image, and a roll related to the y-axis rotation of the first corrected image (roll) containing information
Real-time high resolution orthogonal map generation method for non-surveying using unmanned aerial vehicles. The method of claim 1,
The real-time high-resolution orthogonal map generation method for non-surveying,
Converting the orthogonal image into a first tile map and storing it in the storage so that the first tile map is overlaid on the online map when the service is requested by the TMS
Real-time high-resolution orthogonal map generation method for non-surveying using an unmanned aerial vehicle further comprising a. The method of claim 4,
The real-time high-resolution orthogonal map generation method for non-surveying,
When the first tile map is stored in correspondence with the GPS coordinates in which the image is captured, and a pre-stored second tile map corresponding to the GPS coordinates is stored in the storage, the first tile map and the second Comparing the image occupancy ratio between tile maps and storing a tile map having a relatively large image occupancy ratio in the storage
Real-time high-resolution orthogonal map generation method for non-surveying using an unmanned aerial vehicle further comprising a. The method of claim 1,
The real-time high-resolution orthogonal map generation method for non-surveying,
Correcting the center coordinate of the orthogonal image by reflecting at least one of an angle of view, a pitch rotation angle, and an elevation value of the camera; And
Deriving the GPS coordinates of the upper left corner and the lower right corner of the image based on the corrected center coordinates, converting it into a TIFF file, and inserting it into the orthogonal image together with GEO-Reference information.
Real-time high-resolution orthogonal map generation method for non-surveying using an unmanned aerial vehicle further comprising a. The method of claim 1,
The real-time high-resolution orthogonal map generation method for non-surveying,
Receiving an image captured from the unmanned aerial vehicle in real time through a communication network in connection with the image being captured by the camera
Real-time high-resolution orthogonal map generation method for non-surveying using an unmanned aerial vehicle further comprising a. An acquisition unit that performs calibration to obtain correction data for the camera;
As an image photographed by the camera is received from an unmanned aerial vehicle in flight with the camera mounted, a first correction for correcting lens and tangential distortion is performed on the image using the correction data, and the first correction A processing unit for generating an orthogonal image by performing secondary correction of orthogonal correction on the image; And
A mapping unit that converts the orthogonal image into a tile map image, stores it in a storage associated with TMS (Tile Map Service), and causes the tile map image to be overlaid on an online map when a service request by the TMS is requested.
Including,
The mapping unit,
When the tile map image A converted from the orthogonal image and the tile map image B converted from another orthogonal image are at least partially overlapped when stored in correspondence with the GPS coordinates, and a duplicate tile map occurs,
For the overlapping tile map, a map area corresponding to the tile map image A and a map area corresponding to the tile map image B are compared,
As a result of the comparison, if the map area corresponding to the tile map image A is wider, storing the duplicate tile map as a tile map belonging to the tile map image A in the storage
Real-time high-resolution orthogonal map generation device for non-surveying using an unmanned aerial vehicle. The method of claim 8,
The processing unit,
Reflecting the gimbal posture information of the image, processing orthogonal correction on the first corrected image to generate the orthogonal image
Real-time high-resolution orthogonal map generation device for non-surveying using an unmanned aerial vehicle. The method of claim 9,
The above gimbal position information is:
Yaw information related to the z-axis rotation of the first corrected image, pitch information related to the x-axis rotation of the first corrected image, and a roll related to the y-axis rotation of the first corrected image (roll) containing information
Real-time high-resolution orthogonal map generation device for non-surveying using an unmanned aerial vehicle. The method of claim 8,
The mapping unit,
By converting the orthogonal image into a first tile map and storing it in the storage, the first tile map is overlaid on the online map when the service is requested by the TMS.
Real-time high-resolution orthogonal map generation device for non-surveying using an unmanned aerial vehicle. The method of claim 11,
The mapping unit,
When the first tile map is stored in correspondence with the GPS coordinates in which the image is captured, and a pre-stored second tile map corresponding to the GPS coordinates is stored in the storage, the first tile map and the second Comparing the image occupancy ratio between tile maps, and storing a tile map having a relatively large image occupancy ratio in the storage
Real-time high-resolution orthogonal map generation device for non-surveying using an unmanned aerial vehicle. The method of claim 8,
The processing unit,
Correcting the center coordinate of the orthogonal image by reflecting at least one of the angle of view, the pitch rotation angle, and the altitude value of the camera,
Based on the corrected center coordinates, the GPS coordinates of the upper left corner and the lower right corner of the image are derived, converted into a TIFF file, and inserted into the orthogonal image together with the coordinate information (GEO-Reference) information.
Real-time high-resolution orthogonal map generation device for non-surveying using an unmanned aerial vehicle. The method of claim 8,
Receiving unit for receiving an image captured from the unmanned aerial vehicle in real time through a communication network in conjunction with the image being photographed by the camera
Real-time high-resolution orthogonal map generation device for non-surveying using an unmanned aerial vehicle further comprising a.

Images