KR102151250B1 - Device and method for deriving object coordinate - Google Patents

Abstract

The object coordinate derivation apparatus includes an image receiving unit that receives a plurality of images from a plurality of cameras, a camera parameter derivation unit that derives camera parameters corresponding to the plurality of cameras, and a reference among a plurality of images using the derived camera parameters and a plurality of images. A depth image generator for generating a depth image for an image, a spatial coordinate deriving unit for deriving spatial coordinates for an object included in the depth image, and acoustic coordinates for sound generated from an object based on the spatial coordinates for the derived object It may include an acoustic coordinate derivation unit for deriving.

Description

Device and method for deriving object coordinates {DEVICE AND METHOD FOR DERIVING OBJECT COORDINATE}

The present invention relates to an apparatus and method for deriving object coordinates.

Three-dimensional sound (three-dimensional sound) refers to a sound that has a three-dimensional sense of realism to which a sense of direction, distance, and space are applied.

Recently, 　3D sound technology is developing into an interactive 3D sound technology that can reflect such interactions because the relative position between the sound source and the listener changes when the sound source or the listener moves.

Such 3D sound technology can be implemented more conveniently as the latest sound middleware is developed. However, in order to implement a 3D sound, 3D coordinates for each of the sound source and the listener must be manually set. In addition, it is very difficult to set 3D coordinates in the case of a dynamic sound source (or listener) that changes irregularly in real time.

Korean Patent Application Publication No. 2018-0018464 (published on February 21, 2018)

The present invention is to solve the problems of the prior art described above, and it is intended to generate a depth image for a reference image using camera parameters corresponding to a plurality of cameras and a plurality of images received from a plurality of cameras. In addition, the present invention is to derive acoustic coordinates for sounds generated from objects based on spatial coordinates for objects included in the depth image. However, the technical problem to be achieved by the present embodiment is not limited to the technical problems as described above, and other technical problems may exist.

As a technical means for achieving the above technical problem, the object coordinate derivation apparatus according to the first aspect of the present invention includes an image receiving unit for receiving a plurality of images from a plurality of cameras; A camera parameter derivation unit for deriving camera parameters corresponding to the plurality of cameras; A depth image generator for generating a depth image for a reference image among the plurality of images by using the derived camera parameter and the plurality of images; A spatial coordinate derivation unit for deriving spatial coordinates for an object included in the depth image; It may include an acoustic coordinate derivation unit that derives acoustic coordinates for sound generated from the object based on the spatial coordinates of the derived object.

A method of deriving coordinates of an object in an apparatus for deriving coordinates of an object according to a second aspect of the present invention includes: receiving a plurality of images from a plurality of cameras; Deriving camera parameters corresponding to the plurality of cameras; Generating a depth image for a reference image among the plurality of images by using the derived camera parameter and the plurality of images; Deriving spatial coordinates for the object included in the depth image; And deriving acoustic coordinates for sound generated from the object based on the derived spatial coordinates for the object.

The above-described problem solving means are merely exemplary and should not be construed as limiting the present invention. In addition to the above-described exemplary embodiments, there may be additional embodiments described in the drawings and detailed description of the invention.

According to any one of the above-described problem solving means of the present invention, the present invention can generate a depth image for a reference image using camera parameters corresponding to a plurality of cameras and a plurality of images received from the plurality of cameras. In addition, the present invention may derive acoustic coordinates for sounds generated from objects based on spatial coordinates for objects included in the depth image. Through this, the present invention can provide 3D media content in which a multi-view image and a 3D sound are combined. In addition, the present invention can solve a conventional problem (a problem in which coordinates must be acquired manually) by linking information of a depth image to input information (acoustic coordinates) of a stereophonic sound. In addition, the present invention can solve the problem of setting coordinates of a dynamic object that changes irregularly in the conventional stereoscopic sound expression. In addition, according to the present invention, the conventional manual coordinate acquisition and expression process is automated, so that the working time and manpower for the coordinate acquisition can be reduced, and more precise results can be obtained.

1 is a block diagram of an apparatus for deriving object coordinates according to an embodiment of the present invention.
2A to 2D are diagrams for explaining a method of deriving object coordinates according to an embodiment of the present invention.
3 is a flowchart illustrating a method of deriving object coordinates according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement the present invention. However, the present invention may be implemented in various different forms and is not limited to the embodiments described herein. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and similar reference numerals are assigned to similar parts throughout the specification.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only "directly connected" but also "electrically connected" with another element interposed therebetween. . In addition, when a part "includes" a certain component, it means that other components may be further included rather than excluding other components unless specifically stated to the contrary.

In the present specification, the term "unit" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Further, one unit may be realized using two or more hardware, or two or more units may be realized using one hardware.

In the present specification, some of the operations or functions described as being performed by the terminal or device may be performed instead by a server connected to the terminal or device. Likewise, some of the operations or functions described as being performed by the server may also be performed by a terminal or device connected to the server.

Hereinafter, with reference to the accompanying configuration diagram or processing flow chart, specific details for the implementation of the present invention will be described.

1 is a block diagram of an object coordinate derivation apparatus 10 according to an embodiment of the present invention.

Referring to FIG. 1, the object coordinate derivation apparatus 10 includes an image receiving unit 100, a camera parameter deriving unit 110, a depth image generating unit 120, a spatial coordinate deriving unit 130, and an acoustic coordinate deriving unit 140. ), a listener coordinate setting unit 150 and a listener direction vector setting unit 160. However, the object coordinate derivation apparatus 10 shown in FIG. 1 is only an example of implementation of the present invention, and various modifications are possible based on the elements shown in FIG. 1. Hereinafter, FIG. 1 will be described together with FIGS. 2A to 2D.

The image receiving unit 100 may receive a plurality of images from a plurality of cameras. For example, referring to FIG. 2A, the image receiving unit 100 may receive a plurality of images captured by each camera from a plurality of cameras arranged toward the object in an inward direction around the object. Here, each of the plurality of cameras photographs an object in a state in which a distance between the cameras and a photographing angle are set equally to a preset value.

The camera parameter derivation unit 110 may derive camera parameters corresponding to a plurality of cameras. Here, the camera parameters may include camera internal parameters and camera external parameters. The camera internal parameter and the camera external parameter are values necessary to generate a depth image and represent values representing internal and external conditions of the camera.

The camera internal parameters may include a focal length of a camera lens, a center point of a camera lens, an asymmetry coefficient for an image sensor, and a distortion coefficient for a camera lens. Here, the focal length (f, focal length) of the camera lens means the distance between the center of the camera lens and the image sensor installed in the camera, and the central point (c, principal point) of the camera lens is the distance from the pinhole of the camera to the image sensor. It means the feet of the repair. The skew coefficient (α, skew coefficient) for the image sensor means the degree of inclination of the y-axis for the cell arrangement of the image sensor, and the distortion coefficient (p) for the camera lens is the degree of distortion of the camera lens. Is shown.

For example, referring to FIG. 2B, the relationship between the first point (X, Y, Z) 201 specified in spatial coordinates and the image coordinates (x, y) 203 when the first point is projected into an image is It can be derived from the camera internal parameter 205 and the camera external parameter 207.

The external parameters of the camera may include a rotation transformation matrix and a movement transformation matrix between camera coordinates and spatial coordinates corresponding to the camera position.

The depth image generator 120 may generate a depth image for a reference image among a plurality of images by using camera parameters corresponding to the plurality of cameras and a plurality of images.

For example, referring to FIG. 2C, the camera parameter derivation unit 110 generates a first viewpoint image (left viewpoint image), a second viewpoint image (center viewpoint image, 209), and a third viewpoint image (right viewpoint image). For a plurality of included images, camera internal and external parameters of the first viewpoint image, camera internal and external parameters of the second viewpoint image, and camera internal and external parameters of the third viewpoint image may be derived.

The depth image generator 120 sets the second viewpoint image 209 among the plurality of images as a reference image, sets the remaining first viewpoint images and the third viewpoint images as reference images, and then inside the camera of each viewpoint image And inputting external parameters and a plurality of images to the depth image generation program 211 to perform pre-processing on the plurality of images.

The depth image generator 120 may perform image segmentation by dividing an image for each object with respect to an object included in each of the plurality of images.

The depth image generator 120 may search for a block image having an arbitrary size, search for the same block between two images, perform block matching, and generate a depth image 213 for the reference image by applying a graph cut. have.

The depth image generator 120 may measure depth information of the depth image 213 from the generated depth image 213.

For a moment, an image coordinate system, a camera coordinate system, and a spatial coordinate system will be described with reference to FIG. 2D.

Referring to FIG. 2D, the image coordinate system 215 refers to a coordinate system for a 2D image displayed through a display device. In the image coordinate system 215, when the upper left corner of the image is an origin, the right direction is the x-axis increasing direction, and the lower direction is the y-axis increasing direction. In this case, the unit of the image coordinate system 215 is a pixel unit.

The camera coordinate system 217 is a coordinate system with the camera as a reference point, and when the focus of the camera (that is, the center of the camera lens) is the origin, the front optical axis direction of the camera is the Z axis, the downward direction of the camera is the Y axis, and the right direction is It can be set on the X axis. In this case, the unit of the camera coordinates may be set in meters or centimeters in the same way as the spatial coordinates.

The spatial coordinate system 219 is a coordinate system for a three-dimensional space, and is a coordinate system used as a reference when expressing the position of an object in space. The spatial coordinate 219 is a coordinate system that the user can arbitrarily set and use. For example, if one corner of the user's bedroom is taken as the origin, one wall direction is the X axis, the other wall direction is the Y axis, and the ceiling is viewed. You can set the viewing direction on the Z axis. In this case, the unit of the spatial coordinate may be set to meters or centimeters.

Returning to FIG. 1 again, the spatial coordinate derivation unit 130 may derive the spatial coordinates for the object included in the depth image.

The spatial coordinate derivation unit 130 may extract an object from the depth image and calculate a center of gravity of the extracted object. For example, the spatial coordinate derivation unit 130 may separate an object and a background by applying image binarization to a plurality of frames of a depth image. Thereafter, the spatial coordinate deriving unit 130 may calculate the center of gravity of the object using the number of all pixels of the separated object.

When a plurality of images are displayed through the display, the spatial coordinate derivation unit 130 may derive image coordinates in which the object in the image displayed on the display is located based on the calculated center of gravity of the object. For example, referring to FIG. 2D, the spatial coordinate derivation unit 130 is an image coordinate (P = (x, y)) where an object is located on the image coordinate system 215 based on the center of gravity of the object. Can be obtained.

The spatial coordinate derivation unit 130 may calculate a depth pixel value of the object with respect to the image coordinate of the object derived based on the depth image.

)을 계산한 후, [수학식 1]에 객체의 깊이 화소 값(

) 및 깊이 영상의 깊이 값(

,

)을 대입하여 카메라 좌표 중 객체에 대한 좌표값(

)(223)을 계산할 수 있다. The spatial coordinate derivation unit 130 may calculate a coordinate value for the object on the camera coordinates based on depth information on the depth pixel value of the object and the camera coordinate derived from the depth image. For example, referring to FIG. 2D, the spatial coordinate derivation unit 130 determines the depth pixel value of the object on the image coordinate (P = (x, y)) 221 where the object is located.

) Is calculated, and the depth pixel value of the object (

) And the depth value of the depth image (

,

) By substituting the coordinate value for the object among the camera coordinates (

) 223 can be calculated.

[Equation 1]

)(223)을 [수학식 2]에 대입하여 객체에 대한 카메라 좌표(

=(

))를 도출할 수 있다. The spatial coordinate derivation unit 130 may derive camera coordinates for the object based on image coordinates of the object, camera internal parameters, and coordinate values for the object on the camera coordinates. For example, referring to FIG. 2D, the spatial coordinate derivation unit 130 includes an image coordinate (P = (x, y)) 221 where an object is located, a focal length f of a camera lens, and a center point of the camera lens. Among the (c) and camera coordinates, the coordinate value corresponding to the Z direction of the object (

)(223) is substituted into [Equation 2] and the camera coordinates for the object (

=(

)) can be derived.

[Equation 2]

=(

)), 카메라 좌표와 공간 좌표 간의 회전 변환 행렬(R) 및 이동 변환 행렬(T)을 [수학식 3]에 대입하여 객체에 대한 공간 좌표(

= (X, Y, Z))(225)를 도출할 수 있다. The spatial coordinate derivation unit 130 may derive spatial coordinates for the object based on camera coordinates for the object and parameters outside the camera. For example, referring to FIG. 2D, the spatial coordinate derivation unit 130 includes camera coordinates for an object (

=(

)), the rotation transformation matrix (R) and the movement transformation matrix (T) between the camera coordinates and the spatial coordinates are substituted into [Equation 3] and the spatial coordinates of the object (

= (X, Y, Z)) (225) can be derived.

[Equation 3]

The acoustic coordinate derivation unit 140 may derive acoustic coordinates for sounds generated from each object of a plurality of images based on spatial coordinates of the derived object.

By setting the acoustic coordinates for the sound and the spatial coordinates and direction vectors of the listener, it is possible to implement a 3D stereoscopic sound. Here, the acoustic coordinates represent a point at which a sound source in space is expressed in a three-dimensional coordinate system, and the spatial coordinates of a listener indicate a location in the space where the listener is located in a three-dimensional coordinate system.

Specifically, the listener coordinate setting unit 150 may set spatial coordinates for any one of the plurality of cameras as listener coordinates.

The listener direction vector setting unit 160 may convert a direction vector for any one camera into a vector for spatial coordinates and set it as a listener direction vector. Here, the listener direction vector means the direction the listener looks at in the space. The listener direction vector includes an up vector and a forward vector, and the up vector and the forward vector are in a perpendicular relationship with each other. For example, if the up vector is {0, 1, 0}, the forward vector becomes {0, 0, 1}, and the listener looks at the Z-axis increasing direction while standing in the Y-axis increasing direction. The listener's forward vector can be derived by transforming a basic vector in the Z-axis direction on the camera coordinates into a vector on the spatial coordinates.

Meanwhile, for those skilled in the art, the image receiving unit 100, the camera parameter deriving unit 110, the depth image generating unit 120, the spatial coordinate deriving unit 130, the acoustic coordinate deriving unit 140, the listener coordinate setting unit 150 And it will be fully understood that each of the listener direction vector setting unit 160 may be implemented separately, or one or more of them may be integrated and implemented.

3 is a flowchart illustrating a method of deriving object coordinates according to an embodiment of the present invention.

Referring to FIG. 3, in step S301, the apparatus 10 for deriving object coordinates may receive a plurality of images from a plurality of cameras.

In step S303, the object coordinate derivation apparatus 10 may derive camera parameters corresponding to a plurality of cameras.

In step S305, the apparatus 10 for deriving object coordinates may generate a depth image for a reference image among a plurality of images by using the derived camera parameter and a plurality of images.

In step S307, the object coordinate derivation apparatus 10 may derive spatial coordinates for an object included in the depth image.

In operation S309, the apparatus 10 for deriving object coordinates may derive acoustic coordinates for sound generated from the object based on the spatial coordinates for the derived object.

In the above description, steps S301 to S309 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, and the order between steps may be changed.

An embodiment of the present invention may also be implemented in the form of a recording medium including instructions executable by a computer, such as a program module executed by a computer. Computer-readable media can be any available media that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. Further, the computer-readable medium may include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains can understand that it is possible to easily transform it into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. .

10: object coordinate derivation device
100: video receiver
110: camera parameter derivation unit
120: depth image generator
130: spatial coordinate derivation unit
140: acoustic coordinate derivation unit
150: listener coordinate setting unit
160: listener direction vector setting unit

Claims (10)

In the object coordinate derivation device,
An image receiver configured to receive a plurality of images from a plurality of cameras;
A camera parameter derivation unit for deriving camera parameters corresponding to the plurality of cameras;
A depth image generator for generating a depth image for a reference image among the plurality of images by using the derived camera parameter and the plurality of images;
A spatial coordinate derivation unit for deriving spatial coordinates for an object included in the depth image;
An acoustic coordinate derivation unit that derives acoustic coordinates for the sound generated from the object based on the spatial coordinates of the derived object
Including,
The spatial coordinate derivation unit
Extracting the object from the depth image,
Calculate the center of gravity for the extracted object,
When the plurality of images are displayed through a display, an image coordinate at which the object is located in the image displayed on the display is derived based on the calculated center of gravity of the object,
A listener coordinate setting unit configured to set spatial coordinates of any one of the plurality of cameras as the listener's coordinates; And
Further comprising a listener direction vector setting unit for converting the direction vector for any one of the cameras into a vector for the spatial coordinates and setting it as the listener direction vector of the listener,
The object coordinate derivation apparatus for implementing a three-dimensional stereophonic sound based on the derived acoustic coordinates, the listener's coordinates, and the listener direction vector.
The method of claim 1,
The camera parameters include camera internal parameters and camera external parameters,
The camera internal parameters include a focal length of the camera lens, a center point of the camera lens, an asymmetry coefficient for an image sensor, and a distortion coefficient for the camera lens,
The camera external parameter includes a rotation transformation matrix and a movement transformation matrix between camera coordinates and spatial coordinates corresponding to the camera position.
delete The method of claim 1,
The spatial coordinate derivation unit
To calculate a depth pixel value of the object with respect to the derived image coordinate based on the depth image.
The method of claim 4,
The spatial coordinate derivation unit
An object coordinate derivation apparatus for calculating a coordinate value for the object on the camera coordinates based on the depth pixel value of the object and depth information on the camera coordinates derived from the depth image.
The method of claim 5,
The spatial coordinate derivation unit
The object coordinate derivation apparatus for deriving camera coordinates for the object based on the image coordinates, camera internal parameters, and coordinate values for the object on the camera coordinates.
The method of claim 6,
The spatial coordinate derivation unit
The object coordinate derivation apparatus for deriving the spatial coordinates for the object based on the camera coordinates for the object and a camera external parameter.
delete delete In the method of deriving the coordinates of the object in the object coordinate derivation device,
Receiving a plurality of images from a plurality of cameras;
Deriving camera parameters corresponding to the plurality of cameras;
Generating a depth image for a reference image among the plurality of images by using the derived camera parameter and the plurality of images;
Deriving spatial coordinates for the object included in the depth image; And
Deriving acoustic coordinates for sound generated from the object based on spatial coordinates for the derived object
Including,
The step of deriving spatial coordinates for the object is
Extracting the object from the depth image;
Calculating a center of gravity for the extracted object, and
When the plurality of images are displayed through a display, deriving image coordinates at which the object is located in the image displayed on the display based on the calculated center of gravity of the object,
Setting spatial coordinates of any one of the plurality of cameras as the listener's coordinates; And
The step of converting the direction vector for any one of the cameras into a vector for the spatial coordinates and setting it as the listener direction vector of the listener,
3D stereoscopic sound is implemented based on the derived acoustic coordinates, the listener's coordinates of the listener, and the listener direction vector.

Images