KR20200046789A - Method and apparatus for generating 3-dimensional data of moving object - Google Patents

Abstract

The present disclosure relates to a machine vision system, and more specifically, to a three-dimensional data generation method using a machine vision system. According to the present invention, the three-dimensional data generation method using the machine vision system and including two digital cameras and one optical projector to check an object moving in an observation view includes the steps of: sequentially projecting a plurality of digital patterns into the observation view; capturing a plurality of images by using the digital camera, wherein each of the images is associated with any one of the sequentially projected patterns; compensating for a motion parameter to each image among the images; generating first and second pattern images synthesized from the captured image; detecting boundaries from seventh and eighth patterns; decoding the boundaries by first to seventh pattern images; matching codes of the synthesized pattern and captured pattern; and generating three-dimensional data based on corresponding pairs of the two digital cameras.

Description

Method and apparatus for generating 3D data of a moving object {METHOD AND APPARATUS FOR GENERATING 3-DIMENSIONAL DATA OF MOVING OBJECT}

The present disclosure relates to a method and apparatus for generating 3D data, and more particularly, to a method for generating 3D data of a moving object.

Imaging systems are used to automatically inspect moving parts in a manufacturing environment. Imaging systems are being used to determine three-dimensional information about objects in the field of view (FOV) in the fields of automated production, quality inspection, reverse engineering, robotics and similar systems. These systems use structured light as part of a stereo imaging system to project light into the field of view, capture digital images of objects in the field of view, and use geometric methodology and decoding techniques to calculate the shape and depth of the object. . The projection pattern is designed with a small number of patterns for fast speed, and the decoding algorithm corrects motion to remove errors and noise and improve the accuracy of the reconstructed 3D point cloud.

There are two types of scanning methods for acquiring 3D information: white light scanning and line laser scanning. White light scanning is a method of obtaining three-dimensional information by projecting a specific pattern onto an object and grasping the deformation form of the pattern, and line laser scanning uses a triangulation method. Calculate the three-dimensional position of the points on the laser plane in the field of view. In order to obtain a three-dimensional point cloud, the laser plane must scan the object entirely using a linear stage. The former has the advantage that it is suitable for quickly scanning a small subject, and does not require additional hardware, so it is inexpensive. On the other hand, the latter is suitable for precisely scanning a large subject and requires additional hardware such as a linear stage for precise scanning.

Registered Patent Publication No. 10-1757240 (Registration date: 2017.07.06)

It is intended to provide a 3D data generation method that can ensure both scanning speed and accuracy.

The objects of the present invention are not limited to the above-mentioned objects, and other objects not mentioned are clearly understood by a person having ordinary knowledge in the technical field to which the present invention belongs (hereinafter referred to as a 'normal engineer') from the following description. It could be.

To achieve the object of the present invention as described above, and to perform the characteristic functions of the present invention described later, the features of the present invention are as follows.

The 3D data generation method according to the present invention is a 3D data generation method using a machine vision system including two digital cameras and one optical projector to inspect a moving object in an observation field of view. Sequentially projecting into the observation field of view; Capturing a plurality of images with the digital camera, wherein each of the plurality of images is associated with any one of the sequentially projected patterns; Compensating for a motion parameter to each image among the plurality of images; Generating synthesized first and second pattern images from the captured image; Detecting boundaries from the seventh and eighth patterns; Decoding a boundary by the first to seventh pattern images; Matching the codes of the composite pattern and the captured pattern; And generating 3D data based on corresponding pairs of the two digital cameras.

In one embodiment of the present invention, the plurality of digital patterns are binary code patterns obtained from reciprocals of the third to ninth bits and the ninth bits.

In one embodiment of the present invention, the plurality of images is a plurality of two-dimensional images, and each of the plurality of two-dimensional images is associated with any one of the sequentially projected patterns.

In one embodiment of the present invention, the plurality of images are a plurality of two-dimensional images, and motion parameters are compensated for each of the plurality of two-dimensional images, and each pixel moves in an x- or y-axis with respect to a reference image. do.

In one embodiment of the present invention, the reference image can be set to any one of the captured images.

In one embodiment of the present invention, the synthesized first and second pattern images are obtained based on the stripe signal of the third bit binary code.

In one embodiment of the present invention, the boundary is detected by crossing the seventh and eighth pattern images.

In one embodiment of the present invention, the first to seventh pattern images correspond to binary codes of the third to ninth bits, respectively.

In one embodiment of the present invention, the decoding step decodes a boundary having a code value of 0 and sequentially inputs a decoding value of a boundary having a code value of 1.

Disclosed is a method and apparatus for generating three-dimensional data of a moving object, in which both scanning speed and accuracy can be secured.

The effects of the present invention are not limited to those described above, and other effects not mentioned will be apparent to those skilled in the art from the following description.

1 shows a configuration diagram of a machine vision system according to the present invention,
2 is a schematic flowchart of a 3D data generation method using a machine vision system according to the present invention,
Figure 3 shows the data acquisition step of the three-dimensional data generation method according to the present invention,
4 shows the structure of a coded pattern for a 3D data generation method according to the present invention,
5 illustrates a projection pattern image for each time frame for a 3D data generation method according to the present invention,
6 shows a pattern decoding step of a 3D data generation method according to the present invention,
7 shows a motion calibration procedure according to an embodiment of the present invention,
8 shows a motion correction step according to an embodiment of the present invention,
9 shows a step of generating a composite pattern image according to an embodiment of the present invention,
10 shows that a boundary is detected based on signal pairs of the seventh and eighth pattern images,
11 shows a decoding process according to the present invention,
12 shows decoding values of the composite pattern and the captured pattern,
13 shows the boundary positions of the captured pattern layer and the composite pattern layer,
14 shows the pseudo code of the decoding process,
15 shows a three-dimensional reconstruction process from a decoding map by two cameras,
16 shows 3D data reconstructed according to the 3D data generation method according to the present invention and 3D data reconstructed by a conventional method.

The specific structure or functional descriptions presented in the embodiments of the present invention are exemplified for the purpose of explaining the embodiments according to the concept of the present invention, and the embodiments according to the concept of the present invention may be implemented in various forms. In addition, it should not be construed as being limited to the embodiments described herein, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention.

Meanwhile, in the present invention, terms such as first and / or second may be used to describe various components, but the components are not limited to the terms. The above terms are only for the purpose of distinguishing one component from other components, for example, without departing from the scope of rights according to the concept of the present invention, the first component may be referred to as the second component, Similarly, the second component may also be referred to as the first component.

When a component is said to be "connected" or "connected" to another component, it should be understood that other components may be directly connected or connected to the other component, but may be present in the middle. something to do. On the other hand, when a component is referred to as being “directly connected” to or “directly in contact with” another component, it should be understood that no other component exists in the middle. Other expressions for describing the relationship between the components, such as "between" and "immediately between" or "adjacent to" and "directly adjacent to", should be interpreted similarly.

Throughout the specification, the same reference numbers indicate the same components. Meanwhile, the terminology used herein is for describing the embodiments and is not intended to limit the present invention. In the present specification, the singular form also includes the plural form unless otherwise specified in the phrase. As used herein, "comprises" and / or "comprising" refers to the presence of one or more other components, steps, operations and / or elements in which the mentioned component, step, operation and / or element is present. Or do not exclude additions.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

As shown in FIG. 1, a machine vision system 1 according to the present invention includes two or more cameras 10 and a controllable light projector 20. The two or more cameras 10 and optical projectors 20 are connected by signals to the camera / projector controller 30 and the analysis controller 40. The analysis controller 40 may be a PC. The trigger generator 50 connects the camera 10, the light projector 20, and the camera / light projector controller 30 by signals.

The camera 10 may be a device that photographs digital images. Preferably, the camera 10 may capture a two-dimensional image of a field of view (FOV), and capture a two-dimensional image of the viewing field in response to a trigger signal generated by the trigger generator 50. can do. According to an embodiment of the present invention, a stereo camera is used because the life of the optical projector 20 is much shorter than the life of the camera 10, and thus, even if the projector is replaced, a stereo calibration that requires a complicated process is performed. Is not required.

Image is a concept that shows the field of view (FOV) and encompasses visually recognizable expressions. The image may include some or all of the reflected light in the field of view in the visible light spectrum, including color reflection, black and white reflection, grayscale reflection, or other suitable or desired reflection. The image is captured and stored in a non-transitory storage medium, and the non-transitory storage medium may be a non-transitory digital data storage medium or a photographic film.

In one example, the image captured by the camera 10 is a two-dimensional image showing the field of view (FOV), it may be a bitmap image file in the form of an 8-bit grayscale image. In another example, the two-dimensional image may be a two-dimensional color image, or may take a different form. The camera 10 and the optical projector 20 generate a bitmap image file of the field of view (FOV) at a preset resolution.

According to an embodiment of the present invention, two or more cameras 10 may be two cameras. The two cameras 10 are installed on the same plane, and each of the two cameras 10 is tilted so that the object 70 in the field of view (FOV) is in the center of the two cameras 10. It is formed to come. The optical projector 20 is installed on the same plane as the two cameras 10 and is disposed between the two cameras 10.

The two cameras 10 and the optical projector 20 arranged in this way may be arranged at any position and orientation with respect to the field of view (FOV). The field of view (FOV) includes an object 70 oriented on a plane that is movable at any speed disposed at a predetermined distance from the camera 10 and the light projector 20. The object 70 is a structural object, and may be, for example, a structural object having characteristics such as surface finish indicating spatial dimensions, materials, and reflectance. For example, the object 70 can be a part or part of any object in a factory.

For example, the movable surface may be a conveyor (C). That is, the object 70 may be disposed on the conveyor C moving the linear path at a specific speed (v). In this case, the camera 10 and the light projector 20 may be mounted spaced apart at a constant distance H from the conveyor C.

The optical projector 20 may be a digital light processing (DLP) or liquid crystal display (LCD) projector. The optical projector 20 is configured to generate an optical image with a specific projection resolution in response to a digital input signal. According to one example, the light projector 20 may have a projection resolution of 912Х1,140 pixels.

The trigger signal by the trigger generator 50 may be generated from a projector trigger output port provided in the optical projector 20. Alternatively, the trigger signal may be output from the V-Sync signal of the VGA port. The camera 10 has a general purpose IO (GPIO) port, and is connected to a projector trigger output port to synchronize timing between the optical projector 20 and the camera 10.

The camera / projector controller 30 generates a digital light pattern, and the generated light pattern is transmitted to the light projector 20. The optical projector 20 optically projects the light pattern to the field of view (FOV).

2 is a schematic flowchart of a 3D data generation method using a machine vision system according to the present invention. The method for generating 3D data according to the present invention includes a data acquisition step (S10), a pattern decoding step (S30), and a 3D data generation step (S50). In the data acquisition step (S10), a pattern image is acquired by photographing an observation field of view (FOV) on which a digital light pattern is projected with two cameras 10. In the pattern decoding step S30, the pattern image obtained in the data acquisition step S10 is decoded to obtain decoded data. In the 3D data generation step (S50), the 3D image is reconstructed using the decoded data obtained in the pattern decoding step (S30).

Referring to FIG. 3, the data acquisition step (S10) includes a starting step (S110), a pattern projection step (S210), and a capturing step (S310). The initiation step (S110) is initiated by providing an initiation situation such as the object 70 arriving in the observation field of view (FOV) on the conveyor C moving at a specific speed, and the camera / projector controller 30 is a plurality of digital The light pattern 80 is generated and transmitted to the light projector 20. Next, in the pattern projection step (S210), the optical projector 20 optically projects the plurality of digital light patterns 80 to the viewing field of view (FOV). In the capturing step (S310), the camera 10 captures a bitmap image file of an observation field of view (FOV) by a trigger signal generated by the trigger generation unit 50, and an observation field including an object 70 ( FOV) captures a state in which the light pattern 80 is projected. In one embodiment of the present invention, the projected light pattern 80 may be eight patterns, and the projected eight patterns are captured sequentially. That is, the camera / projector controller 30 sequentially generates eight digital light patterns 80, and all of the eight digital light patterns 80 are projected onto the field of view (FOV) and corresponding bitmap images. The process is repeated until the file is captured.

FIG. 4 shows a light pattern 80 projected by the optical projector 20 to the viewing field of view (FOV). The light pattern 80 is coded with truncated 9-bit binary code of the third to ninth bits. The truncated 9-bit binary codes of the third to ninth bits are set to first to seventh patterns, respectively. The first and second bits are omitted, and the eighth pattern is obtained by taking the reciprocal of the seventh pattern. In the projection step S210, the first to eighth patterns are projected sequentially or continuously.

Referring to FIG. 5, the bit signal of each layer or each binary code is extended vertically to fit the projector image size so that it can be used as a projected image. For example, as described above, when the projector resolution is 912 x 1,140 pixels, it is expanded to fit the size. It takes about 56 ms to perform the data acquisition step (S10) in which eight patterns 80 are exposed and captured in the observation field of view (FOV).

The pattern decoding step S30 is a step of decoding the pattern image obtained in the data acquisition step S10 to obtain decoded data. That is, the pattern decoding step S30 corresponds to a step of obtaining a decimal value unique to each light profile location based on the code value of the captured pattern image. 6, the pattern decoding step (S30) includes a motion correction step (S130), a composite pattern image generation step (S230) and a decoding step (S330).

In the captured pattern image, a motion correction step is first performed to reduce the pattern position error between different frames (S130).

Referring to FIG. 7, the calibration plate CP is disposed on the conveyor C moving at a specific speed v and in the field of view FOV for performing the same step. The calibration plate CP is a checker board with black and white squares arranged to form one or more edges so that the edges are uniquely identified. The optical projector 20 projects a white pattern and the camera 10 captures an image of the field of view (FOV). The optical projector 20 projects a white pattern because the captured image must be sufficiently bright for edge detection.

For the motion correction step (S130), the calibration plate CP must continuously capture two or more calibration plate frames. In FIG. 7, each frame is indicated by first and second frames. The corners of the calibration plate (CP) of each frame are detected from two or more consecutively captured frames. Next, the difference in corner positions between two frames in the x and y directions of the 2D image is calculated and used for motion correction. That is, a difference value of a corner position between two frames is calculated as a pixel value, and motion parameters dx and dy values are calculated in the x and y directions, respectively. This difference value is used for motion correction in the pattern decoding step (S30).

That is, the captured pattern image is compensated through motion parameters (dx, dy) to reduce the pattern position error between different frames. Referring to FIG. 8, in the motion correction step (S130), any one pattern image among the pattern images is set as a reference image. For example, the seventh pattern may be set as a reference image. The other pattern images are moved in the x- and y-axis by motion parameters (dx, dy), and the associated time frame of each pattern is also moved.

Next, referring to FIG. 9, a synthetic pattern image generation step (S230) is performed. As described above, the coded pattern must be recovered to complete the code string because the binary code of the first and second bits is missing. The procedure for generating a composite pattern image for the missing first and second bits is as follows.

First, the conveyor C is turned off under the same system setup, a flat surface is placed parallel to the conveyor C of the field of view (FOV) and perpendicular to the direction of the scanner, and the flat surface is scanned ( S1230).

One captured image is obtained by the above process, and the captured image corresponds to the third bit of the binary code (S2230). As illustrated in FIG. 9, an edge of a stripe formed in a pattern is detected along a line formed in the center of the image (S3230). This step can be done automatically or manually and a specific error can be tolerated, and preferably an error of ± 3 pixels.

Finally, pixel counting is done from left to right. The second bit of the binary code includes two black stripes (code 0) and two white stripes (code 0). Code 0 is assigned to pixels from the left of the image to the first border. Code 1 is assigned to pixels from the first border to the second border. Code 0 is assigned to pixels from the second border to the third border. Pixel 0 from the third border to the right end of the image is assigned a code of 0. Accordingly, a composite pattern image for the second bit is obtained (FIG. 8A, S4230).

The first bit of the binary code includes one black stripe (code 0) and one white stripe (code 1). A code of 0 is assigned to pixels from the left to the second border of the image, and a code of 1 is assigned to pixels from the second border to the right of the image. Accordingly, a composite pattern image for the first bit is obtained (FIG. 8 (B), S4230).

When all the pattern images have been obtained, a decoding step (S330) is performed.

In order to obtain a light profile or a boundary, that is, a boundary diagram, signals of the seventh and eighth patterns are crossed. At each boundary location, the decoding value is calculated based on the binary code string collected from the captured pattern and the composite pattern. In the decoding step (S330), the following steps (a) to (c) are performed.

Step (a): A boundary map is obtained by intersecting signals of the seventh and eighth patterns. A code value of 0 is assigned to the up-down signal pair and a code value of 1 is assigned to the down-up signal pair along each row of the image. The top-bottom signal pair and the bottom-top signal pair are shown in FIG. 10.

Step (b): Decode the captured pattern image. Referring to FIG. 11, an upper layer is tracked for each boundary location having a code value of 1, and a binary code rule is used to convert to a decimal number. On the other hand, for each boundary having a code value of 0, neighboring boundaries on the left and right sides of the boundary are identified. If the decoded decimal value increases by 2 from left to right, the decoded value of the boundary is averaged.

Step (c): Decode two composite pattern images. These are two bits of binary code, and thus have four decimal values. For matching with 9-bit coding, the decoding value is scaled to {0, 128, 256, 384}. The decoded values of the composite pattern and the captured patterns are thus simulated, as shown in FIG. 12.

Step (d): The decoded value of the pattern captured for each row is updated to the decoded value of the synthesized pattern according to the algorithms of FIGS. 13 and 14. Referring to FIG. 13, for each boundary position B associated with the decoded value D of the decoded map of the captured pattern, an inclusion area (i ^th ) having a start area position (S ⁱ ) and an end area position (E ⁱ ) and synthesis The decoding value Ds ⁱ of the decode map of the pattern is displayed. An exemplary pseudo code is presented in FIG. 14.

Next, as shown in FIG. 15, a 3D data generation step (S50) is performed. In the 3D data generation step (S50), after obtaining a decoding map from two cameras 10, corresponding pairs of both cameras are searched along the epipolar line. The epipolar line is related to the epipolar geometry and corresponds to the geometry related to position correspondence. 3D data is calculated by an optical triangulation method. The complete three-dimensional data of the field of view is reconstructed by repeating until both pairs of cameras are found.

In FIG. 16, the midsole (bottom) of a shoe moving at a speed of 40 mm / s generated according to the 3D data generating method according to the present invention and a shoe moving at a speed of 40 mm / s generated according to the conventional 3D data generating method A drawing comparing midsole (top) of is provided. As can be observed in the figure, it can be seen that the midsole reconstructed using the method according to the present invention has much less noise and shows the midsole in detail when compared to the conventional method.

The present invention described above is not limited by the above-described embodiments and the accompanying drawings, and various substitutions, modifications, and changes are possible within the scope of the present invention without departing from the technical spirit of the present invention. It will be obvious to those who have the knowledge of.

1: Machine Vision System 10: Camera
20: Optical projector 30: Camera / projector controller
40: analysis controller 50: trigger generation unit
70: Object FOV: Field of View
C: Conveyor

Claims (9)

A method for generating three-dimensional data using a machine vision system including two digital cameras and one optical projector to inspect objects moving in the field of view,
Sequentially projecting a plurality of digital patterns into the viewing field;
Capturing a plurality of images with the digital camera, wherein each of the plurality of images is associated with any one of the sequentially projected patterns;
Compensating for a motion parameter to each image among the plurality of images;
Generating synthesized first and second pattern images from the captured image;
Detecting boundaries from the seventh and eighth patterns;
Decoding a boundary by the first to seventh pattern images;
Matching the codes of the composite pattern and the captured pattern; And
Generating three-dimensional data based on corresponding pairs of the two digital cameras;
3D data generation method comprising a. The method according to claim 1, wherein the plurality of digital patterns is a binary code pattern obtained from an inverse number of the third to ninth bits and the ninth bit. The method of claim 1, wherein the plurality of images are a plurality of two-dimensional images, and each of the plurality of two-dimensional images is associated with one of sequentially projected patterns. The method according to claim 1, wherein the plurality of images is a plurality of two-dimensional images, each of the plurality of two-dimensional images to compensate for the motion parameter, each pixel is moved in the x- or y-axis relative to the reference image 3 How to generate dimensional data. The method of claim 4, wherein the reference image can be set to any one of captured images. The method of claim 1, wherein the synthesized first and second pattern images are obtained based on a stripe signal of a third bit binary code. The method according to claim 1, wherein the boundary is detected by intersecting the seventh and eighth pattern images. The method of claim 1, wherein the first to seventh pattern images correspond to binary codes of 3 to 9 bits, respectively. The method of claim 8, wherein the decoding comprises decoding a boundary having a code value of 0 and sequentially inputting a decoding value of a boundary having a code value of 1.

Images