KR102694987B1 - Pose estimating device and method using artificial intelligence to estimate pose of robot's end effector for object and robot control system including the same - Google Patents

Abstract

A pose estimation device for estimating a pose of an end effector of a robot for an object using artificial intelligence according to one embodiment of the present disclosure includes at least one processor and at least one memory including a computer program code. The at least one memory and the computer program code are configured such that the pose estimation device, through the at least one processor, obtains a two-dimensional image corresponding to a target area from a 2D camera, estimates three-dimensional transformation information corresponding to the obtained two-dimensional image based on a pre-learned artificial intelligence model, and estimates the pose of the end effector based on the estimated three-dimensional transformation information. The artificial intelligence model is learned to estimate the three-dimensional transformation information by matching the two-dimensional image to a pre-stored three-dimensional comparison image.

Description

{POSE ESTIMATING DEVICE AND METHOD USING ARTIFICIAL INTELLIGENCE TO ESTIMATE POSE OF ROBOT'S END EFFECTOR FOR OBJECT AND ROBOT CONTROL SYSTEM INCLUDING THE SAME}

The present disclosure relates to a pose estimation device and method for estimating a pose of an end effector of a robot with respect to an object using artificial intelligence, and a robot control system including the same.

Robots are the main equipment that performs tasks within the process from product production to shipment, and are used in a wide range of industrial technology fields because they show greater process efficiency than humans. In particular, manufacturing robots can prevent safety accidents caused by workers holding things by hand or lifting and moving them with tools, and improve work efficiency.

These robots are configured with multiple joints for various and precise tasks, and thus the end effector can move and rotate along multiple axes. For example, the end effector can have six degrees of freedom (DoF) that can move and rotate along the X-axis, Y-axis, and Z-axis.

In order to control the motion of the robot, information on the pose of the end effector is required, and for this purpose, the pose of the end effector according to the degree of freedom was detected by using a 3D camera in the past. However, there was a problem that the unit cost of applying the robot system increased due to the use of a relatively expensive 3D camera.

One purpose of the present disclosure is to estimate the pose of an end effector from a two-dimensional image acquired using a 2D camera based on a pre-learned artificial intelligence model.

However, the technical tasks that this embodiment seeks to accomplish are not limited to the technical tasks described above, and other technical tasks may exist.

A pose estimation device for estimating a pose of an end effector of a robot for an object using artificial intelligence according to one embodiment of the present disclosure may include at least one processor and at least one memory including a computer program code. The at least one memory and the computer program code may be configured such that the pose estimation device, through the at least one processor, obtains a two-dimensional image corresponding to a target area from a 2D camera, estimates three-dimensional transformation information corresponding to the obtained two-dimensional image based on a pre-learned artificial intelligence model, and estimates the pose of the end effector based on the estimated three-dimensional transformation information. The artificial intelligence model may be learned to estimate the three-dimensional transformation information by matching the two-dimensional image to a pre-stored three-dimensional comparison image.

A pose estimation method for estimating a pose of an end effector of a robot with respect to an object using artificial intelligence according to one embodiment of the present disclosure may include an operation of acquiring a two-dimensional image of a target area from a 2D camera, an operation of estimating three-dimensional transformation information corresponding to the two-dimensional image based on a pre-learned artificial intelligence model, and an operation of estimating the pose of the end effector based on the estimated three-dimensional transformation information. The artificial intelligence model may be learned to estimate the three-dimensional transformation information by matching the two-dimensional image to a pre-stored three-dimensional comparison image.

A robot control system including a pose estimation device for estimating a pose of an end effector of a robot with respect to an object using artificial intelligence according to one embodiment of the present disclosure may include a 2D camera configured to acquire a 2D image of a target area, a pose estimation device configured to acquire the 2D image from the 2D camera, estimate 3D transformation information corresponding to the acquired 2D image based on a pre-learned artificial intelligence model, and estimate the pose of the end effector based on the estimated 3D transformation information, and a robot device having a base and a rotatable or movable end effector and configured to control the end effector based on the estimated 3D transformation information. The artificial intelligence model may be learned to estimate the 3D transformation information by matching the 2D image to a pre-stored 3D comparison image.

According to one embodiment of the present disclosure, by estimating three-dimensional transformation information corresponding to a two-dimensional image based on a learning model using artificial intelligence, depth data of a three-dimensional space can be obtained, and the pose of an end effector can be estimated accordingly.

In addition, according to one embodiment of the present disclosure, by estimating the pose of the end effector of the robot using a relatively inexpensive 2D camera, the unit cost of the robot system can be reduced and computing resources consumed for large-scale data processing can be reduced.

FIG. 1 illustrates a configuration diagram of a pose estimation device according to one embodiment of the present disclosure.
FIG. 2 is a block diagram illustrating a robot control system including a pose estimation device according to one embodiment of the present disclosure.
FIG. 3 illustrates an example of a robotic device according to one embodiment of the present disclosure.
FIG. 4 is a flowchart of a pose estimation method according to one embodiment of the present disclosure.
FIG. 5 is an example of a two-dimensional image according to one embodiment of the present disclosure.
FIG. 6 is an example of an artificial intelligence model according to one embodiment of the present disclosure.
FIG. 7 is an example of matched feature points between a two-dimensional image and a previously stored three-dimensional comparison image according to one embodiment of the present disclosure.

Below, with reference to the attached drawings, embodiments of the present invention are described in detail so that those with ordinary skill in the art can easily practice the present invention. However, the present invention may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in order to clearly describe the present invention in the drawings, parts that are not related to the description are omitted, and similar parts are assigned similar drawing reference numerals throughout the specification.

Throughout this specification, when it is said that an element is "on" another element, this includes not only cases where the element is in contact with the other element, but also cases where there is another element between the two elements.

Throughout this specification, whenever a part is said to "include" a component, this does not mean that it excludes other components, but rather that it may include other components, unless otherwise specifically stated.

The terms "about," "substantially," etc., used throughout this specification are used to mean at or near the numerical value when manufacturing and material tolerances inherent in the meanings stated are presented, and are used to prevent unscrupulous infringers from unfairly utilizing the disclosure where exact or absolute values are stated to aid understanding of this specification. The terms "step of doing" or "step of" used throughout this specification do not mean "step for."

Throughout this specification, the term "combination(s) thereof" included in the expressions in the Makushi format means one or more mixtures or combinations selected from the group consisting of the components described in the Makushi format, and means including one or more selected from the group consisting of said components.

Throughout this specification, references to “A and/or B” mean “A or B, or A and B.”

Hereinafter, implementation examples and embodiments of the present invention will be described in detail with reference to the attached drawings. However, the present invention may not be limited to these implementation examples and embodiments and drawings.

FIG. 1 illustrates a configuration diagram of a pose estimation device according to one embodiment of the present disclosure. FIG. 2 illustrates a configuration diagram of a robot control system (200) including a pose estimation device (100) according to one embodiment of the present disclosure.

Referring to FIGS. 1 and 2, a robot control system (200) including a pose estimation device (100) according to one embodiment may include a pose estimation device (100), a 2D camera (210), and/or a robot device (230).

A pose estimation device (100) according to one embodiment may include a processor (110) and/or a memory (120).

The processor (110) may control at least one other component (e.g., a hardware or software component) of the pose estimation device (100) connected to the processor (110) by executing, for example, software (e.g., a program), and may perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculations, the processor (110) may store commands or data received from other components in volatile memory, process the commands or data stored in the volatile memory, and store result data in non-volatile memory.

According to one embodiment, the processor (110) may include a main processor (e.g., a central processing unit or an application processor) or an auxiliary processor (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that may operate independently or together with the main processor. For example, the auxiliary processor may be configured to use less power than the main processor or to be specialized for a given function, and the auxiliary processor may be implemented separately from the main processor or as part of the main processor.

The auxiliary processor may control at least a portion of functions or states associated with at least one of the components of the pose estimation device (100), for example, on behalf of the main processor while the main processor is in an inactive (e.g., sleeping) state, or together with the main processor while the main processor is in an active (e.g., running an application) state.

In one embodiment, the auxiliary processor (e.g., an image signal processor or a communication processor) may be implemented as part of another functionally related component. In one embodiment, the auxiliary processor (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models.

The artificial intelligence model can be generated through machine learning. Such learning can be performed, for example, in the pose estimation device (100) on which the artificial intelligence model is performed, or can be performed through a separate server. The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include a plurality of artificial neural network layers. The artificial neural network can be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model can include a software structure, additionally or alternatively.

The memory (120) can store various data used by at least one component (e.g., processor (110)) of the pose estimation device (100). The data can include, for example, input data or output data for software (e.g., program) and commands related thereto. The memory (120) can include volatile memory or nonvolatile memory. The program can be stored as software in the memory (120) and can include, for example, an operating system, middleware, or an application.

A 2D camera (210) according to one embodiment may be configured to acquire a 2D image corresponding to a target area. For example, the 2D image is an image having position data on a plane and may not include depth data.

According to one embodiment, a robot device (230) may include a processor (231), a memory (233), a base (235), a drive module (237), and/or an end effector (239). In one embodiment, the processor (231) and/or the memory (233) of the robot device (230) may be provided in a separate control device, or may be integrated into the robot device (230). The description of the processor (231) and/or the memory (233) is replaced with the description of the pose estimation device (100).

FIG. 3 illustrates an example of a robotic device (230) according to one embodiment of the present disclosure.

Referring further to FIG. 3, a robotic device (230) according to one embodiment may include a base (235), a drive module (237), and/or an end effector (239).

In one embodiment, the base (235) may be fixedly coupled to the ground or an external device. In one embodiment, the coordinate system (R) of the base (235) may be a fixed coordinate system.

In one embodiment, the end effector (239) may be coupled to the base (235) to be movable or rotatable via the drive module (237). In one embodiment, the coordinate system (X) of the end effector (239) may be rotated or translated relative to the coordinate system (R) of the base (235).

In one embodiment, the end effector (239) may be a part that has a function to directly act on an object that is a work target when the robot device (230) performs a work. For example, the end effector (239) may be equipped with a gripper, a welding torch, a spray gun, a nut runner, etc.

In one embodiment, the drive module (237) is provided between the base (235) and the end effector (239) and can move or rotate the end effector (239) relative to the fixed base (235). For example, the drive module (237) may be a motor that is driven by electric power to move or rotate the end effector (239).

In one embodiment, the drive module (237) may be provided in multiple units to enable the end effector (239) to rotate or move in multiple directions. For example, the end effector (239) may have six degrees of freedom (DoF) that enable movement and rotation along the X-axis, Y-axis, and Z-axis.

In one embodiment, the 2D camera (210) may be fixedly coupled to the ground or an external device, similar to the base (235), but may be moved or rotated in response to the movement or rotation of the end effector (239). The coordinate system (C) of the 2D camera (210) may be fixed, or may be moved or rotated.

In one embodiment, the 2D camera (210) may be positioned such that its FOV (Field of View) includes a target area where the robot device (230) is positioned. In one embodiment, the 2D camera (210) may be positioned such that its FOV faces an area where an end effector (239) of the robot device (230) can be positioned. In one embodiment, the 2D camera (210) may acquire a two-dimensional image corresponding to the target area in real time and transmit it to the pose estimation device (100).

In one embodiment, a marker (240) for calibrating the coordinate system of the 2D camera (210) and the coordinate system of the robot may be attached to the end effector (239). In one embodiment, the marker (240) may be attached so as to easily obtain 3D transformation information to be described later in response to a 2D image. For example, the marker (240) may be a chessboard marker such as an Arco (Chess) marker or a Charuco marker, but is not limited thereto, and various known markers may be used.

Fig. 4 is a flowchart (400) of a pose estimation method according to one embodiment of the present disclosure. Fig. 5 is an example of a two-dimensional image (2D Image) according to one embodiment of the present disclosure.

Referring to FIGS. 4 and 5, a pose estimation device (100, e.g., processor (110)) according to one embodiment may, in operation 410, obtain a two-dimensional image (2D Image, e.g., 2D RGB) of a target area from a 2D camera (210).

As an example, a two-dimensional image (2D Image) acquired from a 2D camera (210) may include an image of an end effector (239) of the robot.

A pose estimation device (100) according to one embodiment can preprocess a two-dimensional image (2D Image) in operation 430.

A pose estimation device (100) according to one embodiment can extract a portion of an area corresponding to an end effector (239) of a robot positioned in a target area from a two-dimensional image (2D Image) acquired from a 2D camera (210), and preprocess the extracted portion of the area into gray scale.

In one embodiment, the pose estimation device (100) can detect the end effector (239) of the robot from the acquired two-dimensional image (2D Image), and crop and extract a portion of the region of interest (ROI) including the detected end effector (239). For example, the portion of the region may be a bounding box (BBox).

In one embodiment, the pose estimation device (100) can process a portion of an extracted two-dimensional image (2D Image) in gray scale. Accordingly, a gray scale having a single channel can perform calculations efficiently since it has a smaller number of channels compared to a color image, and can learn and infer features by focusing on the intensity of pixel values, so that it can be simplified. In addition, it can reduce differences occurring in color data as environmental conditions such as lighting change in the matching of point clouds described below. In addition, an image processed in gray scale can be effective in emphasizing structural features (e.g., texture).

A pose estimation device (100) according to one embodiment may extract feature points from a two-dimensional image (e.g., Cropped 2D RGB) based on an artificial intelligence model in operation 450.

In one embodiment, the feature points may be corners, edges, and/or texture changes in the image.

In one embodiment, the pose estimation device (100) can extract a point that best matches a three-dimensional comparison image (3D camera RGB) from a two-dimensional image (e.g., Cropped 2D RGB) as a feature point. In one embodiment, the three-dimensional comparison image (e.g., 3D camera RGB) can be acquired by a 3D camera (not shown) included in a separate device and stored in the pose estimation device (100, e.g., memory (120)).

A pose estimation device (100) according to one embodiment may, in operation 470, match matching points (e.g., 3D camera PointCloud) of a 3D comparison image from feature points (e.g., matching keypoints) extracted based on an artificial intelligence model.

Fig. 6 is an example of an artificial intelligence model according to one embodiment of the present disclosure. Fig. 7 is an example of matched feature points between a two-dimensional image (2D Image) and a pre-stored three-dimensional comparison image (3D Image) according to one embodiment of the present disclosure.

Referring further to FIGS. 6 and 7, in one embodiment, the artificial intelligence model may be trained to extract feature points from a two-dimensional image (2D Image) and match the two-dimensional image (2D Image) to a previously stored three-dimensional comparison image (3D Image) based on the extracted feature points.

In one embodiment, the AI model may be a deep learning model optimized for matching feature points between images using a graph neural network. In one embodiment, the AI model may regard each feature point as a node of a graph and express the interaction between them as an edge. Through this process, the AI model can efficiently learn complex patterns and relationships and produce more sophisticated matching results.

For example, according to the LightGlue architecture illustrated in Fig. 6, given a pair of input local features (d, p) from images, each layer can enhance the visual descriptor with context based on self-attention or cross-attention using positional encoding. The confidence classifier (c) can decide whether to stop the inference. If there are few confident points, the inference proceeds to the next layer, but removes points that do not clearly match. When LightGlue reaches a confident state, it predicts the assignment between points based on pairwise similarity and single matchability.

In one embodiment, the artificial intelligence model may be trained to estimate 3D transformation information including matching points that match a 2D image and a previously stored 3D comparison image (3D Image) from extracted feature points.

In one embodiment, the artificial intelligence model may include an encoder (Keypoint Encoder), a graph neural network (Multiplex Graph Neural Network), and a matching layer (Optimal Matching Layer).

As an example, an encoder (Keypoint Encoder) encodes the location and appearance information of feature points, and the encoded information can be processed by a graph neural network and play an important role in the matching process.

As an example, a multiplex graph neural network can be an advanced network that models the relationship between feature points between two images. The graph neural network can effectively determine which feature points are likely to match each other.

In one embodiment, the Optimal Matching Layer may be a deterministic layer that determines the optimal matching pair based on information generated from the graph neural network. In this process, features that cannot be matched may be automatically excluded.

A pose estimation device (100) according to one embodiment can estimate 3D transformation information (e.g., PnP output, 2D camera matrix) corresponding to a 2D image in operation 480.

As an example, the pose estimation device (100) can estimate the correlation between a 2D camera (210) and a 3D camera (not shown) corresponding to a pre-stored 3D comparison image (3D Image) as 3D transformation information.

In one embodiment, the pose estimation device (100) can estimate 3D transformation information based on a first transformation matrix corresponding to a relative position between a 3D camera (not shown) corresponding to a pre-stored 3D comparison image (3D Image) and an end effector (239), and a second transformation matrix corresponding to a relative position between a 2D camera (210) corresponding to a 2D image and a base (235) of the robot.

In one embodiment, the pose estimation device (100) can calibrate the coordinate system of the robot and the coordinate system of the camera based on the transformation matrix. For example, the transformation matrix (T) for obtaining the relative position and direction between the end effector (239) and the camera can be a mathematical formula as follows.

The above transformation matrix ( ) can be a 4X4 matrix using a homogeneous coordinate system. The upper 3X3 matrix can be a rotation vector representing the rotation along each axis direction. The upper three elements of the last column can be relative positions representing the translation.

In one embodiment, a transformation matrix ( ) between the base (235) and the end effector (239) ) can be obtained from a robotic device (230) having a tool installed on an end effector (239).

In one embodiment, the pose estimation device (100) is a transformation matrix corresponding to the mathematical expression 1. ) and the transformation matrix between the base (235) and the end effector (239). ), a first transformation matrix ( ) corresponding to the relative position between the 3D camera (not shown) and the end effector (239) corresponding to the pre-stored 3D comparison image (3D Image) is generated. ) can be estimated using the following mathematical formula.

In one embodiment, the pose estimation device (100) is a transformation matrix corresponding to the mathematical expression 1. ) and the transformation matrix between the base (235) and the end effector (239). ), a second transformation matrix ( ) corresponding to the relative position between the 2D camera (210) corresponding to the 2D image and the base (235) of the robot ) can be estimated.

In one embodiment, the pose estimation device (100) can estimate pose data of a 2D camera (210) corresponding to a 2D image (2D Image) in comparison with a 3D camera (not shown) corresponding to a pre-stored 3D comparison image (3D Image) by using the PnP (Perspective N Points) method based on matching points.

In one embodiment, the PnP method can estimate the pose data of a 2D camera (210) by connecting the feature points (e.g., matching keypoints) of a 2D image and the matching points (e.g., 3D camera PointCloud) of a previously stored 3D comparison image (3D image). For example, the pose data of the 2D camera (210) can include the position and/or direction of the 2D camera (210).

In one embodiment, the pose estimation device (100) can move feature points of a two-dimensional image (2D image) into a three-dimensional space based on an internal matrix of the 2D camera (210), and estimate pose data of the 2D camera (210) based on the feature points in the three-dimensional space.

As an example, the pose estimation device (100) may use an EPnP (Efficient PnP) method that can estimate an accurate pose with a small number of points.

A pose estimation device (100) according to one embodiment is trained to estimate at least one model based on matching points and select an optimal model among the estimated models by using a RANSAC (random sample consensus) technique.

As an example, the RANSAC technique can be an iterative method that can estimate a robust model even when there are outliers in the data.

For example, according to the RANSAC technique, a pose estimation device (100) randomly selects (samples) a portion from a data set (e.g., feature point-matching points), estimates a model (e.g., pose of a 2D camera (210)) based on the selected portion, calculates an error for each point targeting the entire data using the estimated model, determines points having an error smaller than a predefined threshold as inliers (fitness evaluation), and repeats this process to estimate an optimal model that can obtain the largest number of inliers (optimal model selection). In addition, based on the finally estimated optimal model, the model can be adjusted and refined again using only the inliers.

A pose estimation device (100) according to one embodiment can estimate the pose of an end effector (239) based on the estimated three-dimensional transformation information in operation 490.

In one embodiment, the pose estimation device (100) can obtain 3D data corresponding to a 2D image obtained from a 2D camera (210) based on the estimated 3D transformation information. In one embodiment, the pose estimation device (100) can estimate the pose of the end effector (239) based on the obtained 3D data.

Therefore, according to the pose estimation device (100) according to one embodiment, by combining the advantages of a 2D camera (210) capable of providing a high-resolution image and the advantages of a 3D camera (not shown) capable of providing depth information, it is possible to obtain both a high-resolution image and depth information.

In addition, according to the pose estimation device (100) according to one embodiment, by using a two-dimensional image (2D Image) and three-dimensional data together, the accuracy of object recognition and tracking can be greatly improved.

In addition, while the 2D camera (210) is sensitive to changes in lighting, the 3D camera (not shown) is relatively less affected by lighting, so according to the pose estimation device (100) according to one embodiment, stable performance can be secured even under various lighting conditions.

In addition, according to the pose estimation device (100) according to one embodiment, the cost efficiency of the entire system can be increased by supplementally utilizing 3D data as needed.

In the pose estimation device (100) described above, the pose estimation method may also be implemented in the form of a computer program stored in a computer-readable recording medium executed by a computer or a recording medium including commands executable by a computer. In addition, in the pose estimation device (100) described above, the pose estimation method may also be implemented in the form of a computer program stored in a computer-readable recording medium executed by a computer.

Computer-readable recording media can be any available media that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. Additionally, computer-readable recording media can include computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data.

The functions realized by the components described in the present specification may be implemented in processing circuitry including general purpose processors, special purpose processors, integrated circuits, Application Specific Integrated Circuits (ASICs), Central Processing Units (CPUs), circuits and/or combinations thereof programmed to realize the described functions. The processor includes transistors or other circuits and is considered a circuit or processing circuit. The processor may be a programmed processor that executes a program stored in a memory.

In the present specification, a circuit, a part, a unit, or a means is hardware programmed to realize the described function or hardware that executes it. The hardware may be any hardware disclosed in the present specification or any hardware known to be programmed or executed to realize the described function.

If the hardware in question is a processor considered to be a circuit type, the circuit, the part, means or unit is a combination of hardware and software used to configure the hardware and/or the processor.

The above description of the present disclosure is for illustrative purposes only, and those skilled in the art will appreciate that the present disclosure can be easily modified into other specific forms without changing the technical idea or essential features of the present disclosure. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive. For example, each component described as a single component may be implemented in a distributed manner, and likewise, components described as distributed may be implemented in a combined manner.

The scope of the present disclosure is indicated by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present disclosure.

100 : Pose Estimation Device
200 : Robot Control System
210 : 2D Camera
230 : Robotic Device

Claims (9)

In a pose estimation device that estimates the pose of an end effector of a robot for an object using artificial intelligence,
at least one processor; and
comprising at least one memory containing computer program code;
The at least one memory and the computer program code cause the pose estimation device to:
From a 2D camera, a 2D image corresponding to the target area is acquired,
Based on the learned artificial intelligence model, 3D transformation information corresponding to the acquired 2D image is estimated,
Based on a first transformation matrix corresponding to a relative position between a 3D camera corresponding to a pre-stored 3D comparison image and the end effector, and a second transformation matrix corresponding to a relative position between the 2D camera corresponding to the 2D image and the base of the robot, the pose of the end effector is estimated from the estimated 3D transformation information.
The above artificial intelligence model is trained to estimate the three-dimensional transformation information by matching the two-dimensional image to the stored three-dimensional comparison image.
Pose estimation device.
In paragraph 1,
The at least one memory and the computer program code cause the pose estimation device to:
It is configured to extract a portion of an area from the above two-dimensional image and preprocess the extracted portion of the area into gray scale.
Pose estimation device.
delete In paragraph 1,
The above artificial intelligence model is trained to extract feature points from the two-dimensional image and match the two-dimensional image to the previously stored three-dimensional comparison image based on the extracted feature points.
Pose estimation device.
In paragraph 4,
The above artificial intelligence model is learned to estimate the 3D transformation information including matching points that match the 2D image and the stored 3D comparison image from the extracted feature points based on a graph neural network.
Pose estimation device.
In paragraph 5,
The at least one memory and the computer program code cause the pose estimation device to:
Based on the above matching points, the PnP (Perspective N Points) method is used to estimate the pose data of the 2D camera corresponding to the 2D image compared to the 3D camera corresponding to the stored 3D comparison image.
Pose estimation device.
In paragraph 6,
The at least one memory and the computer program code cause the pose estimation device to:
By using the RANSAC (random sample consensus) technique, at least one model is estimated based on the matching points, and the optimal model is selected among the estimated models.
Pose estimation device.
A pose estimation method for estimating the pose of an end effector of a robot for an object using artificial intelligence,
An action of acquiring a two-dimensional image of a target area from a 2D camera;
An operation of estimating 3D transformation information corresponding to the 2D image based on a pre-learned artificial intelligence model; and
An operation of estimating a pose of the end effector from the estimated 3D transformation information based on a first transformation matrix corresponding to a relative position between a 3D camera corresponding to a pre-stored 3D comparison image and the end effector and a second transformation matrix corresponding to a relative position between the 2D camera corresponding to the 2D image and the base of the robot,
The above artificial intelligence model is trained to estimate the three-dimensional transformation information by matching the two-dimensional image to the stored three-dimensional comparison image.
Pose estimation method.
A robot control system including a pose estimation device that estimates the pose of an end effector of a robot with respect to an object using artificial intelligence,
A 2D camera configured to acquire a two-dimensional image of a target area;
A pose estimation device configured to acquire the two-dimensional image from the two-dimensional camera, estimate three-dimensional transformation information corresponding to the acquired two-dimensional image based on a pre-learned artificial intelligence model, and estimate the pose of the end effector from the estimated three-dimensional transformation information based on a first transformation matrix corresponding to a relative position between the 3D camera corresponding to a pre-stored three-dimensional comparison image and the end effector and a second transformation matrix corresponding to a relative position between the 2D camera corresponding to the two-dimensional image and the base of the robot; and
A robotic device having a base and a rotatable or movable end effector, and configured to control the end effector based on the estimated three-dimensional transformation information,
The above artificial intelligence model is trained to estimate the three-dimensional transformation information by matching the two-dimensional image to the stored three-dimensional comparison image.
Robot control system.

Images