JP2015152338A - Distance information acquisition method, distance information acquisition apparatus, and robot - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a distance information acquisition method of acquiring distance information of an object reliably, a distance information acquisition apparatus, and a robot.SOLUTION: A distance information acquisition method of measuring distance to an object to acquire distance information, by use of a distance sensor includes the steps of: detecting a vector indicating the object (S20); measuring distance by changing a measurement area by means of the distance sensor when there is no distance information measured by the distance sensor on the vector (S50); detecting distance information measured by the distance sensor on the vector (S60); and acquiring distance information of the object, on the basis of the detected distance information (S70).

Description

  The present invention relates to a distance information acquisition method, a distance information acquisition device, and a robot.

  When acquiring three-dimensional information of an object, a multi-view camera in which a plurality of cameras are integrated is often used. In Patent Document 1, an object is imaged by a multi-view camera in which external parameters (camera position and orientation) and internal parameters (focal length, pixel pitch, etc.) of each camera are calibrated in advance. It is described that the corresponding point is detected from the image, the distance to the corresponding point, that is, the distance information of the corresponding point is acquired by the principle of triangulation, and the three-dimensional reconstruction is performed to acquire the three-dimensional information of the object. Has been.

JP 2002-342787 A

  As described above, the distance to the object, that is, the distance information of the object can be easily acquired by the multi-lens camera according to the background art. However, in a multi-lens camera, when one of a plurality of cameras captures and captures an object, another camera cannot capture the object due to the presence of an obstacle or the like. Distance information may not be acquired. Such a situation can also occur when distance information of an object is acquired using a camera and a distance sensor. For example, when the distance sensor tries to acquire the distance information of the object based on the imaging information of the camera that can image up to almost infinity, the distance of the object is small in the distance sensor having a measurement range of 0.5 to 4.0 m. Information cannot be obtained.

  The present invention has been made to solve such problems, and an object of the present invention is to provide a distance information acquisition method, a distance information acquisition device, and a robot that can reliably acquire distance information of an object. To do.

  The distance information acquisition method according to the present invention is a distance information acquisition method for measuring a distance to a target from which distance information is acquired by a distance sensor, the step of detecting a vector pointing to the target, and the distance on the vector When there is no distance information by the sensor, the distance sensor changes the measurement area and measures the distance, the step of detecting the distance information by the distance sensor on the vector, and the detected distance information And obtaining the distance information of the object. With such a configuration, it is possible to reliably acquire target distance information.

  Further, the distance information acquisition device according to the present invention detects a vector detection unit that detects a vector indicating a target for which distance information is acquired, and distance information by a distance sensor on the vector, and based on the distance information, A distance detection unit that acquires distance information of a target, and drives the distance sensor so that the distance sensor changes a measurement region and measures a distance when there is no distance information by the distance sensor on the vector And a sensor driving unit. With such a configuration, it is possible to reliably acquire target distance information.

  In addition, the distance information acquisition program according to the present invention is a method for detecting a vector indicating a target for acquiring distance information in a computer, and the distance sensor is a measurement region when there is no distance information by the distance sensor on the vector. The distance sensor information is acquired based on the procedure for driving the distance sensor to measure the distance by changing the distance, the procedure for detecting the distance information by the distance sensor on the vector, and the distance information. To execute the procedure. With such a configuration, it is possible to reliably acquire target distance information.

  ADVANTAGE OF THE INVENTION By this invention, the distance information acquisition method, distance information acquisition apparatus, and robot which can acquire the distance information of a target object reliably can be provided.

It is a figure which shows the robot 11 incorporating the distance information acquisition apparatus which concerns on Embodiment 1, and its operating environment. It is a figure which shows HMI21 which concerns on Embodiment 1. FIG. It is a figure which shows the positional relationship of the visual field 14 and the measurement area | region 15 when the robot 11 which concerns on Embodiment 1 can acquire distance information. It is a figure which shows the positional relationship of the visual field 14 and the measurement area | region 15 when the robot 11 which concerns on this Embodiment 1 cannot acquire distance information. 6 is a diagram illustrating a positional relationship between the visual field 14 and the measurement region 15 when the robot 11 according to Embodiment 1 changes the measurement region 15. FIG. 1 is a block diagram illustrating a schematic configuration of a robot 11 including a distance information acquisition device 100 according to a first embodiment. 5 is a flowchart illustrating a processing procedure of a distance information acquisition method executed by the distance information acquisition device 100 according to Embodiment 1. It is a figure which shows the instruction | indication vector 34 which the vector detection part 101 which concerns on Embodiment 1 detected. It is the figure which showed the method in which the distance detection part 102 which concerns on Embodiment 1 detects distance information. It is a figure which shows the position and direction of the distance sensor 13 which the sensor drive part 103 concerning Embodiment 1 determines.

(Embodiment 1 of the invention)
Hereinafter, the distance information acquisition apparatus according to the first embodiment will be described with reference to the drawings.
The distance information acquisition apparatus according to the first embodiment acquires distance information by driving the distance sensor to change the measurement region when the distance sensor cannot acquire distance information.
First, an outline of a robot incorporating the distance information acquisition apparatus according to the first embodiment and its operation will be described.

FIG. 1A is a diagram showing a robot 11 and its operating environment according to the first embodiment. FIG. 1B is a diagram showing the HMI 21 that gives an instruction to the robot 11 according to the first embodiment.
The robot 11 has a built-in distance information acquisition device (not shown), and also has a camera 12 for displaying an HMI (Human Machine Interface) and a distance sensor 13 on its head. The field of view 14 of the camera 12 and the measurement area 15 of the distance sensor 13 are each directed in the direction of a remote controller (remote controller) 17 that is an object to be gripped by the robot 11. In such a situation, the user operates the HMI 21 that is a display means that can be operated by the user to give an instruction to the robot 11, and the robot 11 performs an operation of gripping the remote controller 17 by the action unit 16 that is a moving mechanism or an arm / hand mechanism. Run. The distance information acquisition device executes processing related to distance information acquisition of the robot 11.

  The HMI 21 displays an image of the remote controller 17 captured by the camera 12. The user instructs the robot 11 that the remote controller 17 is a gripping object by tapping the displayed image of the remote controller 17. The robot 11 acquires the position information and three-dimensional information of the remote controller 17 based on the distance information of the distance sensor 13 and holds the remote controller 17. At this time, there may be a situation in which the robot 11 cannot acquire the distance information of the remote controller 17.

  FIG. 2 is a diagram showing a positional relationship between the visual field 14 of the camera 12 and the measurement area 15 of the distance sensor 13 according to the first embodiment. The camera 12 is connected to the remote controller 17 placed on the floor near the wall. It is a figure when the situation which is imaging is seen from the horizontal direction (lateral direction) of the camera. The distance sensor 13 acquires distance information as distance points 31 for a part of the floor surface in the measurement region 15 and the surface of the remote controller 17. Further, the point where the user taps the display image of the HMI 21 is reflected as the user instruction point 33 on the virtual image plane 32 of the perspective projection model.

  The perspective projection is to map an object point in space on a virtual image plane. The virtual image plane is a virtual plane located at the focal length f in front of the camera viewpoint O. For technical terms such as “perspective projection”, “virtual image plane”, “viewpoint”, and “optical axis”, which will be described later, Kenichi Kanaya, “Mathematics of Camera Calibration —How was the best lattice pattern derived?” “Part 1 perspective transformation and focal length”, Mathematical Science, Science, October 1993, Vol. 31, No. 10, p.

FIG. 2A is a diagram showing a positional relationship between the visual field 14 and the measurement region 15 when the robot 11 according to the first embodiment can acquire distance information. When the distance point 31 is on the instruction vector 34, which is a vector connecting the viewpoint O of the camera 12 and the user instruction point 33, the robot 11 can acquire distance information.
FIG. 2B is a diagram showing a positional relationship between the visual field 14 and the measurement region 15 when the robot 11 according to the first embodiment cannot acquire distance information. For some reason, the positional relationship between the visual field 14 and the measurement region 15 has changed from that shown in FIG. 2A. For this reason, there is no distance point 31 on the instruction vector 34, and the robot 11 cannot acquire distance information.

  FIG. 2C is a diagram showing a positional relationship between the visual field 14 and the measurement region 15 when the robot 11 according to the first embodiment changes the measurement region 15. When the situation where the distance information cannot be acquired as shown in FIG. 2B occurs, the robot 11 drives the distance sensor 13 to change the position of the distance sensor 13 and change the measurement region 15. Then, scanning is performed on the instruction vector 34, and when the distance point 31 appears on the instruction vector 34, the distance information is acquired as the distance information of the target object designated by the user. Thus, the distance information acquisition apparatus according to Embodiment 1 can reliably acquire the distance information of the target object.

Next, a schematic configuration of the distance information acquisition device according to the first embodiment will be described together with a schematic configuration of the robot 11 including the distance information acquisition device.
FIG. 3 is a block diagram illustrating a schematic configuration of the robot 11 including the distance information acquisition device 100 according to the first embodiment. The robot 11 includes a distance information acquisition device 100, a camera 12, a distance sensor 13, a three-dimensional information acquisition unit 110, an operation control unit 111, an action unit 16, and the like. The distance information acquisition apparatus 100 includes a vector detection unit 101, a distance detection unit 102, a sensor driving unit 103, and the like. Further, an HMI 21 is provided outside the robot 11, for example, at the user's hand.

  When the user designates an object by tapping the display image of the HMI 21, the vector detection unit 101 inputs the coordinate information of the tap position in the display image from the HMI 21, and this is input to the user indication point 33 on the virtual image plane 32. Convert to coordinate information. Further, the position information of the viewpoint O is input from the camera 12. Then, the vector detection unit 101 detects a vector connecting the viewpoint O and the user instruction point 33 as an instruction vector having a start point and a direction in a three-dimensional space, and as a vector indicating a target for obtaining distance information, The data is output to the distance detection unit 102.

  The distance detection unit 102 inputs the instruction vector information from the vector detection unit 101, inputs the distance information from the distance sensor 13, determines whether or not the distance point 31 is on the instruction vector 34, and When there is a distance point 31, the distance information of the object is acquired based on the distance information of the distance point 31 and the position information of the distance sensor, and is output to the three-dimensional information acquisition unit 110. Further, when there is no distance point 31 on the instruction vector 34, the distance detection unit 102 outputs the fact and vector information to the sensor driving unit 103.

  The sensor driving unit 103 drives the distance sensor 13 based on the vector information input together when the distance detection unit 102 inputs that there is no distance point 31 on the instruction vector 34. Then, the sensor driving unit 103 causes the distance sensor 13 to be displaced, such as rotation or movement, to change the measurement region 15 of the distance sensor 13 so that the distance sensor 13 can scan at least the instruction vector 34. The sensor driving unit 103 uses a distance sensor driving device (not shown) mounted on the robot when driving the distance sensor 13.

The three-dimensional information acquisition unit 110 inputs a plurality of distance information of the object from the distance detection unit 102, performs three-dimensional reconstruction, and outputs the three-dimensional information of the object to the operation control unit 111.
The action control unit 111 inputs the three-dimensional information of the object from the three-dimensional information acquisition unit 110 and performs an operation on the object based on the information, for example, an operation such as taking, pushing, and pulling. 16 is instructed. At this time, the operation control unit 111 may input and use other information, for example, an image recognition result of the target object. The operation control unit 111 controls the overall operation of the robot 11.

As the distance sensor 13, a stereo camera, a distance image sensor such as Kinect (registered trademark), a laser sensor such as TOF (Time Of Flight), or the like can be used. When the distance sensor 13 is a stereo camera, one of a plurality of cameras can be used as an HMI display camera.
The distance sensor driving device may be a movable part of the robot 11 or a movable head fixed to the environment.
The robot 11 is a mobile manipulator such as an HSR (Human Support Robot).

  Moreover, each component which the distance information acquisition apparatus 100 implement | achieves is realizable by making a program run by control of the arithmetic unit (not shown) with which the distance information acquisition apparatus 100 which is a computer is provided, for example. More specifically, the distance information acquisition device 100 is realized by loading a program stored in a storage unit (not shown) into a main storage device (not shown) and executing the program under the control of the arithmetic unit. . In addition, each component is not limited to being realized by software by a program, and may be realized by any combination of hardware, firmware, and software.

  The above-described program can be stored using various types of non-transitory computer readable media and supplied to a computer. Non-transitory computer readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (for example, flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (for example, magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / W and semiconductor memory (for example, mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (random access memory)) are included.

  Further, the program may be supplied to the computer by various types of temporary computer readable media. Examples of transitory computer readable media include electrical signals, optical signals, and electromagnetic waves. The temporary computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire and an optical fiber, or a wireless communication path.

Next, the operation of the distance information acquisition apparatus 100 according to the first embodiment will be described.
FIG. 4 is a flowchart showing a processing procedure of the distance information acquisition method executed by the distance information acquisition device 100 according to the first embodiment.
First, the user instructs a target object to the robot (step S10). Specifically, as shown in FIGS. 1A and 1B, the remote controller 17 as an object is imaged by the HMI display camera 12 mounted on the robot 11, and the image of the object is input to the HMI 21 by communication. The object is indicated by tapping on the display image of the HMI 21 by the user.

Next, the vector detection unit 101 detects the instruction vector 34 (step S20).
FIG. 5 is a diagram showing the instruction vector 34 detected by the vector detection unit 101 according to the first embodiment.
The camera viewpoint O for HMI display can be obtained as a three-dimensional coordinate from the design information of the robot 11 and the self-position information of the robot 11. Further, the relationship between the two-dimensional coordinate p (x, y) of the user instruction point 33 on the virtual image plane in the perspective projection model and the three-dimensional coordinate P (X, Y, Z) of the object 17 is expressed by Equations 1 and 2. It becomes as shown in. In the formula, f indicates the focal length, and D indicates the distance from the camera 12 to the object 17. In this case, although the distance D is unknown, the inclination v of the instruction vector 34 can be obtained from Equation 3, and the three-dimensional vector of the starting point position O and the inclination v can be calculated.

Next, the distance detection unit 102 determines whether or not the distance point 31 is on the instruction vector 34 (step S30).
FIG. 6 is a diagram illustrating a method in which the distance detection unit 102 according to the first embodiment detects distance information. The distance detection unit 102 sets a ray (line segment) starting from the camera viewpoint O in a three-dimensional space, and sets the measured distance point group as a sphere group of a certain size. Then, it can be determined whether there is a distance point on the instruction vector 34 by determining whether there is a sphere that collides with the ray by ray casting.
For Raycast, for example, “Measure distance by ray”, [online], [Search on January 28, 2014], Internet <https://www10.atwiki.jp/bambooflow/pages/239.html>, etc. It is described in.

When the distance detection unit 102 determines that there is no distance point 31 on the instruction vector 34 (No in step S30), the sensor driving unit 103 determines the position of the distance sensor 13 in order to change the measurement area of the distance sensor 13. And the direction are determined (step S40).
FIG. 7 is a diagram illustrating the position and direction of the distance sensor 13 determined by the sensor driving unit 103 according to the first embodiment. The sensor driving unit 103 samples a three-dimensional point on the instruction vector 34 to obtain a gaze point 35, and plans the neck angle of the robot 11 so that the optical axis of the distance sensor 13 faces the gaze point 35.

Next, distance information is measured at the determined position and direction of the distance sensor 13 (step S50).
Next, the distance detection unit 102 determines whether or not the distance point 31 is on the instruction vector 34 (step S60). The determination method at this time is the same as the determination method by ray casting performed in step S30, and a description thereof will be omitted.
Next, when the distance point 31 is on the instruction vector 34 (Yes in step S60), the distance detection unit 102 determines the object 3 based on the distance information of the distance point 31 and the position information of the distance sensor 13. The dimensional coordinates are acquired and notified to the three-dimensional information acquisition unit 110 of the robot 11 (step S70).

Then, the robot 11 controls the operation of the robot 11 based on the three-dimensional information of the object output from the three-dimensional information acquisition unit 110, and the operation unit 16 performs an operation on the object. Give a positive effect.
When the distance detection unit 102 determines that the distance point 31 is on the instruction vector 34 before changing the measurement area of the distance sensor 13 (Yes in step S30), the distance detection unit 102 displays the distance information. The three-dimensional coordinates of the target object are notified to the three-dimensional information acquisition unit 110 of the robot 11 (step S70).

  When the distance detection unit 102 determines that there is no distance point 31 on the instruction vector 34 after changing the measurement region of the distance sensor 13 (No in step S60), the sensor operation determination process (step S40) or the distance Returning to the measurement process (step S50), the detection of the distance point 31 on the instruction vector 34 is continued.

  As described above, the distance information acquisition method according to the first embodiment is a distance information acquisition method in which the distance to the target from which the distance information is acquired is measured by the distance sensor, and a vector indicating the target is detected in step S20. When there is no distance information from the distance sensor on the vector, the distance sensor changes the measurement area and measures the distance, and step S50 detects the distance information from the distance sensor on the vector. And step S70 for obtaining target distance information based on the distance information. With such a configuration, it is possible to reliably acquire the distance information of the target object, and it is possible to specify the three-dimensional information even when the user indicates an unobserved region of the distance sensor.

  In addition, the distance information acquisition method according to the first embodiment further includes step S10 in which the camera captures an object and step S10 in which the object on the virtual image plane of the camera is detected, and step S20 in which a vector is detected. Preferably, a vector connecting the camera viewpoint and the object on the virtual image plane is detected as a vector.

(Other Embodiments of the Invention)
The present invention is not limited to the first embodiment, and can be appropriately changed without departing from the spirit of the present invention.
For example, in the first embodiment, the camera 12 for displaying the HMI 21 is mounted on the robot 11, but this camera may be mounted on the HMI 21 or arranged in the environment. Even in this case, the instruction vector 34 can be detected from the camera by specifying the viewpoint O of the camera by a technique such as indoor GPS.

In the first embodiment, the vector detection unit 101 detects the instruction vector 34 based on the information of the camera 12 for displaying the HMI 21. The instruction vector may be detected based on the result recognized using such a method. In skeletal tracking, information such as the shoulder position, elbow position, and hand position is obtained as three-dimensional coordinates, so a vector from the shoulder position to the hand position can be calculated as an instruction vector.
For skeletal tracking, for example, "Kinect for Windows (registered trademark) SDK Introduction 4: Skeletal tracking and 3D display", [online], [Search January 28, 2014], Internet <http: //d.hatena .ne.jp / astrobot / 20110721/1311240780> and the like.

  In the first embodiment, when the measurement area of the distance sensor 13 is changed, the sensor drive unit 103 plans the neck angle of the robot so that the optical axis of the distance sensor 13 faces the gazing point. However, it is sufficient that the gazing point is captured at the center of the image. For example, the image center or the measurement range center of the distance sensor 13 may face the gazing point. In other words, the gazing point may be approximately at the center of the measurement area.

As described above, in the distance information acquisition method according to another embodiment, it is preferable that step S50 for measuring distance information changes the measurement region so that the point on the vector is substantially at the center of the measurement region. . With such a configuration, the distance information of the object can be acquired more reliably and easily.
In the distance information acquisition method according to another embodiment, it is preferable that the step S20 of detecting a vector detects a vector extracted from a gesture in which the user directly points to the target as a vector.

DESCRIPTION OF SYMBOLS 11 Robot 12 Camera 13 Distance sensor 15 Measurement area 16 Action part 31 Distance point 32 Virtual image surface 33 User indication point 34 Instruction vector 100 Distance information acquisition apparatus 101 Vector detection part 102 Distance detection part 103 Sensor drive part

Claims (7)

A distance information acquisition method for measuring a distance to a target for acquiring distance information with a distance sensor,
Detecting a vector pointing to the object;
When the distance information by the distance sensor is not on the vector, the distance sensor changes the measurement area and measures the distance;
Detecting distance information by the distance sensor on the vector;
A distance information acquisition method comprising: acquiring distance information of the target based on the detected distance information. The distance information acquisition method according to claim 1, wherein in the measuring step, the measurement area is changed so that a point on the vector is substantially at the center of the measurement area. A camera imaging the object;
Detecting the object on a virtual image plane of the camera,
Detecting the vector comprises:
The distance information acquisition method according to claim 1, wherein a vector connecting the viewpoint of the camera and the object on the virtual image plane is detected as the vector. Detecting the vector comprises:
The distance information acquisition method according to claim 1, wherein a vector extracted from a gesture directly pointing to the object by the user is detected as the vector. A vector detection unit for detecting a vector indicating a target for which distance information is acquired;
A distance information acquisition device comprising: a distance detection unit that detects distance information by a distance sensor on the vector and acquires distance information of the object based on the distance information;
A distance information acquisition apparatus further comprising: a sensor driving unit that drives the distance sensor so that the distance sensor changes a measurement region and measures the distance when there is no distance information on the vector on the vector. The distance information acquisition device according to claim 5;
A three-dimensional information acquisition unit that acquires the three-dimensional information of the target based on the distance information of the target;
A robot comprising: an action unit that applies a physical action to the object based on the three-dimensional information. On the computer,
A procedure for detecting a vector indicating an object for which distance information is to be obtained;
A procedure for driving the distance sensor so that the distance sensor changes the measurement area and measures the distance when there is no distance information by the distance sensor on the vector;
Detecting distance information by the distance sensor on the vector;
A distance information acquisition program for executing a procedure for acquiring the target distance information based on the distance information.