JP6512852B2 - Information processing apparatus, information processing method - Google Patents

Description

  The present invention relates to distance measurement technology.

  Heretofore, there have been vision systems for realizing so-called bulk picking of parts. For example, as in Patent Document 1, shooting data from various directions of parts as a target object is acquired in advance, and the target object is subjected to pattern matching with the input image and the shooting data acquired in advance. There is a method to determine the position / posture of However, this method has a disadvantage that it takes a very long time to detect if the size of the search area is significantly larger than the size of the part. In order to cope with such a defect, for example, as in Patent Document 2, the highest point in the group of parts piled up separately is found, and the parts search area is set around the point to obtain parts. A technique is disclosed that realizes speeding up of detection processing. The method of finding this high point makes sense from the purpose of holding the part using a robot. This is because if it is possible to detect a component at a high position, the component is likely to be on the surface, so that there is a high probability of being able to pick safely without interference with other components.

Patent No. 3300682 JP, 2011-133273, A

  However, the following problems existed in the prior art. That is, when measuring the distance to the parts group in a piled state, a part which is not originally at a high position may be selected due to the generation of measurement noise. More specifically, even if the location estimated to be high using the conventional method is found, it may actually be located at the location of the valley at the part loading state. As a result, there is a problem that gripping of a part detected from the location becomes impossible, and a pickable part can not be detected.

  In addition, when the number of parts has decreased, the location where the parts should not exist is increased by the distance measurement noise, and this may be set as a search area. As a result, there is also a problem that no part can be detected in the obtained part search area, or an object that is not a part is erroneously recognized as a part.

  The present invention has been made in view of such problems, and provides a technique for identifying an object located at a high position among piled objects at high speed and stably.

According to one aspect of the present invention, there is provided identification means for identifying an existence range of an object group based on a result of distance measurement on the object group.
When the existence range is divided into a plurality of small areas based on the size of the object , a part of the small areas is specified in ascending order of the distance value obtained by the distance measurement among the plurality of small areas, and And determining means for determining an area including the small area and having a size larger than the small area as the search area of the object for each small area of the part .

  According to the configuration of the present invention, an object located at a high position can be identified at high speed and stably from among a group of objects in a stacked state.

6 is a flowchart of processing performed by the information processing apparatus. FIG. 2 is a block diagram showing an example of the hardware configuration of an information processing apparatus. FIG. 8 is a view for explaining the process in step S103. The figure explaining step S111. FIG. 6 is a flowchart of processing performed by the information processing apparatus. The figure which shows parts stacking state.

  Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. The embodiment described below shows an example when the present invention is specifically implemented, and is one of the specific examples of the configuration described in the claims.

First Embodiment
In this embodiment, the existence range of the object group is specified based on the result of the distance measurement on the object group, and the existence range is divided into a plurality of small areas based on the size of the object. An example of an information processing apparatus that determines a search area of an object from the plurality of small areas will be described based on the result corresponding to.

  First, an example of the hardware configuration of the information processing apparatus according to the present embodiment will be described using the block diagram of FIG. For example, a general PC (personal computer) can be applied to the apparatus having the configuration of FIG. 2, but the present invention can be applied to any apparatus that can implement each process described below. I do not care. For example, the configuration shown in FIG. 2 may be incorporated into an apparatus such as an imaging apparatus. Further, the configuration of FIG. 2 may be configured by a plurality of devices capable of data communication with each other via a network.

  The CPU 201 executes processing using the computer program and data stored in the RAM 203 to perform operation control of the entire information processing apparatus, and executes or controls each process described later as being performed by the information processing apparatus. .

  The ROM 202 stores a boot program, setting data, and the like.

  The RAM 203 has an area for storing computer programs and data loaded from the ROM 202 and the secondary storage device 204, data entered from the I / O device 209, and the like. Furthermore, the RAM 203 has a work area used when the CPU 201 executes various processes. Thus, the RAM 203 can appropriately provide various areas.

  The secondary storage device 204 is a large-capacity information storage device such as a hard disk drive. The secondary storage device 204 stores an operating system (OS) 211 and computer programs and data for causing the CPU 201 to execute processing to be described later as being performed by the information processing apparatus. The computer program includes a module 213, an application 212, and the like. Also, this data includes data 214. The secondary storage device 204 also stores information handled as known information in the following description. The computer program and data stored in the secondary storage device 204 are appropriately loaded to the RAM 203 according to the control of the CPU 201, and are to be processed by the CPU 201.

  The display 206 is configured by a CRT, a liquid crystal screen, or the like, and is a device capable of displaying the processing result of the CPU 201 as an image, characters, or the like. Instead of the display 206, a device such as a projector that projects the processing result of the CPU 201 as an image or characters may be used.

  A keyboard 207 and a mouse 208 are devices functioning as a user interface that can input various instructions to the CPU 201 by the operation of the operator of the information processing apparatus.

  The I / O device 209 includes an apparatus for inputting various information to the information processing apparatus, and an apparatus for outputting various information from the information processing apparatus to an external apparatus. In the present embodiment, the I / O device 209 includes an apparatus capable of measuring at least a three-dimensional distance to each position within the distance measurement range.

  For example, if the I / O device 209 is a device that performs three-dimensional distance measurement (for example, a three-dimensional input device such as a TOF sensor device), the three-dimensional distance to each position within the distance measurement range of the device It can be measured.

  Also, for example, when the I / O device 209 is two stereo cameras, each of the two stereo cameras captures a captured image including the target object, and the CPU 201 is operated by each of the two stereo cameras. By applying triangulation technology using a captured image, it is possible to obtain a three-dimensional distance to each position in the space shown in the captured image.

  Also, for example, when the I / O device 209 includes one pattern light projector (projector) and one or more imaging devices, the pattern light projector may be a projection pattern (for example, a spatially encoded image or a phase). The spatial trigonometric function image used in the shift method is projected onto the target object, the imaging device captures an image of the target object onto which the projection pattern is projected, and the CPU 201 applies a known technique using the image. Thus, the distance to the target object can be measured. Further, for the same purpose, there is also a method of projecting a random dot pattern on a target object by a pattern light projector and photographing the target object by two imaging devices. Further, for the same purpose, a light cutting method using laser slit light may be used.

  In addition, the I / O device 209 transmits information related to the candidate position of the target object obtained by each processing described later by the information processing apparatus to an external device such as a robot for gripping the target object. It may be an apparatus.

  The CPU 201, the ROM 202, the RAM 203, the secondary storage device 204, the display 206, the keyboard 207, the mouse 208, and the I / O device 209 are all connected to the bus 205.

  Next, processing performed by the information processing apparatus to obtain a search candidate position for searching for the position of one object in the object group will be described using FIG. 1 showing a flowchart of the processing. Although the case where the object is a part will be described below, the following description can be applied similarly even if the object is another one.

<Step S101>
First, the CPU 201 acquires data representing a place where the parts group (parts stacked in bulk) is to be arranged as the parts absence state data 102. For example, data representing only a container called a pallet for stacking parts in bulk is acquired as the part-free state data 102. Various methods can be considered as a method of acquiring data representing only a container called a pallet as the componentless state data 102.

  For example, using the above-described pattern light projection device and imaging device (in this case, the I / O device 209 includes the pattern light projection device and the imaging device), the captured image of the container on which the pattern light is projected A distance image (three-dimensional distance in the distance image) which represents the three-dimensional distance (the three-dimensional distance from the imaging device to the position in space appearing at the pixel position) corresponding to each pixel position in the captured image is obtained. The pixel value at the pixel position (x, y) represents the three-dimensional distance from the imaging device to the position in the space shown at the pixel position (x, y) in the captured image as You may ask.

  Also, even if a sensor device such as a TOF sensor device is used instead of the pattern light projection device and the imaging device described above, the same distance image (the pixel value of each pixel represents the distance from the sensor device such as the TOF sensor device) The image can be acquired as part-less status data 102 (in this case, the I / O device 209 will have a sensor device such as a TOF sensor device).

  When the component group is arranged on an appropriate surface such as a desk instead of a pallet, data representing the surface is acquired as the component-free state data 102. Also in this case, by using the above method, it is possible to obtain a distance image in which the pixel value of each pixel represents the distance from the I / O device 209.

  In the following, the no-component state data 102 will be described as a distance image in which the pixel value of each pixel represents the distance from the I / O device 209.

  In this manner, the part absence state data 102 can be acquired by the I / O device 209. Since the I / O device 209 stores the component absence state data 102 thus acquired in the RAM 203 or the secondary storage device 204, the CPU 201 reads the component absence state data 102 stored in the RAM 203 or the secondary storage device 204. I will get it.

<Step S103>
Next, the CPU 201 acquires data representing the parts group (parts stacked in bulk) arranged in the above place as the piled state data 104. In step S101, when the distance image of the pallet is acquired as the part absence state data 102, in step S103, the distance image of the part group stacked in bulk on the pallet is acquired as the piled state data 104. That is, after the component group is arranged within the range where the I / O device 209 performs the distance measurement when acquiring the component absence status data 102, the piled status data 104 is performed in the same manner as the component absence status data 102. It is data acquired by the I / O device 209. Therefore, the stacking state data 104 is also a distance image in which the pixel value of each pixel represents the distance from the I / O device 209. Since the I / O device 209 stores the piled state data 104 acquired in this manner in the RAM 203 or the secondary storage device 204, the CPU 201 acquires the piled state data 104 stored in the RAM 203 or the secondary storage device 204. It will be.

  A process for acquiring the piled state data 104 using the I / O device 209 having the pattern light projection device and the imaging device will be described with reference to FIG. FIG. 3 (a) is a side view of the pallet 303 containing the stacked parts group 304, and FIG. 3 (b) is a top view of the pallet 303 containing the stacked parts group 304. It is a figure that I saw.

  As shown in FIG. 3A, the I / O device 209 is a projector for projecting a prescribed pattern light onto a target object (here, a component), and a camera for imaging the target object onto which the pattern light has been projected. And the CPU 201 can obtain the three-dimensional distance from the camera to each point on the target object by a known technique using an image captured by the camera. In FIG. 3A, reference numeral 302 denotes the angle of view (imaging range) of the camera.

  In addition, although the pallet 303 is an object which does not have the surface on the box of a rectangular parallelepiped as a typical form, the shape is not restricted to a rectangular parallelepiped, and what kind of shape may fundamentally be used. For example, the four sides of the bottom may be in the form of a square frustum shorter than the four sides of the upper frame. Also, the corners of the pallet may be round or the sides of the pallet may be curved.

<Step S105>
In the distance image represented by the piled state data 104, the CPU 201 determines a range in which the part will exist as the part existing range 106. For example, the pixel values of the positionally corresponding pixels are compared between the part absence state data 102 and the piled state data 104, and the pixel value of the pixel at the pixel position (x, y) of the part absence state data 102 <piled state data 104 If the pixel position (x, y) is the pixel value of the pixel at (x, y), it is determined that the pixel position (x, y) in the piled state data 104 is included in the component presence range 106. By performing such a determination for all pixel positions, it is possible to obtain an area in which the part is present in the piled state data 104.

  Note that in this step, another method may be used as long as it is possible to obtain an area that seems to be a part on the piled state data 104. For example, even if the background difference method is used to specify a pixel position where the pixel value differs from the component-less state data 102 by a prescribed value or more in the piled-up state data 104, a specified set of pixel positions is specified as the component existing range 106. I do not care. Alternatively, the area of the pallet in the part absence state data 102 may be identified, and the area on the piled state data 104 corresponding to the identified pallet area may be identified as the part existence range 106. In addition, a captured image captured by the imaging apparatus is displayed on the display 206 in order to obtain the piled state data 104, and a user who views the captured image displayed on the display 206 specifies an area designated using the keyboard 207 or the mouse 208. It may be specified as the existing range 106. In addition, the difference in pixel value is calculated between pixels corresponding in position by the captured image captured by the camera to obtain the component-free status data 102 and the captured image captured by the camera to obtain the stacked status data 104. Alternatively, an area on the piled state data 104 consisting of pixels for which the difference is equal to or greater than a threshold may be specified as the component presence range 106.

<Step S107>
Next, the CPU 201 estimates the size (estimated size) of a representative part in the part group 304 in a piled state, as viewed from the position of the I / O device 209, using the piled state data 104.

  For example, a representative value is obtained from the pixel values of the pixels belonging to the component presence range 106 in the piled state data 104. In this embodiment, an average value of pixel values of pixels belonging to the component presence range 106 in the piled state data 104 is used as a representative value. In FIG. 3A, reference numeral 305 denotes the distance represented by the representative value from the I / O device 209 (camera). The representative value may be obtained using another method, and statistical values such as the median value and the mode value of the pixel values of the pixels belonging to the component presence range 106 in the piled state data 104 may be used as the representative value.

  Next, the CPU 201 obtains “the size of the part that can be seen from the I / O device 209 when the part is disposed at a position separated from the I / O device 209 by the distance represented by the representative value” as the part projection size 108. For example, the component projection size 108 = R × f × L / D is determined from the length L of the long side of the component bounding box, the focal length f of the camera, the representative value D, and the resolution R of the camera. Also, using the model of the part created in advance, “an image of the model that can be seen from the I / O device 209 when the model is placed at a position separated from the I / O device 209 by the distance represented by the representative value” The orientation of the model may be generated while being changed to various orientations, and the average value or the maximum value of one side of the region of the model in each of the generated images may be obtained as the component projection size 108. For example, as a model, a CAD model representing the shape of a part can be used.

  That is, in step S107, “I / O is performed when the component is disposed at a position separated from the I / O device 209 by the distance represented by the representative value obtained from the pixel values of the pixels belonging to the component presence range 106 in the piled state data 104 If the size of the part that can be seen from the device 209 can be determined, the process for that is not limited to a specific process.

<Step S109>
Next, the CPU 201 creates a rough height map 110 using the piled state data 104 and the part projection size 108. The rough height map 110 is data representing a representative distance value in each small area when the component presence area 106 in the piled state data 104 is divided into a plurality of small areas (cells). The “representative distance value in the small area” is a statistical value such as the average value or the median value of the pixel values in the small area.

  Although the shape of the small area is not limited to a specific shape, the size thereof is a size based on the component projection size 108. For example, in the case of FIG. 3B, as indicated by 306, the small area is a square, and one side thereof has a size of the component projection size 108 or a constant multiple thereof. Further, for example, in the case of FIG. 5A, the small area is circular, and the center position is shifted for each row and has areas overlapping each other. Also, its diameter (or radius) may be the size of the part projection size 108 or a constant multiple thereof. Also, for example, in the case of FIG. 5 (b), the small regions are hexagonal, forming a honeycomb structure. Also, the diameter (or radius) of the circle circumscribing the small area has the size of the component projection size 108 or a constant multiple thereof. Note that one side of the small area may have a component projection size 108 or a size that is a constant multiple thereof.

  In general, if the size of a small area (which may be a square, a circle, or a hexagon) that divides the part existence range 106 is too small, an algorithm similar to that of Patent Document 2 is obtained, and it is affected by distance measurement noise. The identification of parts that are high (closer to the I / O device 209) becomes unstable. On the other hand, if the size of the small area that divides the part existence range 106 is too large, it is not possible to accurately evaluate the surface undulations of the part pile such as in the part group 304 of FIG. The part identification closer to the / O device 209 is inaccurate.

  On the other hand, in the present embodiment, in the case where the component presence range 106 in the piled state data 104 is divided into a plurality of small areas (cells), the size thereof is the component projection size 108 or a size that is a constant multiple thereof.

  As shown in FIG. 3A, in the state where the parts are stacked in the pallet 303, the distance 305 from the camera to the parts may change significantly. Especially when the pallet is deep and parts can be loaded in large quantities, the surface parts are present near the upper frame of the pallet when the piles of parts are replaced for the first time, but when the parts remain in the pallet, The distance between the camera and the part is approximately the distance between the camera and the bottom of the pallet. At this time, assuming that the distance 305 from the camera to the part is twice as long as originally, the part projection size 108 is halved. As described above, even when the part projection size changes to one half, the size of the above-described small area is fixed without changing as in the present embodiment, and based on this, the position is high in piled parts (I / If an attempt is made to identify a part that is closer to the O device 209, the identification will be inaccurate. On the other hand, according to the present embodiment, since the part existence range 106 is divided into small areas having the size optimum for the current piled state as a unit, stable and accurate identification (high position in piled parts (I Component identification (closer to the / O device 209) can be realized.

<Step S111>
The CPU 201 refers to the general height map 110 to specify, as a central small area, a small area having the smallest representative distance value among the small areas obtained by dividing the component presence range 106. Then, the CPU 201 specifies a small area group consisting of a central small area and a surrounding small area group surrounding the central small area as a component search area 112. The process in step S111 will be described with reference to FIG.

  FIG. 4A shows a top view of the pallet 401 in a state of accommodating the stacked parts group 402, and schematically showing a part 403 of the small area group represented by the general height map 110. It is shown superimposed on. In FIG. 4A, the small area 404 is selected as the central small area, and the small area group 405 including the small area 404 and the surrounding small areas surrounding the small area 404 is specified as the part search area. There is. The vertical and horizontal size of the component search area is, for example, four times the vertical and horizontal size of the small area.

  The size of the parts search area centered on the central small area is not limited to an integral multiple of the small area. For example, the component projection size 108 is determined using the average value of pixel values in the central small region instead of the distance 305, and the vertical and horizontal sizes of the component search area centered on the central small region are It may be a constant multiple.

  FIG. 4B shows a top view of the pallet 401 in a state of accommodating the stacked parts group 402, and schematically showing a part 403 of the small area group represented by the general height map 110. It is shown superimposed on. In FIG. 4B, the small area 406 is selected as the central small area, and the small area group 407 including the small area 406 and the surrounding small area group surrounding the small area 406 is specified as the part search area. There is. The vertical and horizontal size of the component search area is, for example, four times the vertical and horizontal size of the small area. In the parts search area (small area group 407), the part protruding from the pallet 401 may be deleted. At this time, the area of the part search area is smaller than that in the normal case.

  FIG. 4C shows a top view of the pallet 401 in a state of accommodating the stacked parts group 402, and schematically showing a part 403 of the small area group represented by the general height map 110. It is shown superimposed on. In FIG. 4C, although the small area 406 is selected as the central small area, the part search area (the small area group 407) is shifted so that the part protruding from the pallet can be pushed to the opposite side. A small area group 408) is set. The vertical and horizontal size of the component search area is, for example, four times the vertical and horizontal size of the small area. By moving the part search area itself as in the example of FIG. 4C, instead of deleting the part protruding from the pallet, the area of the part search area is always constant.

  Thus, in step S111, a small area having the smallest representative distance value is specified as a central small area among the small areas obtained by dividing the part presence range 106 with reference to the general height map 110, As long as a part search area including the central small area is set, the setting method of the part search area is not limited to a specific method.

  Although FIG. 4 shows an example in which the part search area is set as one rectangle, a plurality of rectangles may be selected as the part search area, or a circular area instead of the rectangle may be used as the part search area. When setting a plurality of places as part search areas, for example, the small areas are sorted in the order of small representative distance values, and a predetermined number of small areas from the top are selected as central small areas in the order of the smallest distance value. For each central small area, a parts search area including the central small area may be set. In addition, there is also a method of selecting a part search area so that the overlap does not overlap each other or the overlap is equal to or less than a predetermined ratio.

Second Embodiment
The present embodiment differs from the first embodiment in that the process according to the flowchart of FIG. 6 is executed instead of the process according to the flowchart of FIG. 1. In the following, differences from the first embodiment will be mainly described, and it is assumed that the second embodiment is the same as the first embodiment unless otherwise specified. Moreover, in FIG. 6, the same step number is attached | subjected to the process step same as the process step shown in FIG. 1, and the description regarding this process step is abbreviate | omitted.

  A part stacking condition suitable for applying the present embodiment is shown in FIG. 7, the same reference numerals as in FIG. 3 denote the same parts as in FIG. 3, and a description thereof will be omitted. In FIG. 7 as well as FIG. 3, the piled state data 104 for the component group 704 is acquired using the I / O device 209 having the pattern light projection device and the imaging device. FIG. 7 (a) is a side view of the pallet 703 containing the stacked parts group 704, and FIG. 7 (b) is a top view of the pallet 703 containing the stacked parts group 704. It is a figure that I saw.

  In FIG. 7, the difference between the closest point and the farthest point from the camera is larger than that in FIG. Specifically, as shown in FIG. 7B, in the small area group 706, the representative distance value in the small area 707 and the representative distance value in the small area 709 are largely different. As a result, the part projection size 708 and the part projection size 710 at the height at the point differ greatly.

  In the present embodiment, the part projection size 108 corresponding to each small region is recalculated by the method described in the first embodiment using the representative distance value determined for each small region, and The representative distance value determined for the small area is corrected based on the part projection size 108 recalculated for the small area. By performing this process, the information processing method of the present embodiment can cope with the case where the component projection size is largely different for each small area.

  Specifically, in step S611, using the representative distance value obtained for each small area, the component projection size 108 corresponding to each small area is recalculated by the method described in the first embodiment. And change the size of the small area to the part projection size 108 recalculated for the small area or a multiple thereof. Then, for each small area after resizing, a representative pixel value is obtained as a representative distance value for the small area from the pixel values in the small area. Thereafter, as the representative distance value of each small area, the representative distance value obtained for the small area in this step S611 is used.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. Can also be realized. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

  201: CPU 203: RAM 204: secondary storage device

Claims (10)

Specifying means for specifying an existence range of the object group based on a result of distance measurement on the object group;
When the existence range is divided into a plurality of small areas based on the size of the object , a part of the small areas is specified in ascending order of the distance value obtained by the distance measurement among the plurality of small areas, and An information processing apparatus comprising: determination means for determining an area having a size larger than the small area including the small area as the search area of the object for each small area of the part . The determining means is
The representative value of the distance value obtained by measuring the distance to the object group is determined, and the size of the object seen from the device when the object is arranged at a position separated from the device for performing the distance measurement by the representative value is determined When the existence range is divided into a plurality of small areas based on the determined size, a part of the small areas is specified in ascending order of the distance value among the plurality of small areas, and each of the partial areas is selected. the information processing apparatus according to claim 1, characterized in that determining the small region a region of greater size than unrealized and the small area as the search area. The determining means is
The size of each of the plurality of small areas is corrected according to the distance value corresponding to the small area, and after the correction, a part of the small areas is specified in the order of smaller distance value among the plurality of small areas. , wherein each of some small areas, the information processing apparatus according to claim 2, characterized in that to determine the regions of a size larger than and the small area viewed including the said small regions as the search area.   The information processing apparatus according to any one of claims 1 to 3, wherein the determination unit divides the existing range into small areas or small areas each having a size which is a constant multiple of the size. The identification means is
The present range is specified using a result of distance measurement performed in a state where the object group is not arranged and a result of distance measurement performed in a state where the object group is arranged. An information processing apparatus according to any one of 1 to 4.   The information processing apparatus according to any one of claims 1 to 5, wherein a size of the search area is a constant multiple of a size of the small area.   The size of the search area is a constant multiple of the size of the object seen from the device when the object is placed at a position separated from the device by a distance value corresponding to the specified small region. The information processing apparatus according to Item 2 or 3. Furthermore,
The information processing apparatus according to any one of claims 1 to 7, wherein information indicating a search area determined by the determination means is output to an apparatus for operating the object. An information processing method performed by the information processing apparatus;
A specifying step of specifying the presence range of the object group based on the result of the distance measurement on the object group, the specification means of the information processing apparatus;
When the determination unit of the information processing apparatus divides the existing range into a plurality of small areas based on the size of the object, a part of the plurality of small areas is in the order of small distance value obtained by the distance measurement And determining, for each of the partial areas, an area including the small area and having a size larger than that of the small area as the search area of the object. Information processing method.   A computer program for causing a computer to function as each means of the information processing apparatus according to any one of claims 1 to 8.

Images