WO2021044751A1

WO2021044751A1 - Information processing device, information processing method, and information processing program

Info

Publication number: WO2021044751A1
Application number: PCT/JP2020/028134
Authority: WO
Inventors: 清和宮澤
Original assignee: ソニー株式会社
Priority date: 2019-09-02
Filing date: 2020-07-20
Publication date: 2021-03-11

Abstract

The information processing device according to the present disclosure is provided with: a prediction unit that predicts, on the basis of image information obtained by photographing a target object which is a candidate for an operation target and an adjacent object which is adjacent to the target object, a change related to an arrangement state of the adjacent object caused by an operation with respect to the target object; and an execution unit that executes a process for operating the adjacent object when the change in the arrangement state of the adjacent object predicted by the prediction unit satisfies a predetermined condition.

Description

Information processing equipment, information processing methods and information processing programs

This disclosure relates to an information processing device, an information processing method, and an information processing program.

Technology related to autonomous robots that autonomously clean up articles (objects) and change the placement position of objects according to the situation is known. For example, by recognizing an object to be operated by an electronic tag or the like, an operation on the object is executed (Patent Document 1).

Japanese Unexamined Patent Publication No. 2004-249389 Japanese Unexamined Patent Publication No. 2005-88137

According to the prior art, when changing the arrangement position of an object, the installation position when another article exists at the specified installation position is corrected.

However, in the above-mentioned prior art, the installation position is only corrected when another article exists in the changed installation position, and the object in which the adjacent object exists in the state before the change is operated as a target. The case is not considered. Therefore, in the above-mentioned conventional technique, there is a possibility that an object that is difficult to operate at that time, such as an object that needs to operate an adjacent object first, may be an operation target.

Therefore, the present disclosure proposes an information processing device, an information processing method, and an information processing program that enable appropriate operation on an object even when an adjacent object exists.

In order to solve the above problems, in one form of the information processing apparatus according to the present disclosure, an object object which is a candidate object to be operated and an adjacent object which is an object adjacent to the object object are imaged. When the prediction unit that predicts the change in the arrangement state of the adjacent object caused by the operation on the target object based on the image information and the change in the arrangement state of the adjacent object predicted by the prediction unit satisfy a predetermined condition. , An execution unit that executes a process of manipulating the adjacent object.

It is a figure which shows an example of information processing which concerns on 1st Embodiment of this disclosure. It is a figure which shows an example of the determination of the operable object which concerns on 1st Embodiment. It is a figure which shows the structural example of the robot apparatus which concerns on 1st Embodiment. It is a figure which shows an example of the threshold value information storage part which concerns on 1st Embodiment. It is a figure which shows an example of the density information storage part which concerns on 1st Embodiment. It is a flowchart which shows the procedure of information processing which concerns on 1st Embodiment. It is a flowchart which shows the procedure of information processing which concerns on 1st Embodiment. It is a figure which shows an example of the conceptual diagram of the structure of a robot. It is a figure which shows an example of the process of the Nth-order object operation. It is a figure which shows the structural example of the robot apparatus which concerns on 2nd Embodiment of this disclosure. It is a figure which shows an example of information processing which concerns on 2nd Embodiment. It is a figure which shows the structural example of the robot apparatus which concerns on 3rd Embodiment of this disclosure. It is a figure which shows an example of information processing which concerns on 3rd Embodiment. It is a figure which shows the structural example of the information processing system which concerns on the modification of this disclosure. It is a figure which shows the structural example of the information processing apparatus which concerns on the modification of this disclosure. It is a hardware block diagram which shows an example of the computer which realizes the function of a robot device and an information processing device.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. The information processing apparatus, information processing method, and information processing program according to the present application are not limited by this embodiment. Further, in each of the following embodiments, duplicate description will be omitted by assigning the same reference numerals to the same parts.

The present disclosure will be described according to the order of items shown below.
1. 1. First Embodiment 1-1. Outline of information processing according to the first embodiment of the present disclosure 1-1-1. Processing example 1-1-2. Manipulable object judgment 1-1-3. Manipulation of object group 1-1-4. Outline and effects of robot devices 1-2. Configuration of the robot device according to the first embodiment 1-3. Information processing procedure according to the first embodiment 1-3-1. Flowchart showing the outline of the information processing procedure 1-3-2. Flowchart showing details of information processing procedure 1-4. Conceptual diagram of the configuration of the information processing device 1-5. Processing example of Nth object operation 2. Second Embodiment 2-1. Configuration of the robot device according to the second embodiment of the present disclosure 2-2. Outline of information processing according to the second embodiment 3. Third Embodiment 3-1. Configuration of the robot device according to the third embodiment of the present disclosure 3-2. Outline of information processing according to the third embodiment 4. Other Embodiments 4-1. Other configuration examples 4-2. Others 5. Effect of this disclosure 6. Hardware configuration

[1. First Embodiment]
[1-1. Outline of information processing according to the first embodiment of the present disclosure]
FIG. 1 is a diagram showing an example of information processing according to the first embodiment of the present disclosure. The information processing according to the first embodiment of the present disclosure is realized by the robot device 100 shown in FIG.

The robot device 100 is an information processing device that executes information processing according to the first embodiment. The robot device 100 is an autonomous robot that has a moving unit 15 having a function for moving a position and can move to a desired position. Further, the robot device 100 has two operation units (manipulators), a first operation unit 16a and a second operation unit 16b. In the following, when the first operation unit 16a and the second operation unit 16b are described without distinction, they may be described as "operation unit 16". The number of operation units 16 included in the robot device 100 is not limited to two, and may be one or three or more. Details of this point will be described later.

The robot device 100 is an information processing device that executes a process of manipulating an object based on image information (also simply referred to as an "image") detected (imaged) by an image sensor 141 (see FIG. 3). The robot device 100 selects a target object as a candidate for an operation target from the objects in the image, predicts a change in the arrangement state of the adjacent object caused by the operation on the target object, and determines the predicted change in the arrangement state of the adjacent object. When the condition of is satisfied, the process of manipulating the adjacent object is executed. In the example of FIG. 1, an object in contact with the target object will be described as an example of an object adjacent to the target object, but the adjacent object is not limited to an object in contact with the target object. It may be an object located within a predetermined range from. For example, the adjacent object may be an object located within the range affected by the removal of the target object. For example, when the target object is a magnetic object, the adjacent object may be an object located within a range affected by the magnetism of the target object. Further, in the example of FIG. 1, the case where the removal of the target object is taken as an example of the operation on the target object and the change regarding the arrangement state of the adjacent object caused by the removal of the target object is predicted will be described. The operation on the target object is not limited to removing (removing) the target object, and various operations that can affect the adjacent object, such as changing the position of the target object and changing the posture of the target object, are performed. It is a concept that includes operations.

[1-1-1. Processing example]
From here, using FIG. 1, a case where processing is performed on a group of objects in which a plurality of objects are stacked will be described as an example. FIG. 1 shows a process for ST1 in which three objects, OB1, OB2, and OB3, which are books, are stacked. The robot device 100 takes an image of the state ST1 by the image sensor 141, and acquires an image (hereinafter, may be referred to as “image IM1”) showing the state ST1 in which the objects OB1, OB2, and OB3 are stacked. The robot device 100 identifies that the image IM1 includes the objects OB1, OB2, and OB3 by analyzing the image IM1 by a technique such as image analysis.

First, the robot device 100 selects an object (target object) that is a candidate for the operation target (step S1). For example, the robot device 100 randomly selects a target object from a group of objects OB1, OB2, and OB3. In the example of FIG. 1, the robot device 100 selects an object OB2 (hereinafter, also referred to as “target object OB2”) as a target object from a group of objects OB1, OB2, and OB3. The robot device 100 selects an object whose weight is estimated to be less than a predetermined threshold value as an operation target, and this point will be described with reference to FIG. The robot device 100 recognizes a physical contact state between an object (adjacent object) around the target object OB2 and the target object OB2. By analyzing the image IM1, the robot device 100 recognizes that the target object OB2 is in contact with the object OB1 and the object OB3.

Then, the robot device 100 removes the target object (step S2). In the example of FIG. 1, the robot device 100 removes the target object OB2. The robot device 100 removes the target object OB2 from the image IM1 showing the state ST1 in which the objects OB1, OB2, and OB3 are stacked. By removing the target object OB2 from the image IM1, the robot device 100 performs processing on the state ST2 in which only the target object OB2 is removed from the object group of the objects OB1, OB2, and OB3.

The robot device 100 predicts a change in the arrangement state of adjacent objects when the target object is removed (step S3). The robot device 100 predicts changes in the posture and position of adjacent objects when the target object is removed. The robot device 100 predicts the posture and position of an adjacent object by appropriately using various techniques related to physics simulation. For example, the robot device 100 estimates the center of gravity from the shape data (W [width], D [depth], H [height]) of each object based on the image, and detects the direction of gravity. Then, the robot device 100 predicts the posture and position of the adjacent object by the built-in physical model simulator. The above is an example, and the robot device 100 may perform prediction by any information or method as long as the posture and position of the adjacent object when the target object is removed can be predicted.

For example, the robot device 100 predicts the posture amount (posture change prediction value) and the position change amount (position change prediction value) of an adjacent object when the target object is removed. In the example of FIG. 1, the robot device 100 predicts a change in the arrangement state of the objects OB1 and OB3 when the target object OB2 is removed. The robot device 100 predicts changes in the posture and position of the object OB1 and changes in the posture and position of the object OB3 when the target object OB2 is removed.

Since the object OB1 is located below the target object OB2, the robot device 100 predicts that the position and posture of the object OB1 will not change due to the removal of the target object OB2. Since the robot device 100 is in a state where the object OB1 supports the target object OB2, it is predicted that the position and posture of the object OB1 will not be changed by removing the target object OB2. For example, the robot device 100 predicts that the posture change predicted value and the position position change predicted value of the object OB1 when the target object OB2 is removed are 0.

Further, since the object OB3 is located on the target object OB2, the robot device 100 predicts that the position and posture of the object OB3 will change due to the removal of the target object OB2. Since the robot device 100 is in a state where the object OB3 is supported by the target object OB2, it is predicted that the position and posture of the object OB3 will change due to the removal of the target object OB2. Further, the robot device 100 predicts the amount of change in the position and posture of the object OB3 due to the removal of the target object OB2. For example, the robot device 100 predicts the posture change predicted value and the position position change predicted value of the object OB1 when the target object OB2 is removed. In this way, when the target object OB2 is removed from the image IM1, the robot device 100 predicts that the object OB3 of the adjacent object has a change in posture or position as shown in the state ST3.

Then, the robot device 100 determines whether the movement of the surrounding object of the adjacent object is equal to or higher than the threshold value (step S4). The robot device 100 determines whether the change in the posture or position of the adjacent object is equal to or greater than the threshold value. For example, the robot device 100 uses a posture-related threshold value (posture threshold value) and a position-related threshold value (position threshold value) stored in the storage unit 12 (see FIG. 4) when the posture or position change of an adjacent object is equal to or greater than the threshold value. Determine if there is.

The robot device 100 executes a process related to the operation based on the determination result (step S5). The robot device 100 compares the predicted value of the posture change of the adjacent object (posture change predicted value) caused by the removal of the target object with the posture threshold, and when the posture change predicted value is equal to or more than the posture threshold, the adjacent object Executes the process of operating. Further, the robot device 100 compares the predicted value (position change predicted value) of the position change amount of the adjacent object caused by the removal of the target object with the position threshold value, and when the position change predicted value is equal to or more than the position threshold value, Executes the process of manipulating adjacent objects. The robot device 100 may execute a process of operating an adjacent object when one of the posture change or the position change becomes the threshold value or more as described above, and both the posture change and the position change become the threshold value or more. If so, the process of manipulating the adjacent object may be executed. Further, the robot device 100 compares the amount of change (combined change amount) obtained by combining the posture change and the position change with a predetermined threshold value (threshold value TH2 or the like in FIG. 4), and the combined change amount is equal to or larger than the predetermined threshold value. In that case, the process of manipulating the adjacent object may be executed.

In the example of FIG. 1, in the robot device 100, the posture change predicted value of the object OB1 caused by the removal of the target object OB2 is less than the posture threshold value, and the posture change predicted value of the object OB1 is less than the posture threshold value. Do not execute the operation.

Further, in the example of FIG. 1, since the posture change predicted value of the object OB3 generated by the removal of the target object OB2 is equal to or more than the posture threshold value, the robot device 100 executes a process of operating the object OB3. For example, in the robot device 100, the first operation unit 16a executes the operation of the target object OB2, and the second operation unit 16b executes the operation of the object OB3. For example, in the robot device 100, the first operation unit 16a executes the operation of the target object OB2, and the second operation unit 16b executes the operation of supporting the object OB3 which is an adjacent object. In this way, the robot device 100 executes a process of suppressing the change in the arrangement state of the adjacent object OB3 due to the movement of the target object in the second operation unit 16b. The robot device 100 executes a process of moving the target object OB2 to the first operation unit 16a. In this case, the robot device 100 may perform an operation of driving the second operation unit 16b and arranging the object OB3 at a stable position after the operation such as moving the target object OB2.

Further, the robot device 100 executes an operation of holding the target object OB2 by the first operation unit 16a, an operation of holding the object OB3 by the second operation unit 16b, and an object to a desired position by the moving unit 15. The object OB3 may be carried together with the OB2. The robot device 100 may move the target object OB2 to the first operation unit 16a and move the object OB3 which is an adjacent object to the second operation unit 16b.

As described above, the robot device 100 executes a process of manipulating the target object and the adjacent object based on the change in the arrangement state of the adjacent object adjacent to the target object when the target object is removed. In this way, the robot device 100 can enable an appropriate operation on an object even when an adjacent object exists.

[1-1-2. Manipulable object judgment]
From here, the outline of the determination of the operable object will be described with reference to FIG. FIG. 2 is a diagram showing an example of determination of a manipulable object according to the first embodiment. The robot device 100 determines an operable object based on the image.

The robot device 100 detects (images) an image including an unknown object operation group (unknown object group SG1) by the image sensor 141 (step S11). The robot device 100 recognizes a plurality of unknown object operation groups (unknown object group SG1) based on the image detected by the image sensor 141. In the example of FIG. 2, the robot device 100 recognizes the unknown object group SG1 including the bookshelf and a plurality of stored books.

The robot device 100 first segments each object from the accumulated image information of a plurality of unknown objects. The robot device 100 segmentes the unknown object group SG1 included in the image detected by the image sensor 141 by appropriately using various techniques related to image segmentation. In the example of FIG. 2, the robot device 100 includes a book group SG11 including a plurality of books such as objects OB11 to OB17, a book group SG12 including a plurality of books, a book group SG13 including a plurality of books, an object OB10 which is a bookshelf, and the like. In addition, the unknown object group SG1 is segmented. The robot device 100 segmentes the book group SG11 into each of the objects OB11 to OB17, and also segments the book group SG12 and the book group SG13 into each book.

Then, the robot device 100 classifies the unknown object group SG1 (step S12). The robot device 100 uses a threshold value such as "carrying weight of the manipulator" (such as the value "Wlood" of the threshold value TH1 in FIG. 4), and the object group G0 that cannot be moved by the external force of the threshold value and the object group G0 within the threshold value. The unknown object group SG1 is classified (classified) into one of the object group G1 that can be moved by an external force.

The robot device 100 compares the weight of each object included in the unknown object group SG1 with the "Wlood", and classifies the object having a weight exceeding the "Wlood" into the object group G0 as an inoperable object. Further, the robot device 100 compares the weight of each object included in the unknown object group SG1 with the "Wlood", and classifies the object having a weight of "Wlood" or less as a manipulable object into the object group G1. ..

Here, since the mass of each object included in the unknown object group SG1 is unknown information, the robot device 100 estimates from the image data. The robot device 100 recognizes each object of the unknown object group SG1 included in the image detected by the image sensor 141 by appropriately using various techniques related to object recognition such as general object recognition. The robot device 100 estimates the shape data (W [width], D [depth], H [height]) of the object extracted from the image. Further, the robot device 100 estimates the material of the object and estimates the density ρ of the object. Then, the robot device 100 calculates the estimated weight of the object by using the estimated shape data of the object (W [width], D [depth], H [height]) and the density ρ of the object. For example, the robot device 100 calculates the estimated weight "Wp" by multiplying the width, depth, height and density of the object. The robot device 100 calculates the estimated weight "Wp" by the following formula (1).

Wp = ρWDH ... (1)

If the density cannot be estimated, the robot device 100 holds the average density of the environment as data in advance, and estimates the weight using the average density. For example, the robot device 100 uses the value “VL1” of the environmental average density DS1 stored in the density information storage unit 122 (see FIG. 5) instead of the estimated density “ρ” of the object to determine the weight of the object. presume.

Then, the robot device 100 compares the calculated estimated weight "Wp" of each object with the threshold value "Wlood", determines whether or not each object can be operated, and sets the object group G1 capable of operating each object. It is classified as one of the inoperable object group G0.

In the example of FIG. 2, the robot device 100 estimates the weight of each book such as the objects OB11 to OB17 included in the book group SG11 to SG13, and determines that the estimated weight is equal to or less than the threshold value “Wlood”. As a result, the robot device 100 classifies each book such as the objects OB11 to OB17 included in the book group SG11 to SG13 into the operable object group G1.

Further, the robot device 100 estimates the weight of the object OB10 which is a bookshelf, and determines that the estimated weight is larger than the threshold value "Wlood". As a result, the robot device 100 classifies the object OB10, which is a bookshelf, into the inoperable object group G0. Then, the robot device 100 selects a target object to be operated from the objects belonging to the object group G1.

The robot device 100 targets only the object group of the object group G1, but when there are few objects belonging to the object group G1, the robot device 100 requests support from another robot or cooperates with a plurality of manipulators. May be good. When the number of objects belonging to the object group G1 is small, the robot device 100 may select an object belonging to the object group G0 as an operation target, request support from another robot, or perform cooperative work with a plurality of manipulators.

For example, when the number of objects belonging to the object group G1 is less than a predetermined reference value, the robot device 100 selects an object belonging to the object group G0 as an operation target, requests support from another robot, or has a plurality of objects. You may work together with a manipulator. For example, when all the objects belonging to the object group G1 cannot be operated due to the change condition of the adjacent object, the robot device 100 selects the object belonging to the object group G0 as the operation target and requests support from other robots. Or you may work together with multiple manipulators.

Further, as described above, the robot device 100 may autonomously select the operation target object, or may select the operation target object according to a human instruction such as the administrator of the robot device 100. .. For example, when there is a person who can determine whether or not the robot device 100 can be operated remotely or on the spot, the robot device 100 may acquire information indicating whether or not the object can be operated from the outside. Further, even if the robot device 100 stores in advance the operationability information indicating whether or not each object can be operated as knowledge in the storage unit 12, and determines whether or not the operation is possible based on the operationability information stored in the storage unit 12. Good.

[1-1-3. Manipulation of objects]
Next, an outline of a method of manipulating a group of objects that physically interact with each other will be described. The details of the flow for manipulating the objects in physical contact with each other will be described in detail in FIG. The robot device 100 analyzes motion from image data and operates a group of objects that physically interact with each other.

For example, if there is an object B that is in contact with the object A while being supported by its own weight, when trying to operate the object A, the position and posture of the object B that is being supported by the object A and maintaining its posture collapses, and the object B may fall down, fall or be damaged, or affect the surrounding environment. In order to suppress the occurrence of such an event, the robot device 100 predicts the movement of the object B in advance. First, the robot device 100 analyzes in advance what kind of operation other objects (object B, etc.) in contact with the object (object B, etc.) behave when the object A to be operated is removed, and the adjacent object. Predict the movement and posture of (object B). At this time, the robot device 100 estimates the center of gravity from the shape data (W [width], D [depth], H [height]) of each object, detects the direction of gravity, and uses the built-in physical model simulator to detect the object. Predict the posture and position of B. When the predicted movement of the adjacent object (object B) converges within the threshold value, the object A is determined to be operable.

If the object operation is selected depending on whether the prediction operation of the adjacent object is within the threshold value, the operation may not be possible. In such a case, the robot device 100 may operate the target object while supporting and fixing an adjacent object such as a large-moving object by another manipulator (operation unit 16) so as not to move. The robot device 100 is not only simply fixed, but is also supported by two or more manipulators (operation units 16), temporarily gripped, handed, and the like. , The role of object operation may be divided between the operation units 16. Further, the robot device 100 divides the role of object operation among the robot devices by performing cooperative work such as supporting each other, temporarily grasping, and handing the robot device in cooperation with other robot devices. May be good.

Further, when the robot device 100 is provided with a camera (image sensor) in the operation unit 16 (end effector unit of the manipulator), the posture and position of the object can be predicted with reference to the image from the end effector unit. Therefore, the robot device 100 avoids occlusion and enables more accurate prediction and determination of posture and position.

[1-1-4. Outline and effects of robot equipment]
As described above, the robot device 100 is movable and includes an operation unit 16 (arm, hand), a moving unit 15 (vehicle, etc.), and a recognition unit (image sensor 141 as a visual sensor, force-tactile sensor 142, etc.). It is a robot and operates an unknown object by the above-mentioned flow.

Specifically, the robot device 100 determines whether or not the manipulator can operate an object for a group of objects in contact with each other by using image information, and distinguishes between an operable object and an inoperable object. To do. Further, the robot device 100 extracts the operation target object from the group of objects in contact with each other, excludes the operation target object from the image, and determines the posture and position of the object in contact with the operation target object after the operation target object is removed. Predict. Then, when the change in the posture or position of the object in contact with the operation target object is equal to or greater than the threshold value, the robot device 100 executes an operation on the object in contact with the operation target object or changes the operation target. Further, the robot device 100 enables the operation target object to be operated when the change in the posture or position of the object in contact with the operation target object is less than the threshold value.

When a plurality of manipulators (operation units 16) are provided, the robot device 100 grips and controls the operation unit 16 so that the posture of the object in contact with the operation target object does not move. When the robot device 100 includes a plurality of operation units 16, one operation unit 16 executes an operation on an operation target object, and another operation unit 16 executes an operation on an object in contact with the operation target object.

Further, the robot device 100 may select the operation target object after contacting the surface of the target object with the robot hand (operation unit 16) with a force within the threshold value and confirming whether or not the object has a moving portion. .. The details of this point will be described later.

As described above, the robot device 100 can stably operate the target object from the stacked objects such as OB1 to OB3 without changing the position and orientation of the peripheral objects. The robot device 100 can stably operate a specific object from a plurality of stacked unknown object groups without changing the position and orientation of peripheral objects. For example, the robot device 100 can stably operate a specific object from among objects whose physical characteristics such as mass and friction coefficient are unknown without changing the position and orientation of peripheral objects.

In this way, the robot device 100 can move in a space such as a room, can autonomously clean up the room, and smoothly executes the operation even when the information on the space or the object is not known. be able to.

Conventionally, for example, by recognizing or specifying an object to be operated by a non-contact tag or the like, detailed information such as the shape, type, position, and posture of the object can be referred to a database prepared in advance. Then, the processing for the object is executed. In this case, the robot operates or moves the object by using the information such as the parameters of the target object stored in the database.

However, in the above case, it is difficult to operate on an object whose physical parameters are unknown, and not all objects including unknown objects can be operated. Since an autonomous robot is required to be able to operate autonomously in all situations and environments, it is desired that the robot can be operated appropriately when it is placed in an unknown environment.

Therefore, the robot device 100 recognizes the object from the image, considers the influence on the adjacent object, and operates the object. Therefore, the robot device 100 can operate on any object including an unknown object, and can operate in consideration of the influence on other objects.

As described above, the robot device 100 can autonomously move and operate an object with unknown physical parameters in an unstructured unknown environment. As a result, the robot device 100 does not require a database or the like and can be operated in various environments. In addition, the labor and cost of creating a database are not required.

Further, since the robot device 100 predicts the posture and position of surrounding objects before the operation, it is possible to reduce the risk of the objects falling or being damaged due to the operation. Further, since the number of objects and environments that can be operated by the robot device 100 without detailed instructions by humans increases, the autonomy of the robot device 100 can be enhanced and the productivity can be improved.

[1-2. Configuration of Robot Device According to First Embodiment]
Next, the configuration of the robot device 100, which is an example of the information processing device that executes the information processing according to the first embodiment, will be described. FIG. 3 is a diagram showing a configuration example of the robot device 100 according to the first embodiment.

As shown in FIG. 3, the robot device 100 includes a communication unit 11, a storage unit 12, a control unit 13, a sensor unit 14, a moving unit 15, a first operation unit 16a, and a second operation unit 16b. Has.

The communication unit 11 is realized by, for example, a NIC (Network Interface Card), a communication circuit, or the like. The communication unit 11 is connected to the network N (Internet, etc.) by wire or wirelessly, and transmits / receives information to / from other devices via the network N.

The storage unit 12 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory (Flash Memory), or a storage device such as a hard disk or an optical disk. The storage unit 12 has a threshold information storage unit 121 and a density information storage unit 122. The storage unit 12 is not limited to the threshold information storage unit 121 and the density information storage unit 122, and various types of information are stored. The storage unit 12 may store various information related to the operation unit 16. For example, the storage unit 12 may store information indicating the number of operation units 16 and the installation position of the operation unit 16. For example, the storage unit 12 may store various types of information used for identifying (estimating) an object.

The threshold information storage unit 121 according to the first embodiment stores various information related to the threshold value. The threshold information storage unit 121 stores various information related to the threshold used for various determinations. FIG. 4 is a diagram showing an example of the threshold information storage unit according to the first embodiment. The threshold information storage unit 121 shown in FIG. 4 includes items such as "threshold ID", "target", "use", and "threshold".

"Threshold ID" indicates identification information for identifying the threshold value. “Object” indicates an object for which a threshold value is used. "Use" indicates the use of the threshold. The "threshold value" indicates a specific value of the threshold value identified by the corresponding threshold ID.

In the example of FIG. 4, the threshold value (threshold value TH1) identified by the threshold value ID “TH1” indicates that the target is “weight”. The threshold value TH1 indicates that the threshold value is used for determining the weight of the object. The use of the threshold value TH1 is for determining a portable object, and indicates that the object is used for determining whether or not the object can be operated by the robot device 100. Further, the value of the threshold value TH1 indicates that it is "Wroad". In the example of FIG. 4, although it is indicated by an abstract code such as “Wlood”, the value of the threshold value TH1 is a specific numerical value.

Further, the threshold value (threshold value TH2) identified by the threshold value ID "TH2" indicates that the target is the "positional posture". The threshold value TH2 indicates that the threshold value is used for determining the position and orientation of the object. The use of the threshold value TH2 is to change an adjacent object, and it is shown that the threshold value TH2 is used for determining a change in the arrangement state of the adjacent object due to an operation on the target object. Further, the value of the threshold value TH2 indicates that it is "PVL". In the example of FIG. 4, although it is indicated by an abstract reference numeral such as “PVL”, the value of the threshold value TH2 is a specific numerical value.

The threshold information storage unit 121 is not limited to the above, and may store various information depending on the purpose. The threshold information storage unit 121 may store a threshold value related to posture (posture threshold value) and a threshold value related to position (position threshold value). In this case, the robot device 100 compares the predicted value (posture change predicted value) of the posture change amount of the adjacent object caused by the operation on the target object with the posture threshold, and the posture change predicted value is equal to or more than the posture threshold. , Executes the process of manipulating adjacent objects. Further, the robot device 100 compares the predicted value (position change predicted value) of the position change amount of the adjacent object caused by the operation with respect to the target object and the position threshold value, and when the position change predicted value is equal to or more than the position threshold value, Executes the process of manipulating adjacent objects.

The density information storage unit 122 according to the first embodiment stores various information related to the density. The density information storage unit 122 stores various information related to the density used for estimating the weight of the object. FIG. 5 is a diagram showing an example of the density information storage unit according to the first embodiment. The density information storage unit 122 shown in FIG. 5 includes items such as "density ID", "density name", "use", and "density".

"Density ID" indicates identification information for identifying the density. “Object” indicates an object of density. The "value" indicates a specific value of the density identified by the corresponding density ID.

In the example of FIG. 5, the density (density DS1) identified by the density ID "DS1" indicates that the target is the "environmental average". Density DS1 is the average density of objects in the global environment and indicates the density applied to each object in the environment. Further, the value of the density DS1 indicates that it is "VL1". In the example of FIG. 4, it is indicated by an abstract code such as "VL1", but the value of the density DS1 is a specific value such as "3 (g / cm ³ )" or "4 (g / cm ³ )". It is a numerical value.

The density information storage unit 122 is not limited to the above, and may store various information depending on the purpose. The density information storage unit 122 may store information indicating the density of each object. The density information storage unit 122 may store the average density of each object in association with each other.

Return to Fig. 3 and continue the explanation. In the control unit 13, for example, a program (for example, an information processing program according to the present disclosure) stored inside the robot device 100 by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like is a RAM (Random Access Memory). It is realized by executing such as as a work area. Further, the control unit 13 may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

As shown in FIG. 3, the control unit 13 includes an acquisition unit 131, an analysis unit 132, a classification unit 133, a selection unit 134, a prediction unit 135, a determination unit 136, a planning unit 137, and an execution unit 138. And realizes or executes the functions and actions of information processing described below. The internal configuration of the control unit 13 is not limited to the configuration shown in FIG. 3, and may be another configuration as long as it is a configuration for performing information processing described later.

The acquisition unit 131 acquires various information. The acquisition unit 131 acquires various information from an external information processing device. The acquisition unit 131 acquires various information from the storage unit 12. The acquisition unit 131 acquires various types of information from the threshold information storage unit 121 and the density information storage unit 122. The acquisition unit 131 acquires information from the analysis unit 132, the classification unit 133, the selection unit 134, the prediction unit 135, the determination unit 136, and the planning unit 137. The acquisition unit 131 stores the acquired information in the storage unit 12.

The acquisition unit 131 acquires the sensor information detected by the sensor unit 14. The acquisition unit 131 acquires the sensor information (image information) detected by the image sensor 141. The acquisition unit 131 acquires the image information (image) captured by the image sensor 141. The acquisition unit 131 acquires sensor information (contact information) detected by the force sensor 142.

The acquisition unit 131 acquires image information obtained by capturing an image of a target object, which is a candidate object for operation, and an adjacent object, which is an object adjacent to the target object. The acquisition unit 131 acquires image information obtained by capturing an image of the target object and an adjacent object in contact with the target object. The acquisition unit 131 acquires image information obtained by capturing images of the stacked target objects and adjacent objects. The acquisition unit 131 acquires image information obtained by capturing an image of the target object and an adjacent object located within a range affected by the operation on the target object.

In the example of FIG. 1, the acquisition unit 131 acquires an image (image IM1) showing the state ST1 in which the objects OB1, OB2, and OB3 are stacked.

The analysis unit 132 analyzes various information. The analysis unit 132 functions as a physical analysis unit that performs physical analysis. The analysis unit 132 analyzes various kinds of information by using the information regarding the physical properties. The analysis unit 132 analyzes the image information. The analysis unit 132 analyzes various information from the image information based on the information from the external information processing device and the information stored in the storage unit 120. The analysis unit 132 identifies various types of information from the image information. The analysis unit 132 extracts various information from the image information. The analysis unit 132 performs recognition based on the analysis result. The analysis unit 132 recognizes various information based on the analysis result.

The analysis unit 132 performs analysis processing related to the image. The analysis unit 132 performs various processes related to image processing. The analysis unit 132 processes the image information (image) acquired by the acquisition unit 131. The analysis unit 132 processes the image information (image) captured by the image sensor 141. The analysis unit 132 processes the image by appropriately using a technique related to image processing.

The analysis unit 132 executes a process of removing the target object in the image. The analysis unit 132 executes a process of removing the target object in the image by appropriately using a technique related to image processing. In the example of FIG. 1, the analysis unit 132 executes a process of removing the target object OB2 in the image obtained by capturing the situation ST1 in which the objects OB1 to OB3 are adjacent to each other. The analysis unit 132 executes a process of removing the target object OB2 from the image of the situation ST1 in which the object OB2 is in contact with the object OB1 and the object OB3.

In the example of FIG. 1, the analysis unit 132 identifies that the image IM1 includes the objects OB1, OB2, and OB3 by analyzing the image IM1 by a technique such as image analysis. The analysis unit 132 recognizes the physical contact state between the object (adjacent object) around the target object OB2 and the target object OB2. By analyzing the image IM1, the robot device 100 recognizes that the target object OB2 is in contact with the object OB1 and the object OB3.

The classification unit 133 performs various classifications. The classification unit 133 classifies various types of information. The classification unit 133 performs the classification process based on the information acquired by the acquisition unit 131. The classification unit 133 classifies the information acquired by the acquisition unit 131. The classification unit 133 performs the classification process based on the information stored in the storage unit 12. The classification unit 133 makes various estimations. The classification unit 133 estimates various types of information. The classification unit 133 estimates the weight of the object.

The classification unit 133 performs various classifications based on the information acquired by the acquisition unit 131. The classification unit 133 performs various classifications using various sensor information detected by the sensor unit 14. The classification unit 133 performs various classifications using the sensor information detected by the image sensor 141. The classification unit 133 performs various classifications using the sensor information detected by the force sensor 142.

The classification unit 133 estimates the weight of the object included in the image information. The classification unit 133 estimates the weight of the object included in the image information based on the image of the object included in the image information and the density information. The classification unit 133 estimates the weight of the object included in the image information based on the size of the object included in the image information and the density information. The classification unit 133 estimates the size of the object included in the image information, and estimates the weight of the object included in the image information using the estimated size and the density information.

The classification unit 133 classifies the object group included in the image information into an operable object and an inoperable object. The classification unit 133 classifies the object into either a manipulable object or an inoperable object by comparing the estimated weight of the object with the threshold value.

The classification unit 133 compares the weight of each object included in the unknown object group SG1 with the "Wlood", and classifies the object whose weight exceeds the "Wlood" as an inoperable object into the object group G0. The classification unit 133 compares the weight of each object included in the unknown object group SG1 with the "Wlood", and classifies the object having a weight of "Wlood" or less as a manipulable object into the object group G1. The classification unit 133 classifies each book such as the objects OB11 to OB17 included in the book groups SG11 to SG13 into the operable object group G1. The classification unit 133 classifies the object OB10, which is a bookshelf, into the inoperable object group G0.

The selection unit 134 selects various information. The selection unit 134 extracts various information. The selection unit 134 specifies various types of information. The selection unit 134 selects various information based on the information acquired from the external information processing device. The selection unit 134 selects various information based on the information stored in the storage unit 12.

The selection unit 134 makes various selections based on the information acquired by the acquisition unit 131. The selection unit 134 makes various selections based on the information classified by the classification unit 133. The selection unit 134 makes various selections using various sensor information detected by the sensor unit 14. The selection unit 134 makes various selections using the sensor information detected by the image sensor 141. The selection unit 134 makes various selections using the sensor information detected by the force sensor 142.

The selection unit 134 selects an operable object from the object group as the target object based on the classification result by the classification unit 133.

In the example of FIG. 1, the selection unit 134 randomly selects a target object from the object group of the objects OB1, OB2, and OB3. The selection unit 134 selects the object OB2 as the target object from the object group of the objects OB1, OB2, and OB3.

The prediction unit 135 predicts various types of information. The prediction unit 135 predicts various types of information based on the information acquired from the external information processing device. The prediction unit 135 predicts various types of information based on the information stored in the storage unit 12. The prediction unit 135 predicts various information based on the result of the analysis process by the analysis unit 132.

The prediction unit 135 makes various predictions based on the information acquired by the acquisition unit 131. The prediction unit 135 makes various predictions using various sensor information detected by the sensor unit 14. The prediction unit 135 makes various predictions using the sensor information detected by the image sensor 141. The prediction unit 135 makes various predictions using the sensor information detected by the force sensor 142.

The prediction unit 135 predicts a change in the arrangement state of an adjacent object caused by an operation on the target object based on the image information acquired by the acquisition unit 131. For example, the prediction unit 135 predicts a change in the arrangement state of the adjacent object caused by the removal of the target object based on the image information obtained by capturing the image of the target object and the adjacent object. For example, the prediction unit 135 predicts a change in the arrangement state of the adjacent object caused by the change in the position of the target object based on the image information obtained by capturing the image of the target object and the adjacent object. For example, the prediction unit 135 predicts a change in the arrangement state of the adjacent object caused by a change in the posture of the target object based on the image information obtained by capturing the image of the target object and the adjacent object. The prediction unit 135 predicts a change in the posture of an adjacent object caused by an operation on the target object. The prediction unit 135 predicts a change in the position of an adjacent object caused by an operation on the target object.

The prediction unit 135 predicts a change in the arrangement state of the adjacent object caused by the operation on the target object based on the image information obtained by capturing the image information of the target object and the adjacent object in contact with the target object. The prediction unit 135 predicts a change in the arrangement state of the adjacent object caused by the operation on the target object based on the image information obtained by capturing the stacked target object and the adjacent object. The prediction unit 135 predicts a change in the arrangement state of the adjacent object caused by the operation on the target object based on the image information obtained by capturing the image information of the target object and the adjacent object located within the range affected by the operation on the target object. To do. The prediction unit 135 predicts a change in the arrangement state of the adjacent object caused by the operation on the target object selected by the selection unit 134.

In the example of FIG. 1, the prediction unit 135 predicts the posture amount (posture change prediction value) and the position change amount (position change prediction value) of the adjacent object when the target object is removed. The prediction unit 135 predicts a change in the arrangement state of the objects OB1 and OB3 when the target object OB2 is removed. The prediction unit 135 predicts changes in the posture and position of the object OB1 and changes in the posture and position of the object OB3 when the target object OB2 is removed.

The prediction unit 135 predicts that the posture change prediction value and the position position change prediction value of the object OB1 when the target object OB2 is removed are 0. The prediction unit 135 predicts the amount of change in the position and posture of the object OB3 due to the removal of the target object OB2. The prediction unit 135 predicts the posture change prediction value of the object OB1 and the position change prediction value of the position when the target object OB2 is removed.

The determination unit 136 determines various information. The determination unit 136 determines various information. The determination unit 136 specifies various types of information. The determination unit 136 determines various types of information based on the information acquired from the external information processing device. The determination unit 136 determines various types of information based on the information stored in the storage unit 12.

The determination unit 136 makes various determinations based on the information acquired by the acquisition unit 131. The determination unit 136 makes various determinations using various sensor information detected by the sensor unit 14. The determination unit 136 makes various determinations using the sensor information detected by the image sensor 141. The determination unit 136 makes various determinations using the sensor information detected by the force sensor 142. The determination unit 136 determines various information based on the result of the analysis process by the analysis unit 132. The determination unit 136 determines various information based on the result of the prediction process by the prediction unit 135.

The determination unit 136 determines whether or not the object has a portion that moves independently of the object, based on the result of contact with the object by the operation unit 16.

In the example of FIG. 1, the determination unit 136 determines whether the change in the posture or position of the adjacent object is equal to or greater than the threshold value. The determination unit 136 determines whether or not the change in the attitude or position of the adjacent object is equal to or greater than the threshold value by using the threshold value (posture threshold value) regarding the posture and the threshold value (position threshold value) regarding the position stored in the storage unit 12.

In the example of FIG. 2, the determination unit 136 estimates the weight of each book such as the objects OB11 to OB17 included in the book group SG11 to SG13, and determines that the estimated weight is equal to or less than the threshold value "Wlood". The determination unit 136 estimates the weight of the object OB10, which is a bookshelf, and determines that the estimated weight is larger than the threshold value “Wlood”.

Planning department 137 makes various plans. The planning unit 137 generates various information regarding the action plan. The planning unit 137 makes various plans based on the information acquired by the acquisition unit 131. The planning unit 137 makes various plans based on the prediction result by the prediction unit 135. The planning unit 137 makes various plans based on the determination result by the determination unit 136. The planning unit 137 makes an action plan by using various techniques related to the action plan.

Execution unit 138 executes various processes. The execution unit 138 executes various processes based on information from an external information processing device. The execution unit 138 executes various processes based on the information stored in the storage unit 12. The execution unit 138 executes various processes based on the information stored in the threshold information storage unit 121 and the density information storage unit 122. The execution unit 138 executes various processes based on the information acquired by the acquisition unit 131. The execution unit 138 functions as an operation control unit that controls the operation of the operation unit 16.

Execution unit 138 executes various processes based on the prediction result by the prediction unit 135. The execution unit 138 executes various processes based on the determination result by the determination unit 136. The execution unit 138 executes various processes based on the action plan by the planning unit 137.

The execution unit 138 controls the moving unit 15 to execute the action corresponding to the action plan based on the information of the action plan generated by the planning unit 137. The execution unit 138 executes the movement process of the robot device 100 according to the action plan under the control of the movement unit 15 based on the information of the action plan.

The execution unit 138 controls the operation unit 16 based on the information of the action plan generated by the planning unit 137 to execute the action corresponding to the action plan. The execution unit 138 executes the operation processing of the object by the robot device 100 according to the action plan under the control of the operation unit 16 based on the information of the action plan.

In the example of FIG. 1, since the posture change predicted value of the object OB3 caused by the removal of the target object OB2 is equal to or more than the posture threshold value, the execution unit 138 executes a process of operating the object OB3. The execution unit 138 executes the operation of the target object OB2 by the first operation unit 16a, and executes the operation of the object OB3 by the second operation unit 16b. The execution unit 138 executes the operation of the target object OB2 by the first operation unit 16a, and executes the operation of supporting the object OB3 which is an adjacent object by the second operation unit 16b.

The sensor unit 14 detects predetermined information. The sensor unit 14 includes an image sensor 141 and a force sensor 142 as an imaging means for capturing an image.

The image sensor 141 detects image information and functions as vision for the robot device 100. For example, the image sensor 141 is provided on the head of the robot device 100. The image sensor 141 captures image information. In the example of FIG. 1, the image sensor 141 detects (images) an image including an unknown object operation group (unknown object group SG1).

The force sensor 142 detects the force and functions as a tactile sense of the robot device 100. For example, the force sensor 142 is provided at the tip end portion (holding portion) of the operation portion 16. The force sensor 142 detects the contact of the operation unit 16 with an object.

Further, the sensor unit 14 is not limited to the image sensor 141 and the force sensor 142, and may have various sensors. The sensor unit 14 may have a proximity sensor. The sensor unit 14 may have a range finder such as a LiDAR (Light Detection and Ringing, Laser Imaging Detection and Ringing), a ToF (Time of Flight) sensor, or a stereo camera. The sensor unit 14 may have a sensor (position sensor) that detects the position information of the robot device 100 such as a GPS (Global Positioning System) sensor. The sensor unit 14 is not limited to the above, and may have various sensors. The sensor unit 14 may have various sensors such as an acceleration sensor and a gyro sensor. Further, the sensors that detect the above-mentioned various information in the sensor unit 14 may be common sensors, or may be realized by different sensors.

The moving unit 15 has a function of driving the physical configuration of the robot device 100. The moving unit 15 has a function for moving the position of the robot device 100. The moving unit 15 is, for example, an actuator. The moving unit 15 may have any configuration as long as the robot device 100 can realize a desired operation. The moving unit 15 may have any configuration as long as the position of the robot device 100 can be moved. When the robot device 100 has a moving mechanism such as caterpillars and tires, the moving unit 15 drives the caterpillars and tires. For example, the moving unit 15 moves the robot device 100 and changes the position of the robot device 100 by driving the moving mechanism of the robot device 100 in response to an instruction from the execution unit 138.

The robot device 100 according to the first embodiment has two operation units 16 of a first operation unit 16a and a second operation unit 16b. The operation unit 16 is a unit corresponding to a human “hand (arm)” and realizes a function for the robot device 100 to act on another object. The robot device 100 has a first operation unit 16a and a second operation unit 16b as two hands.

The operation unit 16 is driven according to the processing by the execution unit 138. The operation unit 16 is a manipulator that operates an object. For example, the operating unit 16 may be a manipulator having an arm and an end effector. The operation unit 16 operates the adjacent object when the change regarding the arrangement state of the adjacent object satisfies a predetermined condition. At least one of the plurality of operation units 16 operates the adjacent object when the change regarding the arrangement state of the adjacent object satisfies a predetermined condition.

The operation unit 16 has a holding unit that holds an object such as an end effector or a robot hand, and a driving unit that drives the holding unit such as an actuator. The holding unit of the operation unit 16 may be of any method as long as a desired function can be realized, such as a gripper, a multi-finger hand, a jamming hand, a suction hand, and a soft hand. The holding portion of the operating portion 16 may be realized by any configuration as long as it can hold the object, may be a gripping portion that grips the object, or is a suction portion that sucks and holds the object. There may be.

Further, the holding unit of the operation unit 16 may be provided with a force sensor, an image sensor, a proximity sensor, or the like so that information on the position and force of the target object can be acquired. For example, in the robot device 100, a force sensor 142 is provided in the holding portion of the operation unit 16, and information on the force due to the contact of the operation unit 16 with the target object can be acquired.

The first operation unit 16a and the second operation unit 16b are provided on both side portions of the body portion (base portion) of the robot device 100, respectively. The first operation unit 16a extends from the left side portion of the robot device 100 and functions as the left hand of the robot device 100. Further, the second operation unit 16b extends from the right side portion of the robot device 100 and functions as the right hand of the robot device 100. The operation units 16 may be provided at various positions depending on the number of the operation units 16 and the shape of the robot device 100.

[1-3. Information processing procedure according to the first embodiment]
Next, the procedure of information processing according to the first embodiment will be described with reference to FIGS. 6 and 7. 6 and 7 are flowcharts showing the information processing procedure according to the first embodiment. FIG. 6 is a flowchart showing an outline of the information processing procedure by the robot device 100. FIG. 7 is a flowchart showing details of the information processing procedure by the robot device 100.

[1-3-1. Flowchart showing the outline of the information processing procedure]
First, with reference to FIG. 6, an outline of the flow of information processing according to the first embodiment will be described.

As shown in FIG. 6, the robot device 100 acquires image information obtained by capturing an image of the target object and an adjacent object adjacent to the target object (step S101). For example, the robot device 100 acquires image information obtained by capturing images of a plurality of objects from the image sensor 141.

The robot device 100 predicts a change in the arrangement state of an adjacent object caused by an operation on the target object based on the image information (step S102). For example, the robot device 100 predicts a change in the arrangement state of an adjacent object caused by the removal of the object selected as the target object among the plurality of objects in the image information.

Then, the robot device 100 executes a process of operating the adjacent object when the change in the arrangement state of the adjacent object satisfies a predetermined condition (step S103). For example, the robot device 100 executes a process of manipulating an adjacent object when the position or posture of the adjacent object changes by a predetermined threshold value or more due to the removal of the target object.

[1-3-2. Flowchart showing details of information processing procedure]
Next, the details of the flow of information processing according to the first embodiment will be described with reference to FIG. 7. In the example of FIG. 7, it is assumed that the robot device 100 has already acquired image information obtained by capturing images of a plurality of objects.

As shown in FIG. 7, the robot device 100 randomly selects an operation target (step S201). The robot device 100 selects, as an operation target, an object whose weight is estimated to be less than a predetermined threshold value among a plurality of objects included in the image. In the example of FIG. 1, the robot device 100 selects the object OB2 from the objects OB1 to OB3 as the operation target object (target object). The robot device 100 may end the process when there is no selectable object.

Then, the robot device 100 recognizes the physical contact state with the object around the target object (step S202). The robot device 100 recognizes a physical contact state with an object around the target object by analyzing the image. In the example of FIG. 1, the robot device 100 recognizes that the target object OB2 is in contact with the object OB1 and the object OB3.

Then, the robot device 100 determines whether or not there is an object in physical contact with the surroundings (step S203). The robot device 100 determines whether or not there is an adjacent object in physical contact around the target object. In the example of FIG. 1, the robot device 100 determines whether or not there is an adjacent object in physical contact around the target object OB2.

Then, when the robot device 100 determines that there is no object in physical contact with the surroundings (step S203: No), the robot device 100 executes the operation of the target object (step S208). For example, the robot device 100 controls the first operation unit 16a and the second operation unit 16b, and executes an operation of changing the position and orientation of the target object.

On the other hand, when the robot device 100 determines that there is an object in physical contact in the vicinity (step S203: Yes), the robot device 100 predicts the posture of the peripheral object (adjacent object) in physical contact when the target object is removed (step S204). ). When the robot device 100 has an object in physical contact with the surrounding object, the robot device 100 predicts a change in the posture or position of the peripheral object in physical contact when the target object is removed.

Then, the robot device 100 determines whether the movement of the surrounding object is equal to or higher than the threshold value (step S205). The robot device 100 determines whether the change in the posture or position of the adjacent object is equal to or greater than the threshold value. In the example of FIG. 1, the robot device 100 determines whether the change in the posture or the position of the objects OB1 and OB3, which are adjacent objects of the target object OB2, is equal to or more than the threshold value.

Then, when the robot device 100 determines that the movement of the surrounding object is not equal to or greater than the threshold value (step S205: No), the robot device 100 executes the operation of the target object (step S208).

On the other hand, when the robot device 100 determines that the movement of the surrounding object is equal to or higher than the threshold value (step S205: Yes), the robot device 100 determines whether or not there is another manipulator that can be operated (step S206). The robot device 100 determines whether or not there is an operable manipulator (operation unit 16) in addition to the manipulator (operation unit 16) that operates the target object.

The robot device 100 determines whether or not there are other operation units 16 having a number of peripheral objects whose movement is equal to or higher than the threshold value, in addition to the operation unit 16 that operates the target object. For example, when the robot device 100 has two peripheral objects whose movements are equal to or higher than the threshold value, the robot device 100 determines whether or not there are two operable operation units 16 in addition to the operation unit 16 that operates the target object. For example, when the robot device 100 has one peripheral object whose movement is equal to or higher than the threshold value, the robot device 100 determines whether or not there is one operable operating unit 16 in addition to the operating unit 16 that operates the target object. For example, the robot device 100 determines whether or not there is another operation unit 16 (for example, the second operation unit 16b) in addition to the operation unit 16 (for example, the first operation unit 16a) that operates the target object.

When the robot device 100 determines that there is no other manipulator that can be operated (step S206: No), the robot device 100 returns to step S201 and repeats the process.

When the robot device 100 determines that there is another manipulator that can be operated (step S206: Yes), the robot device 100 supports the peripheral object with another manipulator (step S207). When the robot device 100 has another operable operation unit 16 in addition to the operation unit 16 that operates the target object, the robot device 100 executes an operation on an adjacent object such as supporting a peripheral object (adjacent object) by the other operation unit 16. To do. The robot device 100 may perform not only the operation of supporting the adjacent object but also the operation of changing the position and the posture of the adjacent object as the operation for the adjacent object. In the robot device 100, one operation unit 16 may move the target object, and another operation unit 16 may move the adjacent object.

Then, the robot device 100 executes the operation of the target object (step S208). For example, in the robot device 100, the first operation unit 16a may execute the operation of the target object, and the second operation unit 16b may execute the operation of the adjacent object. In the example of FIG. 1, the robot device 100 may execute the operation of the target object OB2 by the first operation unit 16a and the operation of supporting the adjacent object OB3 by the second operation unit 16b.

In this way, the robot device 100 can perform operations on the target object and surrounding objects according to the number of operation units 16.

[1-4. Conceptual diagram of the configuration of the information processing device]
Here, with reference to FIG. 8, each functional configuration in the robot device 100 is conceptually shown. FIG. 8 is a diagram showing an example of a conceptual diagram of a robot configuration. The configuration group FCB1 shown in FIG. 8 includes a sensor processing unit, an object / environment determination unit, a task planning unit, an operation planning unit, a control unit, and the like.

The sensor processing unit corresponds to, for example, the sensor unit 14 and the acquisition unit 131 in FIG. 4, and detects various types of information such as visual sense, force sense, tactile sense (vibration), proximity sense, and temperature. The object / environment determination unit corresponds to, for example, the analysis unit 132 to the determination unit 136 in FIG. 4, and executes various processes such as estimation and determination. The object / environment determination unit executes various processes such as determination of an adjacent object, weight estimation of the adjacent object, estimation of the center of gravity of the adjacent object, motion analysis of the adjacent object, motion prediction of the adjacent object, and stability determination of the adjacent object. ..

The motion planning unit, for example, corresponds to the planning unit 137 in FIG. 4, and performs gripping planning (end effector), movement route planning (moving body), and arm trajectory planning (manipulator). The motion planning unit performs gripping planning by the holding unit (end effector) of the operating unit 16, movement route planning by the moving unit 15, and arm trajectory planning by the operating unit 16 (manipulator). The control unit corresponds to, for example, the execution unit 138 in FIG. 4 and performs actuator control and sensor control.

[1-5. Processing example of Nth object operation]
It is assumed that the robot device 100 not only operates a single object, but also operates a plurality of stacked objects and a plurality of objects such that a main object is accompanied by a follower. The processing by the robot device 100 in this case will be described with reference to FIG. FIG. 9 is a diagram showing an example of processing of Nth-order object operation. Specifically, FIG. 9 is a diagram showing an example of processing of secondary object manipulation. The same points as in FIG. 1 will be omitted as appropriate.

For example, when another object B is stacked on the object A in the direction of gravity and the object A is operated while paying attention to the posture of the object B and the contact state with the object A, this is defined as a secondary object operation. .. FIG. 9 shows, as an example, a case where the teapot (object OB21) is operated so that the lid (object PT1) of the teapot does not spill. As described above, FIG. 9 shows an example in which the object OB21 is operated so that the object PT1 (subordinate) which is the lid of the teapot does not fall from the object OB21 (main object) which is the teapot.

Here, in the case of such an Nth-order object operation (N is an arbitrary number of 2 or more, “2” in the case of FIG. 9), object extraction, center of gravity detection, and physics based on visual information by an image sensor 141 or the like. It may not be possible to judge whether stable operation can be performed only by model simulation. For example, when pouring tea by tilting the teapot while holding down the lid (object PT1) of the teapot (object OB21), the lid of the teapot is only supported in the direction of gravity by the body of the teapot, and it can be moved in any direction with a weak force. May move. In this way, in the operation stability detection of an unknown object group that may move due to a change in posture or a weak external force, for example, is there a place where the object moves by applying a weak external force to the object by a manipulator in advance of the operation? Add a process to check if.

In FIG. 9, it is assumed that the robot device 100 does not have the knowledge that the main body of the kyusu and the lid of the kyusu move. That is, the robot device 100 does not have the knowledge that the kyusu (object OB21) and the lid (object PT1) can be separated. The robot device 100 confirms whether or not there is a moving portion of the target object by bringing the surface of the target object recognized by the image sensor 141 into contact with the robot hand with a force within the threshold value.

The robot device 100 brings the operation unit 16 into contact with the object OB21 and the object PT1 (step S21). The robot device 100 brings the second operation unit 16b into contact with the object OB21 (object PT1) with a force within the threshold value. The robot device 100 detects the force related to the contact of the second operation unit 16b with the object OB21 (object PT1) by the force sensor 142, and based on the information regarding the detected force, the second operation unit 16b is used as the object OB21 (object PT1). The strength of contact with PT1) is controlled. When the robot device 100 moves due to contact, the robot device 100 extracts and segments the difference from the image before the movement, and recognizes the object. When the object PT1 moves due to the contact of the second operation unit 16b with the object OB21 (object PT1), the robot device 100 extracts and segments the difference from the image before the movement, and sets the kyusu body (object OB21) and the lid. (Object PT1) is recognized. The robot device 100 may recognize the shape of a moving object (object PT1) by using a distance measuring sensor such as ToF mounted on the holding unit (end effector) of the operating unit 16.

The robot device 100 determines whether or not the object has a portion that moves independently of the object based on the result of contact with the object by the operation unit 16 (step S22). In the example of FIG. 9, in the robot device 100, the object PT1 moves independently of the object OB21 in response to the contact of the second operation unit 16b with the object OB21 (object PT1). judge.

The robot device 100 operates an object according to the determination result (step S23). When the robot device 100 determines that the object has a portion that moves independently of the object, the robot device 100 executes a process of operating the object according to the number of operation units 16. In the example of FIG. 9, the robot device 100 determines that the object OB21 has a portion (object PT1) that moves independently of the object OB21, and executes a process of operating the object according to the number of operation units 16. .. Since the robot device 100 has two operation units 16, the second operation unit 16b holds the object PT1 which is a lid, and the first operation unit 16a operates the object OB21 which is a steeple. Perform the action of pouring tea into (cup).

In this way, the robot device 100 determines that the object has a portion that moves independently of the object, and operates the object according to the result of the determination to execute an appropriate operation according to the object. Can be done.

[2. Second Embodiment]
[2-1. Configuration of Robot Device According to Second Embodiment of the Present Disclosure]
In the first embodiment, the case where the robot device 100 has a plurality of (two) operation units 16 is shown, but the robot device may have a single (one) operation unit 16. .. In the second embodiment, the case where the robot device 100A has only one operation unit 16 will be described as an example. The same points as the robot device 100 according to the first embodiment will be omitted as appropriate.

First, the configuration of the robot device 100A, which is an example of the information processing device that executes the information processing according to the second embodiment, will be described. FIG. 10 is a diagram showing a configuration example of the robot device according to the second embodiment of the present disclosure.

As shown in FIG. 10, the robot device 100A includes a communication unit 11, a storage unit 12, a control unit 13, a sensor unit 14, a moving unit 15, and an operation unit 16.

In the example of FIG. 11, the operation unit 16 is provided at the base of the robot device 100a. The operation unit 16 is provided so as to extend from a base portion that connects the moving unit 15 of the robot device 100a. The operation unit 16 may be provided at a different position depending on the shape of the robot device 100A.

[2-2. Outline of information processing according to the second embodiment]
Next, the outline of the information processing according to the second embodiment will be described with reference to FIG. FIG. 11 is a diagram showing an example of information processing according to the second embodiment. The information processing according to the second embodiment is realized by the robot device 100A shown in FIG. A case where processing is performed on a group of objects in which a plurality of objects are stacked will be described as an example with reference to FIG. The same points as in FIG. 1 in FIG. 11 will be omitted as appropriate. Since the processes of the states ST1 to ST3 and steps S1 to S4 shown in FIG. 11 are the same as those of FIG. 1, the description thereof will be omitted.

By the process of step S4, the robot device 100a determines that the posture change predicted value of the object OB3 generated by the removal of the target object OB2 is equal to or more than the posture threshold value. Further, in the example of FIG. 11, the robot device 100a has only one operation unit 16. Therefore, the robot device 100a determines that the operation of the target object OB2 is impossible, and selects a target object as a candidate for another operation target (step S31). The robot device 100a selects an object OB3 (hereinafter, also referred to as “target object OB3”) as a target object from the object group of the remaining objects OB1 and OB3 other than the object OB2. That is, the robot device 100a executes the process for the object OB3 which is an adjacent object of the target object OB2.

Then, the robot device 100a removes the target object (step S2). In the example of FIG. 11, the robot device 100a removes the target object OB3. The robot device 100a removes the target object OB3 from the image IM1 showing the state ST1 in which the objects OB1, OB2, and OB3 are stacked. By removing the target object OB3 from the image IM1, the robot device 100a performs processing on the state ST32 in which only the target object OB3 is removed from the object group of the objects OB1, OB2, and OB3.

The robot device 100a predicts a change in the arrangement state of adjacent objects when the target object is removed (step S33). The robot device 100a predicts changes in the posture and position of adjacent objects when the target object is removed. The robot device 100a is not limited to an object that is in direct contact with the target object (contact object), but an object that is in contact with the contact object, that is, an object that is in chain contact with the target object, or the like, is an adjacent object. It may be processed as. As shown in the state ST1 of FIG. 11, only the object OB2 is in direct contact with the target object OB3, but the object OB1 is in contact with the object OB2 and is in contact with the target object OB3 in a chain reaction. Therefore, the robot device 100a processes the object OB1 as an adjacent object. In this way, the robot device 100a predicts a change in the arrangement state of the objects OB1 and OB2 when the target object OB3 is removed. The robot device 100a predicts changes in the posture and position of the object OB1 and changes in the posture and position of the object OB2 when the target object OB3 is removed.

Since the object OB2 is located below the target object OB3, the robot device 100a predicts that the position and posture of the object OB2 will not change due to the removal of the target object OB3. Since the robot device 100a is in a state where the object OB2 supports the target object OB3, it is predicted that the position and posture of the object OB2 will not be changed by removing the target object OB3. For example, the robot device 100a predicts that the posture change predicted value and the position position change predicted value of the object OB2 when the target object OB3 is removed are 0.

Further, since the object OB1 is located below the target object OB3 and the object OB2, the robot device 100a predicts that the position and posture of the object OB1 will not be changed by removing the target object OB3. Since the robot device 100a is in a state where the object OB1 supports the target object OB3 and the object OB2, it is predicted that the position and the posture of the object OB1 will not be changed by removing the target object OB3. For example, the robot device 100a predicts that the posture change predicted value and the position position change predicted value of the object OB1 when the target object OB3 is removed are 0.

Then, the robot device 100a determines whether the movement of the surrounding object of the adjacent object is equal to or higher than the threshold value (step S4). The robot device 100a determines that the change in posture and position of both the object OB1 and the object OB2, which are adjacent objects of the target object OB3, is less than the threshold value.

The robot device 100a executes a process related to the operation based on the determination result (step S5). In the example of FIG. 11, the robot device 100a assumes that the target object OB3 can be operated because the change in posture and position of both the object OB1 and the object OB2, which are adjacent objects of the target object OB3, is less than the threshold value. The operation unit 16 executes the operation of the target object OB2. In this case, the robot device 100a executes an operation such as moving the target object OB3.

After that, the robot device 100a completes the execution of all the operations of the objects OB1 to OB3 by executing the operations in the order of the object OB2 and the object OB1.

As described above, the robot device 100a executes a process of manipulating the target object and the adjacent object based on the change in the arrangement state of the adjacent object adjacent to the target object when the target object is removed. In this way, the robot device 100a can enable an appropriate operation on an object even when an adjacent object exists.

[3. Third Embodiment]
[3-1. Configuration of robot device according to the third embodiment of the present disclosure]
In the first embodiment and the second embodiment, the case where the robot devices 100 and 100a have one or two operation units 16 is shown, but the robot device has three or more operation units 16. You may have. In the third embodiment, the case where the robot device 100B has three operation units 16 will be described as an example. The same points as the robot device 100 according to the first embodiment and the robot device 100a according to the second embodiment will be omitted as appropriate.

First of all. The configuration of the robot device 100B, which is an example of the information processing device that executes the information processing according to the third embodiment, will be described. FIG. 12 is a diagram showing a configuration example of the robot device according to the third embodiment of the present disclosure.

As shown in FIG. 12, the robot device 100B includes a communication unit 11, a storage unit 12, a control unit 13, a sensor unit 14, a moving unit 15, a first operation unit 16a, and a second operation unit 16b. , And a third operation unit 16c.

The robot device 100b according to the third embodiment has three operation units 16 of a first operation unit 16a, a second operation unit 16b, and a third operation unit 16c. The first operation unit 16a, the second operation unit 16b, and the third operation unit 16c are provided on both sides of the body portion (base portion) of the robot device 100. The first operation unit 16a extends from the left side portion of the robot device 100 and functions as the left hand of the robot device 100. Further, the second operation unit 16b and the third operation unit 16c extend from the right side portion of the robot device 100 and function as the right hand of the robot device 100. The operation units 16 may be provided at various positions depending on the number of the operation units 16 and the shape of the robot device 100. For example, the third operation unit 16c may be provided in the central portion of the base portion.

The determination unit 136 of the robot device 100B can operate the objects OB1 to OB3 as shown in the state ST1 in FIG. 1 even when the object OB1 is first selected as the target object. judge. The determination unit 136 of the robot device 100B may also process the object OB1 in contact with the object OB2 in contact with the object OB1 as an adjacent object.

For example, the robot device 100B executes a process of operating the objects OB2 and OB3 because the predicted posture change values of the objects OB2 and OB3 caused by the removal of the target object OB1 are equal to or higher than the posture threshold value. For example, in the robot device 100, the first operation unit 16a executes the operation of the target object OB1, the second operation unit 16b executes the operation of the object OB2, and the third operation unit 16c executes the operation of the object OB3. For example, in the robot device 100, the first operation unit 16a executes the operation of the target object OB1, the second operation unit 16b executes the operation of supporting the adjacent object OB2, and the third operation unit 16c executes the operation of the adjacent object OB2. An operation for supporting a certain object OB3 is executed. In this case, the robot device 100 drives the second operation unit 16b after an operation such as moving the target object OB1, executes an operation of arranging the object OB2 at a stable position, and drives the third operation unit 16c. , The operation of arranging the object OB3 at a stable position may be executed.

Further, in the robot device 100, the first operation unit 16a executes an operation of holding the target object OB1, the second operation unit 16b executes an operation of holding the object OB3, and the third operation unit 16c holds the object OB3. You may perform the operation to do. Then, the robot device 100 may carry the objects OB2 and OB3 together with the object OB1 to a desired position by the moving unit 15 while the operating unit 16 holds the objects OB1 to OB3.

As described above, the robot device 100b executes a process of manipulating the target object and the adjacent object based on the change in the arrangement state of the adjacent object adjacent to the target object when the target object is removed. In this way, the robot device 100b can enable an appropriate operation on an object even when an adjacent object exists.

[3-2. Outline of information processing according to the third embodiment]
Next, the outline of the information processing according to the third embodiment will be described with reference to FIG. FIG. 13 is a diagram showing an example of information processing according to the third embodiment. The information processing according to the third embodiment is realized by the robot device 100B shown in FIG. A case where a process of carrying a tray on which a plurality of objects are placed is performed will be described as an example with reference to FIG. The same points as in the above example will be omitted as appropriate.

In the example of FIG. 13, the robot device 100B shows a case where the object OB40, which is a tray on which the objects OB41 to OB46, which are a plurality of dishes (tableware), are placed. In this way, when the robot device 100B places food (objects OB41 to OB46) on the tray (object OB40) and serves the food, it is necessary to be careful not to spill the drink in the cup on the tray. There is. In this case, the relationship is as follows: operation unit 16 (robot arm) → tray → cup → drink, and when viewed from the operation unit 16, the drink is a tertiary object. Also, when carrying food on the tray, food and drink may spill due to vibration and external contact.

For example, when serving dishes in Obon, the dishes may be labeled in advance in order of ease of movement. The robot device 100B may label tableware that is easy to move by an external force by the operation unit 16 (manipulator) in order of ease of movement. For example, the robot device 100B may apply an external force from the operation unit 16 (manipulator) to the tableware, measure the movement of each object OB41 to OB46, and label the objects OB41 to OB46 in the order in which they are easy to move. Then, the robot device 100B may hold an easily movable object by the remaining operation units 16 other than the operation unit 16 necessary for holding the object OB40 which is a tray. The robot device 100B is labeled as being easy to move in the order of objects OB46, OB42, OB45, OB41, OB43, and OB44. In this way, the robot device 100B labels the objects OB46 and OB42 whose contents are liquid when they are easy to move.

In the example of FIG. 13, the robot device 100B holds the object OB40, which is a tray, by the second operation unit 16b, and holds the object OB46, which is liable to spill the contents (liquid), by the remaining first operation unit 16a. The operation unit 16c holds an object OB42 in which the contents (liquid) are likely to spill. In this way, the robot device 100B can stably perform the serving task by holding down the unstable tableware with the operation unit 16 (manipulator) different from the operation unit 16 (manipulator) having the tray. Become.

[4. Other embodiments]
The processing according to each of the above-described embodiments may be carried out in various different forms (modifications) other than each of the above-described embodiments.

[4-1. Other configuration examples]
For example, in the above-described example, the information processing devices that perform information processing are the

robot devices

100, 100A, and 100B, but the information processing device and the robot device may be separate bodies. This point will be described with reference to FIGS. 14 and 15. FIG. 14 is a diagram showing a configuration example of an information processing system according to a modified example of the present disclosure. FIG. 15 is a diagram showing a configuration example of an information processing device according to a modified example of the present disclosure.

As shown in FIG. 14, the information processing system 1 includes a robot device 10 and an information processing device 100C. The robot device 10 and the information processing device 100C are connected to each other via a network N so as to be communicable by wire or wirelessly. The information processing system 1 shown in FIG. 14 may include a plurality of robot devices 10 and a plurality of information processing devices 100C. In this case, the information processing device 100C may communicate with the robot device 10 via the network N and give an instruction to control the robot device 10 based on the information collected by the robot device 10 and various sensors.

The robot device 10 transmits sensor information detected by sensors such as an image sensor and a force sensor to the information processing device 100C. The robot device 10 transmits image information obtained by capturing an image of a group of objects by an image sensor to the information processing device 100C. As a result, the information processing apparatus 100C acquires image information including a group of objects. The robot device 10 may be any device as long as information can be transmitted and received to and from the information processing device 100C, and may be various robots such as an autonomous mobile robot.

The information processing device 100C is an information processing device that transmits information (control information) for controlling the robot device 10, such as an action plan, to the robot device 10. For example, the information processing device 100C generates control information for controlling the robot device 10, such as an action plan of the robot device 10, based on the information stored in the storage unit 12C and the information acquired from the robot device 10. .. The information processing device 100C transmits the generated control information to the robot device 10. The robot device 10 that has received the robot device 10 from the information processing device 100C controls the moving unit 15 based on the control information and moves, or controls the operating unit 16 based on the control information to operate the object. ..

As shown in FIG. 15, the information processing device 100C includes a communication unit 11C, a storage unit 12C, and a control unit 13C. The communication unit 11C is connected to the network N (Internet or the like) by wire or wirelessly, and transmits / receives information to / from the robot device 10 via the network N.

The storage unit 12C is realized by, for example, a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 12C stores the same information as the storage unit 12. The storage unit 12C has a threshold information storage unit 121 and a density information storage unit 122. The storage unit 12C stores information for controlling the movement of the robot device 10, various information received from the robot device 10, and various information to be transmitted to the robot device 10.

The control unit 13C is realized by, for example, a CPU, an MPU, or the like executing a program stored inside the information processing device 100C (for example, an information processing program according to the present disclosure) using a RAM or the like as a work area. Further, the control unit 13C may be realized by an integrated circuit such as an ASIC or FPGA. The control unit 13C includes an acquisition unit 131, an analysis unit 132, a classification unit 133, a selection unit 134, a prediction unit 135, a determination unit 136, a planning unit 137, and a transmission unit 138C.

The transmission unit 138C transmits various information to an external information processing device. The transmission unit 138C transmits various information to an external information processing device. For example, the transmission unit 138C transmits various information to the robot device 10. The transmission unit 138C provides the information stored in the storage unit 12. The transmission unit 138C transmits the information stored in the storage unit 12.

The transmission unit 138C provides various information based on the information from the robot device 10. The transmission unit 138C provides various information based on the information stored in the storage unit 12.

The transmission unit 138C transmits the control information to the robot device 10. The transmission unit 138C transmits the action plan created by the action planning unit to the robot device 10. The transmission unit 138C executes a process of transmitting control information to the robot device 10 in order to cause the robot device 10 to operate an adjacent object. In this way, the transmission unit 138C functions as an execution unit that executes a process of manipulating an adjacent object by executing a process of transmitting control information to the robot device 10.

As described above, the information processing device 100C does not have a sensor unit, a moving unit, an operation unit, or the like, and does not have to have a configuration for realizing a function as a robot device. The information processing device 100C includes an input unit (for example, a keyboard, a mouse, etc.) that receives various operations from an administrator or the like that manages the information processing device 100C, and a display unit (for example, a liquid crystal display, etc.) for displaying various information. ) May have.

[4-2. Others]
Further, among the processes described in each of the above embodiments, all or a part of the processes described as being automatically performed can be manually performed, or the processes described as being manually performed. It is also possible to automatically perform all or part of the above by a known method. In addition, the processing procedure, specific name, and information including various data and parameters shown in the above document and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each figure is not limited to the illustrated information.

Further, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. That is, the specific form of distribution / integration of each device is not limited to the one shown in the figure, and all or part of the device is functionally or physically distributed / physically in arbitrary units according to various loads and usage conditions. Can be integrated and configured.

Further, each of the above-described embodiments and modifications can be appropriately combined as long as the processing contents do not contradict each other.

Further, the effects described in the present specification are merely examples and are not limited, and other effects may be obtained.

[5. Effect of this disclosure]
As described above, the information processing devices (

robot devices

100, 100A, 100B, information processing device 100C in the embodiment) according to the present disclosure include a prediction unit (prediction unit 135 in the embodiment) and an execution unit (execution in the embodiment). A unit 138) is provided. The prediction unit relates to the arrangement state of the adjacent object generated by the operation on the target object based on the image information obtained by capturing the image information of the target object which is a candidate object to be operated and the adjacent object which is an object adjacent to the target object. Predict changes. The execution unit executes a process of manipulating the adjacent object when the change in the arrangement state of the adjacent object predicted by the prediction unit satisfies a predetermined condition.

As described above, when the target object, which is a candidate object to be operated, has an adjacent object, the information processing apparatus according to the present disclosure refers to the target object based on the image information captured by the target object and the adjacent object. Predict changes in the arrangement of adjacent objects caused by the operation. For example, the information processing apparatus removes the target object, changes the position of the target object, or changes the posture of the target object based on the image information obtained by capturing the image information of the target object and the adjacent object. Predict changes in. Then, when the change in the arrangement state of the adjacent object satisfies a predetermined condition, the information processing device executes a process of operating the adjacent object, so that an appropriate operation on the object is performed even if the adjacent object exists. Can be made possible.

Further, when the amount of change regarding the arrangement state of the adjacent object is equal to or greater than the threshold value, the execution unit executes a process of operating the adjacent object. In this way, when the amount of change in the arrangement state of the adjacent object is equal to or greater than the threshold value, the information processing device executes a process of operating the adjacent object, so that the information processing device is appropriate for the object even if the adjacent object exists. Operation is possible.

In addition, the prediction unit predicts changes in the posture of adjacent objects caused by operations on the target object. When the change in the posture of the adjacent object satisfies the condition related to the change in posture, the execution unit executes a process of operating the adjacent object. In this way, the information processing device predicts the change in the posture of the adjacent object, and when the change in the posture of the adjacent object satisfies the condition related to the posture change, the information processing device executes a process of manipulating the adjacent object to cause the adjacent object to move. Appropriate manipulation of the object, even if present, can be made possible.

In addition, the prediction unit predicts changes in the position of adjacent objects caused by operations on the target object. The execution unit executes a process of manipulating the adjacent object when the change in the position of the adjacent object satisfies the condition regarding the position change. In this way, the information processing device predicts the change in the position of the adjacent object, and when the change in the position of the adjacent object satisfies the condition regarding the position change, the information processing device executes a process of manipulating the adjacent object so that the adjacent object can be moved. Appropriate manipulation of the object, even if present, can be made possible.

Further, the prediction unit predicts changes in the arrangement state of the adjacent object caused by the operation on the target object based on the image information obtained by capturing the image information of the target object and the adjacent object in contact with the target object. As described above, when the change in the arrangement state of the adjacent object in contact with the target object satisfies a predetermined condition, the information processing device may execute a process of manipulating the adjacent object so that the contacting object exists. However, proper operation can be enabled.

Further, the prediction unit predicts changes in the arrangement state of the adjacent objects caused by the operation on the target objects based on the image information obtained by capturing the stacked target objects and the adjacent objects. As described above, when the change in the arrangement state of the adjacent objects stacked on the target object satisfies a predetermined condition, the information processing device executes a process of operating the adjacent objects to exist the stacked objects. Even so, it is possible to enable an appropriate operation.

Further, the prediction unit relates to the arrangement state of the adjacent object generated by the operation on the target object based on the image information obtained by capturing the image information of the target object and the adjacent object located within the range affected by the operation on the target object. Predict change. In this way, when the change in the arrangement state of the adjacent object located within the range affected by the operation on the target object satisfies a predetermined condition, the information processing device executes a process of operating the adjacent object to be adjacent. Even if there is an object to be processed, it is possible to appropriately operate the object.

Further, the information processing device includes an operation unit (operation unit 16 in the embodiment). The operation unit is driven according to the processing by the execution unit. As a result, the information processing apparatus can perform an appropriate operation on the object even when an adjacent object exists by the operation unit driven by the operation unit driven by the processing by the execution unit.

Further, the operation unit operates the adjacent object when the change regarding the arrangement state of the adjacent object satisfies a predetermined condition. In this way, when the change in the arrangement state of the adjacent object satisfies a predetermined condition, the information processing apparatus can operate the adjacent object to perform an appropriate operation on the object even if the adjacent object exists. Can be.

Further, the information processing device includes a plurality of operation units (in the embodiment, the first operation unit 16a, the second operation unit 16b, and the third operation unit 16c). The plurality of operation units are driven according to the processing by the execution unit. As a result, the information processing apparatus can perform an appropriate operation on the object even when there are adjacent objects by the plurality of operation units driven by the processing by the execution unit.

Further, at least one of the plurality of operation units operates the adjacent object when the change regarding the arrangement state of the adjacent object satisfies a predetermined condition. In this way, when the change in the arrangement state of the adjacent object satisfies a predetermined condition, the information processing apparatus can operate the adjacent object to perform an appropriate operation on the object even if the adjacent object exists. Can be.

Further, when the change regarding the arrangement state of the adjacent object satisfies a predetermined condition, the execution unit executes a process of causing one of the plurality of operation units to operate the target object, and the other of the plurality of operation units. Performs a process of causing the operation unit of the above to operate an adjacent object. In this way, when the change in the arrangement state of the adjacent object satisfies a predetermined condition, the information processing device executes a process of causing one operation unit of the plurality of operation units to operate the target object, and causes the other operation units to operate the target object. By executing the process of operating the adjacent object, it is possible to appropriately operate the object even when the adjacent object exists.

In addition, the execution unit executes a process of moving the target object to one operation unit. In this way, the information processing apparatus can perform an appropriate operation on the object even when the adjacent object exists by executing the process of moving the target object.

In addition, the execution unit executes a process of suppressing the change in the arrangement state of the adjacent object due to the movement of the target object to another operation unit. In this way, the information processing device executes a process of suppressing a change in the arrangement state of the adjacent object due to the movement of the target object, so that an appropriate operation can be performed on the object even if the adjacent object exists. can do.

In addition, the execution unit executes a process of causing another operation unit to support an adjacent object. In this way, the information processing apparatus can perform an appropriate operation on the adjacent object even when the adjacent object exists by executing the process of supporting the adjacent object.

In addition, the execution unit executes a process of moving an adjacent object to another operation unit. In this way, the information processing apparatus can perform an appropriate operation on the object even when the adjacent object exists by executing the process of moving the adjacent object.

Further, the information processing device includes a force sensor (force sensor 142 in the embodiment). The force sensor detects the contact of the operation unit with an object. The execution unit executes a process of bringing the operation unit into contact with an object based on the sensor information detected by the force sensor. As a result, the information processing apparatus can acquire information on the state of the object by detecting the contact with the object by the operation unit, so that it is possible to perform an appropriate operation according to the object.

Further, the information processing device includes a determination unit (determination unit 136 in the embodiment). The determination unit determines whether or not the object has a portion that moves independently of the object based on the result of contact with the object by the operation unit. When the determination unit determines that the object has a location, the execution unit executes a process of operating the object according to the number of operation units. As a result, the information processing device determines whether or not the object has a portion that moves independently of the object based on the result of contact with the object by the operation unit, and corresponds to the determination result and the number of operation units. By executing the process of operating the object, it is possible to perform an appropriate operation according to the state of the object and the number of operation units.

[6. Hardware configuration]
The information devices such as the

robot devices

100, 100A, 100B and the information processing device 100C according to the above-described embodiments are realized by, for example, the computer 1000 having the configuration shown in FIG. FIG. 16 is a hardware configuration diagram showing an example of a computer 1000 that realizes the functions of information processing devices such as

robot devices

100, 100A, 100B and information processing device 100C. Hereinafter, the robot device 100 according to the first embodiment will be described as an example. The computer 1000 includes a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, an HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. Each part of the computer 1000 is connected by a bus 1050.

The CPU 1100 operates based on the program stored in the ROM 1300 or the HDD 1400, and controls each part. For example, the CPU 1100 expands the program stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to various programs.

The ROM 1300 stores a boot program such as a BIOS (Basic Input Output System) executed by the CPU 1100 when the computer 1000 is started, a program that depends on the hardware of the computer 1000, and the like.

The HDD 1400 is a computer-readable recording medium that non-temporarily records a program executed by the CPU 1100 and data used by the program. Specifically, the HDD 1400 is a recording medium for recording an information processing program according to the present disclosure, which is an example of program data 1450.

The communication interface 1500 is an interface for the computer 1000 to connect to an external network 1550 (for example, the Internet). For example, the CPU 1100 receives data from another device or transmits data generated by the CPU 1100 to another device via the communication interface 1500.

The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or mouse via the input / output interface 1600. Further, the CPU 1100 transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. Further, the input / output interface 1600 may function as a media interface for reading a program or the like recorded on a predetermined recording medium (media). The media is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory. Is. For example, when the computer 1000 functions as the robot device 100 according to the embodiment, the CPU 1100 of the computer 1000 realizes the functions of the control unit 13 and the like by executing the information processing program loaded on the RAM 1200. Further, the information processing program according to the present disclosure and the data in the storage unit 12 are stored in the HDD 1400. The CPU 1100 reads the program data 1450 from the HDD 1400 and executes the program, but as another example, these programs may be acquired from another device via the external network 1550.

The present technology can also have the following configurations.
(1)
Changes related to the arrangement state of the adjacent object caused by the operation on the target object based on the image information captured by the target object which is a candidate object to be operated and the adjacent object which is an object adjacent to the target object. Prediction unit that predicts
When the change in the arrangement state of the adjacent object predicted by the prediction unit satisfies a predetermined condition, the execution unit that executes the process of operating the adjacent object and the execution unit.
Information processing device equipped with.
(2)
The execution unit
The information processing apparatus according to (1), wherein when the amount of change in the arrangement state of the adjacent object is equal to or greater than the threshold value, the process of operating the adjacent object is executed.
(3)
The prediction unit
Predicting changes in the posture of the adjacent object caused by the operation on the target object,
The execution unit
The information processing apparatus according to (1) or (2), wherein when the change in the posture of the adjacent object satisfies the condition regarding the change in posture, the process of operating the adjacent object is executed.
(4)
The prediction unit
Predicting changes in the position of the adjacent object caused by the operation on the target object,
The execution unit
The information processing apparatus according to any one of (1) to (3), which executes a process of manipulating the adjacent object when the change in the position of the adjacent object satisfies the condition relating to the position change.
(5)
The prediction unit
Based on the image information obtained by capturing the image of the target object and the adjacent object in contact with the target object, the change in the arrangement state of the adjacent object caused by the operation on the target object is predicted (1) to (4). The information processing apparatus according to any one of ().
(6)
The prediction unit
Any of (1) to (5) for predicting a change in the arrangement state of the adjacent object caused by an operation on the target object based on the image information obtained by capturing the stacked target object and the adjacent object. The information processing apparatus according to item 1.
(7)
The prediction unit
Based on the image information obtained by imaging the target object and the adjacent object located within the range affected by the operation on the target object, the change regarding the arrangement state of the adjacent object caused by the operation on the target object is changed. Prediction The information processing apparatus according to any one of (1) to (6).
(8)
An image sensor that captures the image information and
An acquisition unit that acquires the image information captured by the image sensor, and
The information processing apparatus according to any one of (1) to (7).
(9)
An operation unit that is driven in response to processing by the execution unit,
The information processing apparatus according to any one of (1) to (8).
(10)
The operation unit
The information processing device according to (9), which is a manipulator that operates an object.
(11)
The operation unit
The information processing apparatus according to (9) or (10), which operates the adjacent object when the change in the arrangement state of the adjacent object satisfies a predetermined condition.
(12)
A plurality of operation units driven according to processing by the execution unit,
The information processing apparatus according to any one of (1) to (8).
(13)
Each of the plurality of operation units
The information processing apparatus according to (12), which is a manipulator that operates an object.
(14)
At least one of the plurality of operation units
The information processing apparatus according to (12) or (13), which operates the adjacent object when the change in the arrangement state of the adjacent object satisfies a predetermined condition.
(15)
The execution unit
When the change in the arrangement state of the adjacent object satisfies a predetermined condition, a process of causing one of the plurality of operation units to operate the target object is executed, and another operation of the plurality of operation units is performed. The information processing apparatus according to any one of (12) to (14), which executes a process of causing a unit to operate the adjacent object.
(16)
The execution unit
The information processing apparatus according to (15), which executes a process of moving the target object to the one operation unit.
(17)
The execution unit
The information processing apparatus according to (16), wherein the other operation unit executes a process of suppressing a change in the arrangement state of the adjacent object due to the movement of the target object.
(18)
The execution unit
The information processing apparatus according to (16) or (17), which executes a process of causing the other operating unit to support the adjacent object.
(19)
The execution unit
The information processing apparatus according to (16) or (17), which executes a process of moving the adjacent object to the other operation unit.
(20)
A force sensor that detects contact with an object by the operation unit,
With more
The execution unit
The information processing apparatus according to any one of (9) to (19), which executes a process of bringing the operation unit into contact with an object based on the sensor information detected by the force sensor.
(21)
A determination unit that determines whether or not the object has a part that moves independently of the object based on the result of contact with the object by the operation unit.
With more
The execution unit
The information processing apparatus according to (20), wherein when the determination unit determines that the object has the location, the processing for operating the object according to the number of operation units is executed.
(22)
A classification unit that classifies a group of objects included in the image information into an operable object and an inoperable object.
A selection unit that selects a manipulable object from the object group as the target object based on the classification result by the classification unit.
With more
The prediction unit
The information processing apparatus according to any one of (1) to (21), which predicts a change in the arrangement state of the adjacent object caused by an operation on the target object selected by the selection unit.
(23)
Changes related to the arrangement state of the adjacent object caused by the operation on the target object based on the image information captured by the target object which is a candidate object to be operated and the adjacent object which is an object adjacent to the target object. Predict and
An information processing method that executes control to execute a process of operating the adjacent object when the predicted change in the arrangement state of the adjacent object satisfies a predetermined condition.
(24)
Changes related to the arrangement state of the adjacent object caused by the operation on the target object based on the image information captured by the target object which is a candidate object to be operated and the adjacent object which is an object adjacent to the target object. Predict and
When the predicted change in the arrangement state of the adjacent object satisfies a predetermined condition, the process of manipulating the adjacent object is executed.
An information processing program that executes control.

100, 100A, 100B Robot device 100C Information processing device 11, 11C Communication unit 12, 12C Storage unit 121 Threshold information storage unit 122 Density information storage unit 13, 13C Control unit 131 Acquisition unit 132 Analysis unit 133 Classification unit 134 Selection unit 135 Prediction Unit 136 Judgment unit 137 Planning unit 138 Execution unit 138C Transmission unit 14 Sensor unit 141 Image sensor 142 Force sensor 15 Moving unit 16 Operation unit 16a 1st operation unit 16b 2nd operation unit 16c 3rd operation unit

Claims

Changes related to the arrangement state of the adjacent object caused by the operation on the target object based on the image information captured by the target object which is a candidate object to be operated and the adjacent object which is an object adjacent to the target object. Prediction unit that predicts
When the change in the arrangement state of the adjacent object predicted by the prediction unit satisfies a predetermined condition, the execution unit that executes the process of operating the adjacent object and the execution unit.
Information processing device equipped with.
The execution unit
The information processing apparatus according to claim 1, wherein when the amount of change in the arrangement state of the adjacent object is equal to or greater than a threshold value, a process of operating the adjacent object is executed.
The prediction unit
Predicting changes in the posture of the adjacent object caused by the operation on the target object,
The execution unit
The information processing apparatus according to claim 1, wherein when the change in the posture of the adjacent object satisfies the condition related to the change in posture, the process of operating the adjacent object is executed.
The prediction unit
Predicting changes in the position of the adjacent object caused by the operation on the target object,
The execution unit
The information processing apparatus according to claim 1, wherein when the change in the position of the adjacent object satisfies the condition regarding the position change, the process of operating the adjacent object is executed.
The prediction unit
The first aspect of claim 1, wherein the change in the arrangement state of the adjacent object caused by the operation on the target object is predicted based on the image information obtained by capturing the image of the target object and the adjacent object in contact with the target object. Information processing device.
The prediction unit
The information processing apparatus according to claim 1, wherein a change in the arrangement state of the adjacent object caused by an operation on the target object is predicted based on the image information obtained by capturing the stacked target object and the adjacent object.
The prediction unit
Based on the image information obtained by imaging the target object and the adjacent object located within the range affected by the operation on the target object, the change regarding the arrangement state of the adjacent object caused by the operation on the target object is changed. The information processing apparatus according to claim 1.
An operation unit that is driven in response to processing by the execution unit,
The information processing apparatus according to claim 1.
The operation unit
The information processing device according to claim 8, wherein when the change in the arrangement state of the adjacent object satisfies a predetermined condition, the adjacent object is operated.
A plurality of operation units driven according to processing by the execution unit,
The information processing apparatus according to claim 1.
At least one of the plurality of operation units
The information processing apparatus according to claim 10, wherein the information processing device operates the adjacent object when the change in the arrangement state of the adjacent object satisfies a predetermined condition.
The execution unit
When the change in the arrangement state of the adjacent object satisfies a predetermined condition, a process of causing one of the plurality of operation units to operate the target object is executed, and another operation of the plurality of operation units is performed. The information processing apparatus according to claim 10, wherein a process of causing a unit to operate the adjacent object is executed.
The execution unit
The information processing device according to claim 12, wherein the process of moving the target object to the one operation unit is executed.
The execution unit
The information processing apparatus according to claim 13, wherein the other operation unit executes a process of suppressing a change in the arrangement state of the adjacent object due to the movement of the target object.
The execution unit
The information processing device according to claim 13, wherein a process of causing the other operating unit to support the adjacent object is executed.
The execution unit
The information processing device according to claim 13, wherein the process of moving the adjacent object to the other operation unit is executed.
A force sensor that detects contact with an object by the operation unit,
With more
The execution unit
The information processing device according to claim 8, wherein the processing of bringing the operation unit into contact with an object is executed based on the sensor information detected by the force sensor.
A determination unit that determines whether or not the object has a part that moves independently of the object based on the result of contact with the object by the operation unit.
With more
The execution unit
The information processing apparatus according to claim 17, wherein when the determination unit determines that the object has the location, the information processing device executes a process of operating the object according to the number of operation units.
Changes related to the arrangement state of the adjacent object caused by the operation on the target object based on the image information captured by the target object which is a candidate object to be operated and the adjacent object which is an object adjacent to the target object. Predict and
An information processing method that executes control to execute a process of operating the adjacent object when the predicted change in the arrangement state of the adjacent object satisfies a predetermined condition.
Changes related to the arrangement state of the adjacent object caused by the operation on the target object based on the image information captured by the target object which is a candidate object to be operated and the adjacent object which is an object adjacent to the target object. Predict and
When the predicted change in the arrangement state of the adjacent object satisfies a predetermined condition, the process of manipulating the adjacent object is executed.
An information processing program that executes control.