JP7569742B2 - SERVER, REMOTE CONTROL SYSTEM, REMOTE CONTROL METHOD, AND PROGRAM - Google Patents

Description

The present invention relates to a server, a remote control system, a remote control method, and a program.

Conventionally, there is known technology for remotely controlling a robot to operate experimental equipment or devices using data transmitted from a monitor terminal or another terminal (see, for example, Patent Document 1).

JP 2003-092749 A

However, with conventional technology, there were limitations to the instructions that could be given from a terminal device, so the robot could only be made to perform specific tasks, and there were cases where the robot could not be made to operate appropriately.

The aspects of the present invention have been made in consideration of these circumstances, and one of the objectives is to provide a server, a remote control system, a remote control method, and a program that can operate a robot more appropriately.

The server, the remote control system, the remote control method, and the program according to the present invention employ the following configuration.
(1): A server according to one embodiment of the present invention is a server including: a communication unit that communicates via a network with one or more robotic devices and one or more operation terminals that remotely operate at least one of the one or more robotic devices; an acquisition unit that acquires operation content input by the operation terminal; an intention estimation unit that estimates the intention of an operator of the operation terminal based on the operation content acquired by the acquisition unit; and an operation control unit that controls operation of the robotic device based on the intention result estimated by the intention estimation unit.

(2): In the above aspect (1), the acquisition unit acquires robot control information related to the operation of the robot device and sensor information detected by a sensor provided in the robot device, and further includes a management unit that sets different access rights to the robot control information and the sensor information acquired by the acquisition unit, and enables acquisition of either the robot control information or the sensor information when the set access rights are confirmed.

(3): In the aspect of (2) above, the management unit sets the strength of the access rights for the sensor information to be greater than that for the robot control information.

(4): In any one of the above aspects (1) to (3), when the operation control unit controls the operation of one robot device based on operation contents acquired from multiple operation terminals, the operation control unit generates operation control information for the robot device based on the priority of the operator who operates the operation terminal.

(5): In the aspect of (4) above, when the acquisition unit acquires the same or similar operation content from the multiple operation terminals, the operation control unit controls the operation of the robot device by prioritizing the operation content from the operator with the highest priority.

(6): In the above aspect (4) or (5), the intention estimation unit further includes an output control unit that estimates the intention of each of the operators of a plurality of operation terminals that operate the single robot device, and outputs the intention estimation results of each of the operators estimated by the intention estimation unit to each of the plurality of operation terminals.

(7): Another aspect of the present invention is a remote control system comprising a server as described in any one of (1) to (6) above, the one or more robot devices, and the one or more operation terminals.

(8): Another aspect of the present invention is a remote operation method in which a computer communicates via a network with one or more robotic devices and one or more operation terminals that remotely operate at least one of the one or more robotic devices, acquires operation content input by the operation terminal, estimates the intention of the operator of the operation terminal based on the acquired operation content, and controls the operation of the robotic device based on the estimated intention result.

(9): Another aspect of the present invention is a program that causes a computer to communicate via a network with one or more robotic devices and one or more operation terminals that remotely operate at least one of the one or more robotic devices, acquire operation content input by the operation terminal, estimate the intention of the operator of the operation terminal based on the acquired operation content, and control the operation of the robotic device based on the estimated intention result.

According to the above aspects (1) to (9), the robot can be operated more appropriately.

According to the above aspects (2) to (3), the confidentiality of data is protected. Therefore, for example, it is possible to prevent sensor information from being stolen or tampered with by companies that do not have the access rights, thereby improving the security of the entire system. In addition, by managing the information provided based on the access rights, various companies can use the service with peace of mind.

According to the above aspects (4) to (5), the robot can be operated appropriately even when operated by multiple operators.

According to the above aspect (5), multiple operators can more accurately grasp the situation in which the robot device is being operated by other operators. Therefore, the robot device can be operated by inputting more appropriate operation content.

1 is a diagram illustrating an example of a configuration of a remote control system 1 according to an embodiment. FIG. 2 is a diagram illustrating an example of a configuration of a management server 100 according to the embodiment. FIG. 13 is a diagram for explaining the contents of intention estimation information 172B. FIG. 13 is a diagram for explaining the contents of company-specific user information 174A. FIG. 13 is a diagram for explaining the contents of company-specific sensor information 174B. FIG. 2 is a diagram illustrating an example of a configuration of a robot device 200 according to an embodiment. FIG. 2 is a diagram illustrating an example of a configuration of an operation terminal 300 according to the embodiment. FIG. 2 is a diagram for explaining remote control of the robot device 200 according to the present embodiment. FIG. 2 is a diagram showing an example of an image IM10 displayed by an operation terminal 300. FIG. 13 is a diagram showing an example of an image IM20 displayed on the operation terminal 300 when a task based on a priority is executed. FIG. 13 is a diagram showing an example of an image IM30 for inquiring of an operator about an operation to be executed by the robot device. FIG. 2 is a diagram showing an example of an image IM40 displayed on a monitor 214 of the robot device 200. FIG. 2 is a sequence diagram for explaining a process executed by the remote control system 1 in the embodiment.

Below, we will explain the embodiments of the server, remote control system, remote control method, and program of the present invention with reference to the drawings.

[System configuration]
FIG. 1 is a diagram showing an example of the configuration of a remote operation system 1 according to an embodiment. The remote operation system 1 shown in FIG. 1 includes, for example, a management server 100, a robot device 200, and an operation terminal 300. In addition to the above configuration, the management server 100 may also include a robot manufacturer terminal 400 and a telecommunications company terminal 500. The management server 100 is communicatively connected to the robot device 200, the operation terminal 300, the robot manufacturer terminal 400, and the telecommunications company terminal 500 via a network NW. The network NW includes, for example, the Internet, a cellular network, a Wi-Fi (registered trademark) network, a WAN (Wide Area Network), a LAN (Local Area Network), a provider device, a wireless base station, and the like. In the remote operation system 1, the robot device 200 and the operation terminal 300 may each include one or more. In the example of FIG. 1, robot devices 200A, 200B, and 200C are shown, but the number and types of the robot devices are not limited to this. 1 shows operation terminals 300A-1 to 300A-n (n is a natural number of 2 or more) and 300B-1 to 300B-n managed by each of a plurality of companies A and B, but the numbers are not limited to this. In the following description, when the robot device and the operation terminal are not particularly identified, they will be simply referred to as "robot device 200" and "operation terminal 300". The management server 100 is an example of a "server".

The management server 100 manages one or more operation terminals 300 in association with one or more robotic devices 200 to be operated. In this case, the management server 100 may associate one robotic device 200 with each operation terminal 300, may associate one robotic device 200 with multiple operation terminals 300, or may associate multiple robotic devices 200 with one operation terminal 300. The management server 100 also acquires operation content input from the operation terminal 300 and transmits operation control information corresponding to the operation content to the robotic device 200, thereby remotely operating the robotic device 200. The management server 100 also transmits information acquired by the robotic device 200 to the operation terminal 300.

The robot device 200 executes a predetermined operation based on control information transmitted from the management server 100 via the network NW. The robot device 200 is equipped with a moving mechanism that can move by driving wheels, a cart, a crawler, a crane, or the like. In the example of FIG. 1, the robot device 200A is a robot equipped with both arms that can move by at least a crane. The robot device 200B is a robot equipped with both arms that can move by wheels provided at the bottom. The robot device 200C is a robot equipped with both arms that can move by bipedal walking. The robot device 200 is also equipped with an arm unit that performs tasks such as grasping, moving, and operating an object. The robot device 200 is also equipped with multiple various sensors such as vibration, temperature, pressure, and tactile sensors, and transmits data detected by each sensor to the management server 100 at a predetermined cycle or at the timing when a request is received. The robot device 200 may also be equipped with a camera that captures images of the surroundings, a monitor that displays images, a speaker that outputs sound, a microphone that captures surrounding sounds, and the like.

For example, one or more operation terminals 300 are provided for each company that uses the robot device 200. The operation terminal 300 inputs the operation contents of the robot device 200 and notifies the operator of information acquired by the robot device 200. The operation terminal 300 can operate one or more robot devices 200. The operation terminal 300 may also operate some parts of the robot device 200 (e.g., the right arm or the left arm, etc.). The operation terminal 300 may also be provided with, for example, an HMD (head mounted display), an operation device, and an environmental sensor that detects the movement of the operator. Note that the operation terminal 300 is not limited to being provided in a facility within the company, and may be a terminal provided in the operator's home, or a terminal provided in a telework station installed in a station, department store, public facility, internet cafe, etc.

The robot manufacturer terminal 400 is a terminal used, for example, by the manufacturer or management company of the robot device 200. The robot manufacturer terminal 400, for example, updates the version of the software (program) installed in each robot device 200, acquires error information from the robot device 200, and performs remote operation or stop control of the robot device 200 based on the acquired error information.

The telecommunications company terminal 500 is, for example, a terminal used by a telecommunications company that manages communications on the network NW. The telecommunications company terminal 500 manages the amount of data communicated for remote operation of the remote control system 1, manages communication delays, etc., and performs maintenance of the communication environment within the system. The telecommunications company terminal 500 may also respond when a failure occurs on the network NW.

Next, the functions of the management server 100, the robot device 200, and the operation terminal 300 will be specifically described.
[Administration Server]
FIG. 2 is a diagram showing an example of the configuration of the management server 100 according to the embodiment. The management server 100 includes, for example, a communication unit 110, a communication control unit 120, an acquisition unit 130, a management unit 140, a control unit 150, and a storage unit 170. The communication control unit 120, the acquisition unit 130, the management unit 140, and the control unit 150 are each realized by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by cooperation between software and hardware. Some or all of the functions of these components may be realized by a dedicated LSI. The program may be stored in advance in a storage device (storage device with a non-transient storage medium) such as a hard disk drive (HDD) or flash memory provided in the management server 100, or may be stored in a removable storage medium (non-transient storage medium) such as a DVD or CD-ROM, and the storage medium may be installed in the HDD or flash memory provided in the management server 100 by being attached to a drive device provided in the management server 100. The management server 100 may be realized in a server device or storage device incorporated in a cloud computing system. In this case, the functions of the management server 100 may be realized by a plurality of server devices and storage devices in the cloud computing system.

Under the control of the communication control unit 120, the communication unit 110 communicates with one or more robot devices 200, an operation terminal 300 that remotely operates one or more robot devices 200, and other external devices via the network NW. The communication control unit 120 controls the communication in the communication unit 110.

The communication control unit 120 controls the communication in the communication unit 110. For example, the communication control unit 120 receives the operation content of the robot device 200 input from the operation terminal 300, and transmits operation control information corresponding to the operation content to the robot device 200. The communication control unit 120 also receives information such as the detection results of sensors and camera images transmitted from the robot device 200, and transmits information to be provided based on the received information to the operation terminal.

The acquisition unit 130 acquires various types of information from the robot device 200 or the operation terminal 300. For example, the acquisition unit 130 acquires operation information for operating the robot device 200 from the operation terminal 300. The acquisition unit 130 also acquires information from sensors and cameras provided in the robot device 200. The acquisition unit 130 may also acquire information from external devices connected to the network NW.

The management unit 140 includes, for example, an authentication management unit 142 and an information management unit 144. The authentication management unit 142 manages the authentication and usage authority of the operator (system user) who operates the robot device 200. Furthermore, the authentication management unit 142 associates and manages the robot device 200 operated by the operation terminal 300 based on the authentication result.

The information management unit 144 manages information available for each user. For example, the information management unit 144 stores robot control information for controlling the robot and information (sensor information) acquired from a sensor provided in the robot device 200 in the storage unit 170, and, for example, sets different access rights for each, and transmits information to the operation terminal 300 that has the corresponding authority to provide when the set access rights are confirmed. In addition, the information management unit 144 may, for example, increase the strength of the access rights for the sensor information compared to the robot control information. Increasing the strength of the access rights means, for example, storing the data in a storage area that cannot be accessed unless a specific authentication is permitted, encrypting and managing the data itself, increasing the number and types of authentication conditions such as passwords for acquiring data, etc. This protects the confidentiality of the data. Therefore, for example, it is possible to prevent the sensor information from being stolen or tampered with by a company that does not have the access rights, and the security of the entire system can be improved. In addition, by managing the information provided by the access rights, various companies and the like can use the remote operation system 1 with peace of mind.

The control unit 150 includes, for example, a robot information control unit 151, a customized data control unit, an intention estimation unit 153, an operation control unit 154, an operation cancellation unit 155, and an output control unit 156. The robot information control unit 151 manages, for example, what operations the robot device 200 actually performed in response to the operation control information for the robot device 200 generated by the operation control unit 154. The robot information control unit 151 also causes the robot device 200 to perform basic operations based on the robot control information 172A stored in the storage unit 170. The basic operations are, for example, operations to move the robot device 200, change its posture, and pick up or place an object.

The customization data control unit 152 generates customization data for the robot device 200 to execute specific operations for each company (e.g., assembling objects, soldering, screwing, various cooking processes, etc.).

The intention estimation unit 153 estimates the intention of the operator from at least a part of the information of the operation contents input by the operation terminal 300. The intention of the operator includes, for example, not only the intention of the action (motion) that the operator wants the robot device 200 to perform, but also prediction of the action, etc., and assisting the driving of the robot device 200. For example, in remote operation, a discrepancy occurs between the feeling that the operator is instructing through communication, etc. and the motion of the robot device 200. Therefore, the intention estimation unit 153 does not directly execute the operation contents from the operator to the robot device 200, but estimates the intention of the operator based on the operation contents, and generates motion control information for the robot device 200 based on the estimation result, thereby more appropriately assisting the movement of the robot device 200 and realizing remote operation with less discomfort. For example, the intention estimation unit 153 refers to a learning model (intention estimation information 172B) learned using teacher data, and estimates the intention corresponding to the operation contents. During learning, for example, the operation content is input, and training data is used to output correct intention information for that operation content set by the operator or the like.

The intention estimation unit 153 may also update the intention estimation information 172B based on history information indicating the operation content input by the operator and the actual operation content of the robot device 200. This allows for more appropriate intention estimation.

Furthermore, the intention estimation unit 153 may estimate the intention of the operator by, for example, a GRASP Taxonomy method (see, for example, Reference 1). In the embodiment, the intention of the operator is estimated by classifying the operator state by classifying the posture, i.e., the gripping posture, of the operator or the robot device 200 by, for example, the GRASP Taxonomy method.
Reference 1: Thomas Feix, Javier Romero, et al., “The GRASP Taxonomy of Human GraspTypes” IEEE Transactions on Human-Machine Systems (Volume: 46, Issue: 1, Feb.2016), IEEE, p66-77.

The operation control unit 154 generates operation control information for operating the target robot device 200 based on the intention of the operator estimated by the intention estimation unit 153, transmits the generated operation control information to the target robot device 200, and controls the operation of the robot device 200. The operation control unit 154 may also acquire information on the operation of the robot device 200 obtained by the robot information control unit 151, the customized data control unit 152, etc. in addition to (or instead of) the intention of the operator, and generate operation control information based on the acquired information. For example, when multiple operators operate two arms simultaneously, the operation control unit 154 may perform synthesis processing such as addition or subtraction of forces based on sensor information such as tactile sense, and perform operation control for working in cooperation. This makes it possible to achieve well-balanced operations even when one robot device 200 is operated by multiple operators.

The action cancellation unit 155 selects an action that is different from the operator's intention from among the actions that the action control unit 154 causes the robot device 200 to execute, and cancels the execution of the selected action. In this case, the action cancellation unit 155 inquires of the operator whether or not the operation content input by the operator is the content to be executed by the robot device 200, and based on the inquiry result, causes the action control unit 154 to generate action control information and update the intention estimation information 172B.

The output control unit 156 generates information to be provided to the operator operating the operation terminal 300 based on the sound and images acquired from the robot device 200. The information to be provided includes images and sound. The output control unit 156 also outputs sensor information, etc., based on the result of the authentication process performed by the authentication management unit 142.

The storage unit 170 may be realized by the above-mentioned various storage devices, or a solid state drive (SSD), an electrically erasable programmable read only memory (EEPROM), a read only memory (ROM), or a random access memory (RAM). The storage unit 170 includes, for example, a first storage unit 172 and a second storage unit 174. The first storage unit 172 and the second storage unit 174 are storage areas with different strengths of access rights. The first storage unit 172 stores, for example, information that can be used by all users of the remote control system 1. The second storage unit 174 stores, for example, information that can be acquired only by companies or users whose access rights have been confirmed in advance. The second storage unit 174 may be, for example, an area in which data encrypted for each company is stored and cannot be viewed unless decryption is performed using a key or the like assigned to each company. The setting of the access rights by the first storage unit 172 and the second storage unit 174 is managed by the authentication management unit 142. Additionally, the first storage unit 172 and the second storage unit 174 may be different storage units, and one or both may be managed by an external device such as a data server.

For example, the first storage unit 172 stores robot control information 172A and intention estimation information 172B. The robot control information 172A is basic control information for causing the robot device 200 to execute a predetermined operation. The predetermined operation is, for example, an operation such as moving, changing direction, moving an arm, holding (grasping) an object, or placing an object. These may be set for each type of the robot device 200, for example.

The intention estimation information 172B is, for example, information on the intention estimation result estimated by the intention estimation unit 153. FIG. 3 is a diagram for explaining the contents of the intention estimation information 172B. In the intention estimation information 172B, for example, information on the estimated intention is associated with the operation content acquired from the operation terminal 300. This information may be managed for each operator, or the intention estimation information 172B of multiple operators may be integrated to store only information on frequently occurring intention estimations, or may be updated based on history information, etc.

The second storage unit 174 stores information that can be accessed by a user whose access rights have been confirmed. For example, the second storage unit 174 stores company-specific user information 174A and company-specific sensor information 174B.

Figure 4 is a diagram for explaining the contents of the company-specific user information 174A. In the example of Figure 4, user information for each company is stored, and each piece of information is managed so that it cannot be accessed unless the access right of each company is confirmed. The company-specific user information 174A is, for example, information such as name, authority, priority, and image information associated with authentication information that authenticates a user when using a service by the remote control system 1. The authentication information includes, for example, a user ID and a password that are identification information that identify a user. The authentication information may also include biometric authentication information such as fingerprint information and iris information. The authority includes, for example, authority information such as access rights assigned to a user. Based on this authority information, sensor information acquired by the robot device 200 can be acquired. The information management unit 144 can manage who can access which data, which data cannot be accessed, etc. by referring to the information stored in the authority. The priority information stores, for example, the priority when multiple operators remotely operate the robot device 200 and the operation contents of each are the same or similar. Image information is an image used to recognize a user, and includes, for example, an image of the user's face, an avatar image created to resemble the user, or an illustrated image.

Figure 5 is a diagram for explaining the contents of the company-specific sensor information 174B. In the example of Figure 5, sensor information for each company is stored, and each piece of information is managed so that it cannot be accessed unless the access rights of each company are confirmed. In the company-specific sensor information 174B, the robot ID, which is identification information for identifying the robot device 200, stores the date and time, operator ID, operation parts, sensor information, etc. The operator ID stores the user ID of the operator who performed the operation. The operation parts store the parts of the robot ID operated by the operator. This makes it possible to manage the operation history of each operator even when one robot device 200 is operated by multiple operators.

[Robot device]
6 is a diagram showing an example of the configuration of the robot device 200 according to the embodiment. The robot device 200 includes, for example, a communication unit 202, a sensor 204, a camera (an example of an imaging unit) 206, a microphone 208, a driving unit 210, a monitor 214, a speaker 216, a control device 240, and a storage unit 260. The driving unit 210 includes, for example, an arm unit 210A and a moving driving unit 210B. The control device 240 includes, for example, a communication control unit 242, an acquisition unit 244, a drive control unit 246, and an output control unit 248. The communication control unit 242, the acquisition unit 244, the drive control unit 246, and the output control unit 248 are each realized by a hardware processor such as a CPU executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as an LSI, ASIC, FPGA, or GPU, or may be realized by a combination of software and hardware. Some or all of the functions of these components may be realized by a dedicated LSI. The program may be stored in advance in a storage device (storage device having a non-transient storage medium) such as an HDD or flash memory provided in the robot device 200, or may be stored in a removable storage medium (non-transient storage medium) such as a DVD or CD-ROM, and the storage medium may be installed in the HDD or flash memory provided in the robot device 200 by being attached to a drive device provided in the robot device 200.

The communication unit 202 will be described later. Under the control of the communication control unit 242, the communication unit 202 communicates with the management server 100 and other external devices via the network NW. The communication unit 202 may also communicate with other robot devices 200, or with the operation terminal 300.

The sensor 204 includes a position sensor that detects the position of the robot device 200, a speed sensor that detects the speed, and a temperature sensor that detects the temperature of a specific position such as the periphery of the robot device 200 or the tip of the arm unit 210A. The position sensor receives information from, for example, a GPS (Global Positioning System) receiving device, and obtains position information (longitude and latitude information) based on the received information. The sensor 204 may also include a humidity sensor that detects the surrounding humidity, and a vibration sensor that detects vibrations of an object operated by the robot device 200. The sensor 204 may also include an object detection sensor that detects surrounding objects. Examples of surrounding objects include other robot devices 200, people, and obstacles.

The camera 206 is, for example, a digital camera using a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 206 captures an image of the periphery of the robot device 200 at a predetermined timing, for example. The number of cameras 206 is not limited to one, and multiple cameras may be provided on the robot device 200. For example, the camera 206 is provided on the head of the robot device 200. The camera 206 may also be provided near the tip of the arm unit 210A. This allows the object to be worked on to be photographed at a close distance, making it easier to perform more detailed work.

Sounds around the robot device 200 are input to the microphone 208. The microphone 208 outputs information based on the input sound to the control device 240.

The arm unit 210A grasps a target object and performs a predetermined task on the target object. The arm unit 210A is, for example, a multi-joint robot arm, and includes, for example, an actuator, a gear, an artificial muscle, and the like. For example, the arm unit 210A includes a first arm unit in which one end of the multi-joint robot arm is connected near the right side of the robot device main body, and a second arm unit in which one end of the other multi-joint robot arm is connected near the left side of the robot device main body. The arm unit 210A may also include other arm units in addition to the first arm unit and the second arm unit. Hereinafter, when there is no need to distinguish between the respective arms, they will simply be referred to as "arm unit 210A". The other end of the arm unit is configured with a gripping unit capable of gripping a predetermined object. The arm unit 210A is driven based on the control of the control device 220. The arm unit is capable of movement equivalent to that of a human arm.

The moving drive unit 210B is a drive unit for moving the robot device 200 on the floor or ground. The moving drive unit 210B may be, for example, two legs, or may be provided with a moving mechanism such as wheels, a dolly, or a crawler. For example, when the moving drive unit 210B is a leg, the leg operates to make the robot device 200 walk based on the control of the control device 220. The moving drive unit 210B may also be structured to be movable along a ceiling or rail by driving a crane or the like. With these configurations, the robot device 200 can move in a desired direction. Note that the drive unit 210 may be provided with a mechanism for driving other joints such as the waist and head in addition to the arm unit 210A or the moving drive unit 210B. The drive unit 210 is driven based on the control of the drive control unit 246.

The monitor 214 is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display device. The monitor 214 displays the information output by the output control unit 248 as an image. A plurality of monitors 214 may be provided in the robot device 200. The monitor 214 may be provided, for example, in the head, abdomen, or back. The speaker 216 outputs the information output by the output control unit 248 as sound.

The control device 240 includes, for example, a communication control unit 242, an acquisition unit 244, a drive control unit 246, and an output control unit 248. The communication control unit 242, the acquisition unit 244, the drive control unit 246, and the output control unit 248 are realized, for example, by a hardware processor such as a CPU executing a program (software). Some or all of these components may be realized by hardware (circuitry) such as an LSI, ASIC, FPGA, or GPU, or may be realized by a combination of software and hardware. The program may be stored in advance in a storage device (non-transient storage medium) such as an HDD or flash memory, or may be stored in a removable storage medium (non-transient storage medium) such as a DVD or CD-ROM, and installed by inserting the storage medium into the drive device.

The communication control unit 242 communicates with the management server 100, for example, by wireless communication via the communication unit 202, to send and receive information. The communication control unit 242 may also communicate with other robotic devices 200 via the communication unit 202.

The acquisition unit 244 acquires the detection results (sensor information) of the sensor 204 provided in the robot device 200. The acquisition unit 244 also acquires images captured by the camera 206. The acquisition unit 244 also acquires information based on sounds input from the microphone 208.

The drive control unit 246 operates the arm unit 210A and the movement drive unit 210B based on the operation control information acquired from the management server 100. The drive control unit 246 also drives other parts of the robot device 200 (e.g., the head, torso, waist, etc.) based on the operation control information to change the posture, etc. The drive control unit 246 may also drive the robot device 200 based on the basic operation information 262 and customized operation information 264 stored in the storage unit 260. The basic operation information 262 is drive control information for executing basic operations according to the type, etc. of the robot device 200. The customized operation information 264 is, for example, drive control information for causing the robot device 200 to execute a specific operation registered in advance for each operator.

The output control unit 248 causes the monitor 214 to output an image based on the information acquired from the management server 100. The output control unit 248 also causes the speaker 216 to output voice, warning sounds, etc. based on the information acquired from the management server 100.

The storage unit 260 is realized by, for example, a HDD, a flash memory, an EEPROM, a ROM, or a RAM. The storage unit 260 stores, for example, basic operation information 262, customized operation information 264, programs, and other information.

[Operation terminal]
7 is a diagram showing an example of the configuration of the operation terminal 300 according to the embodiment. The operation terminal 300 includes, for example, a communication unit 310, an HMD 320, an operation device 330, an environmental sensor 340, a control device 360, and a storage unit 380. The HMD 320, the operation device 330, and the environmental sensor 340 are examples of an "HMI (Human Machine Interface)."

The communication unit 310 communicates with the management server 100 via the network NW under the control of the control device 360. The communication unit 310 may also communicate with other operation terminals 300 and robot devices 200.

The HMD 320 includes, for example, an image display unit 322, a gaze detection unit 324, a sensor 326, a speaker 328, and a control unit 329. The HMD 320 outputs and displays the state image and sound of the robot device 200 that the control device 360 receives from the management server 100, and detects the movement of the operator's gaze, etc.

The image display unit 322 is, for example, an LCD or an organic EL display. The image display unit displays an image output by the output control unit 368 (described later) in accordance with the control of the control unit 329.

The gaze detection unit 324 detects the gaze of the operator wearing the HMD 320 under the control of the control unit 329, and outputs the detected gaze information (operator sensor value) to the control device 360. The gaze information is, for example, information including a gaze vector.

The sensor 326 is, for example, an acceleration sensor, a gyroscope sensor, a magnetic sensor, etc. The sensor 326 detects the tilt and rotation of the head of the operator wearing the HMD 320, and outputs the detected head movement information (operator sensor value) to the control device 360.

The speaker 328 outputs voice, warning sounds, etc. output by the output control unit 368.

The control unit 329 controls the execution of gaze detection by the gaze detection unit 324 based on control information from the control device 360, controls detection by the sensor 326, controls the image display on the image display unit 322, and accepts information input from the operator via the HMD 320.

The operation device 330 includes, for example, a sensor (operator sensor) 332, a control unit 334, a feedback unit 336, and an input unit 338. The operation device 330 includes, for example, a tactile data glove that is worn on the operator's hand.

The sensor 332 is, for example, an acceleration sensor, a gyroscope sensor, a magnetic sensor, etc. The sensor 332 includes a plurality of sensors. The sensor 332 tracks the movement of each finger using, for example, two sensors. The sensor 332 detects operator arm information (operator sensor value) which is information on the orientation and movement of each finger of the operator, the posture and position of the operator's arm, such as the movement of the hand, under the control of the control unit 334, for example. The operator arm information (operator sensor value) includes information covering the entire human arm, such as hand position and posture information, angle information of each finger, elbow position and posture information, and information tracking the movement of each part.

The control unit 334 outputs the operator's arm information detected by the sensor 332 to the control device 360. The control unit 334 also controls the feedback unit 336 based on the feedback information obtained from the control device 360.

The feedback unit 336 feeds back feedback information to the operator in response to the control of the control unit 334. In response to the feedback information, the feedback unit 336 feeds back sensations to the operator, for example, by means of vibration (not shown), air pressure (not shown), hand movement restriction (not shown), temperature sensation (not shown), hardness or softness sensation (not shown), vibration sensation (not shown), etc., attached to the arm unit 210A of the robot device 200.

The input unit 338 is, for example, an input device other than the tactile data glove, such as a keyboard, a mouse, a lever, a touch panel, a microphone, etc. The input unit 338 accepts input of operation details for the robot device 200 via each input device.

The environmental sensor 340 detects, for example, the movement of the operator. The environmental sensor 340 includes, for example, a camera (an example of an imaging unit) 342, a sensor 344, and an object position detection unit 346. The camera 342 captures an image including the operator. The camera 342 is, for example, an RGB camera. The camera 342 outputs the captured image to the object position detection unit 346. Note that the positional relationship between the camera 342 and the sensor 344 is known in the environmental sensor 340.

Sensor 344 is, for example, a depth sensor. Sensor 344 outputs the detection result to object position detection unit 346. Note that camera 342 and sensor 344 may be distance sensors.

Based on the image captured by the camera 342 and the detection result detected by the sensor 344, the object position detection unit 346 detects the three-dimensional position, size, shape, etc. of the target object in the captured image using a known method. The object position detection unit 346 estimates the position of the object by performing image processing (edge detection, binarization processing, feature extraction, image enhancement processing, image extraction, pattern matching processing, etc.) on the image captured by the camera 342 with reference to a pattern matching model stored in the object position detection unit 346. Note that, when multiple objects are detected from the captured image, the object position detection unit 346 detects the position of each object. The object position detection unit 346 transmits the detected object position information to the control device 360. Note that the data transmitted by the environmental sensor 340 may be, for example, a point cloud having position information.

The control device 360 includes, for example, a communication control unit 362, an acquisition unit 364, an operation content generation unit 366, and an output control unit 368. The communication control unit 362, the acquisition unit 364, the operation content generation unit 366, and the output control unit 368 are realized, for example, by a hardware processor such as a CPU executing a program (software). Some or all of these components may be realized by hardware (circuitry) such as an LSI, ASIC, FPGA, or GPU, or may be realized by a combination of software and hardware. The program may be stored in advance in a storage device (non-transient storage medium) such as an HDD or flash memory, or may be stored in a removable storage medium (non-transient storage medium) such as a DVD or CD-ROM, and installed by inserting the storage medium into a drive device.

The communication control unit 362 communicates with the management server 100, for example, by wireless communication via the communication unit 310, to send and receive information. The communication control unit 362 may also communicate with other operation terminals 300 via the communication unit 310. The communication control unit 362 also communicates with the HMD 320, the operation device 330, and the environmental sensor 340.

The acquisition unit 364 acquires information obtained from the HMD 320, the operation device 330, and the environmental sensor 340. The operation content generation unit 366 generates operation content for the robot device 200 based on the information acquired by the acquisition unit 364. For example, the operation content generation unit 366 generates operation content related to gaze information and head direction based on information acquired from the HMD 320. The operation content generation unit 366 also generates operation content related to the movement of the arm unit 210A of the robot device 200 based on information acquired from the operation device 330. The operation content generation unit 366 also generates operation content related to the posture, movement direction, and movement amount of the robot device 200 based on information acquired from the environmental sensor 340. The information generated by the operation content generation unit 366 is transmitted to the management server 100 via the communication unit 310.

The output control unit 368 outputs the provided information acquired from the management server 100 to the HMD 320 and the operation device 330. For example, the output control unit 368 causes the image acquired from the management server 100 to be output to the image display unit 322 of the HMD 320. The output control unit 368 also causes the speaker 328 to output voice, warning sound, etc. based on the information acquired from the management server 100. The output control unit 368 also causes the operation device 330 to output feedback information for conveying tactile information acquired from the management server 100 to the operator.

The storage unit 380 is realized, for example, by a HDD, a flash memory, an EEPROM, a ROM, or a RAM. The storage unit 380 stores, for example, programs and other information.

[Regarding remote control of the robot device according to the embodiment]
Next, the remote operation of the robot device 200 by the remote operation system 1 according to the embodiment will be specifically described. FIG. 8 is a diagram for explaining the remote operation of the robot device 200 according to the embodiment. FIG. 8 shows an example in which one robot device 200 is operated based on the operation contents of two operators U1 and U2. As shown in FIG. 8, the operators U1 and U2 each wear an HMD 320 and an operation device 330. The operation terminals 300A-1 and 300A-2 are terminals of the same company (for example, company A), but may be installed in different places.

In addition, the operators U1 and U2 are provided with environmental sensors 340 that measure the movements of the operators and the surrounding environment. Note that a configuration similar to the environmental sensor 340 may be attached to the robot device 200, for example. The operation device 330 includes, for example, an operation device 330a worn on the left hand of the operator and an operation device 330b worn on the right hand. The operation contents input from each of the operation devices 330a and 330b are used to control the movements of the left hand LH and right hand RH of the robot device 200. The position of the operator's arms, the direction of the face, etc. are detected by the environmental sensor 340. Each detected data is transmitted to the management server 100 via a network.

The management server 100 estimates the intention of each user based on the operation contents obtained from the operation terminals 300A-1 and 300A-2. Here, an example of information estimated by the intention estimation unit 153 will be specifically described. The intention estimation unit 153 estimates the operator's intention of operation based on the acquired operation contents from the operator. The intention estimation unit 153 classifies the posture of the arm including the gripping part of the robot device by classifying the posture of the operator's arm based on the operator sensor value of the operation terminal 300. In addition, the intention estimation unit 153 estimates the intention of the operation that the operator wants the robot device 200 to perform based on the classification result. The intention estimation unit 153 estimates, for example, the movement of hands and fingers at each time, the purpose of the work that the robot device 200 wants to perform, the work content, the movement of hands and fingers at each time, etc. as the operator's intention of operation. The purpose of the work is, for example, grasping an object, moving an object, etc. Examples of tasks include grasping and lifting an object, grasping and moving an object, etc.

The intention estimation unit 153 may also estimate the operator's intention to operate by, for example, inputting the operation content into a learned model or intention estimation information 172B stored in the storage unit 170. In the embodiment, the intention estimation is performed by classifying the grip posture, so that the operator's intention to operate can be estimated with high accuracy. Note that other methods may be used to classify the grip posture.

The intention estimation unit 153 may also perform an integrated estimation using the gaze and arm movement. In this case, the intention estimation unit 153 may input gaze information, hand movement information, and position information of the object OB on the table TB to a trained model to estimate the operator's intention to operate.

8, the intention estimation unit 153 estimates the object to be grasped based on, for example, line of sight information. Next, the intention estimation unit 153 estimates the posture of the operator's hand based on the estimated object to be grasped. In addition, the intention estimation unit 153 may, for example, first estimate the posture of the operator's hand based on the operation content, and then estimate the object to be grasped from the estimated posture of the operator's hand. Note that, when the same or similar operation content is input from the operators U1 and U2, the intention estimation is performed based on the operation content from the operator with the higher priority. In addition, when the operation of different parts is assigned to each of the operators U1 and U2, the intention estimation unit 153 estimates the intention based on the operation content of each. In the following, it is assumed that the operator U1 operates the arm unit 210A with his right hand, and the operator U2 operates the arm unit 210A with his left hand.

For example, when an object OB is placed on the table TB shown in FIG. 8, the intention estimation unit 153 estimates that the object OB will be grasped based on the posture of the hand. The intention estimation unit 153 may also estimate in advance the future trajectory of the hand that the operator intends to grasp based on the operation content and state information of the robot device 200. The intention estimation unit 153 may also estimate the object to be grasped and the position of the object by using the detection results detected by the sensor, the results of image processing of the image captured by the environmental sensor 340, etc.

In addition, since the coordinate system is different between the environment in which the operator operates and the robot operating environment, for example, calibration between the operator's operating environment and the robot operating environment may be performed when the robot device 200 is started. In addition, when grasping, the management server 100 may determine the grasping position based on the grasping force of the robot device and the frictional force between the object and the grasping part, etc., taking into account the error in the grasping position at the time of grasping.

When the robot device 200 is not remotely operated, its operation is controlled according to the control of the control device 240, and when it is remotely operated, its operation is controlled according to operation control information generated by the management server 100 based on the estimated intention, etc.

The control device 240 of the robot device 200 controls the driving unit 210 based on the operation control information from the management server 100. The robot device 200 also outputs to the management server 100 information such as sensor information detected by the sensor 204, images captured by the camera 206, and audio output by the microphone 208. The management server 100 generates information to be provided to the operation terminals 300A-1 and 300A-2 based on the acquired sensor information, images, audio, and the like. In this case, the management server 100 may generate an image in which at least a part of the sensor information is superimposed on the image. The management server 100 generates information to be provided that also includes information to be transmitted by touch. In addition, since the robot device 200 can be operated by multiple operators, an image in which information for identifying the operator of each part of the robot device 200 is superimposed on the camera image may be generated. The operation terminal 300 displays the image output from the management server 100.

Figure 9 is a diagram showing an example of an image IM10 displayed by the operation terminal 300. The display mode of the image IM10, such as the layout and display contents, is not limited to the example of Figure 9. The same applies to the display modes of other images described below. In the image IM10 shown in Figure 9, a sensor information display area A11, an operator information display area A12, and a work status display area A13 are superimposed on the image IM10 including the arm part (left hand LH, right hand RH) of the robot device 200 captured by the camera 206. In the example of Figure 9, an object OB1 is placed on a table TB in front of the robot device 200. The left hand LH of the robot device 200 is held up while holding an object OB2, and the right hand RH of the robot device 200 is held up while holding an object OB3. The sensor information display area A11, the operator information display area A12, and the work status display area A13 may be displayed with a predetermined transmittance so that objects in real space are not obstructed.

9, the sensor information display area A11L displays sensor information related to the left hand of the robot device 200. The sensor information display area A11R displays sensor information related to the right hand of the robot device 200 (e.g., position, pressure, temperature, etc.). In the example of FIG. 9, the operator information display area A12L displays information related to the operator of the left hand of the robot device 200. The operator information display area A12R displays information related to the operator of the left hand of the robot device 200. This allows, for example, when the robot device 200 is operated by multiple people, the other operators to accurately know who is operating each part of the robot device 200. In addition, since it is possible to know not only the sensor information of the part being operated by oneself but also information related to the operation of other operators, more accurate processing can be performed.

The work status display area A13 displays the results of each intention estimation. In the example of FIG. 9, information on the intentions estimated from the operation contents of each of the operators U1 and U2 is displayed. Furthermore, when the management server 100 estimates through intention estimation that each operator will perform the same or similar processing, it executes one intention with priority based on priority information. Furthermore, in this case, a warning may be presented to the other operator by displaying the information in the work status display area A13 and changing the audio or display mode. In the example of FIG. 9, warning information is presented because both operators U1 and U2 are trying to place the object they are holding on the object OB1. The warning mode may be different depending on the degree of danger.

In this case, the operation control unit 154, for example, refers to the priority of each operator and causes the robot device 200 to execute an operation based on the intention of the operator with the highest priority. Furthermore, when a process is executed based on the priority, the management server 100 causes the display unit of the operation terminal 300 to display an image including the process content.

Figure 10 is a diagram showing an example of an image IM20 displayed on the operation terminal 300 when a task based on priority is performed. In the example of Figure 10, the content displayed in the work status display area A13 is different from that of the image IM10 shown in Figure 9. Therefore, the following mainly describes the work status display area A13. In the example of Figure 10, the work status display area A13 displays information indicating that the instructions of the first operator U2 have been executed with priority based on the priority. This allows the operators U1 and U2 to more accurately grasp the intention estimation results for their respective operation instructions and the actual work status.

The intention estimation unit 153 may also store the estimation result in a memory unit and use it the next time or later when estimating intentions. When an unknown action is detected, the intention estimation unit 153 may inquire of the operator as to whether to cause the robot device 200 to execute the action.

11 is a diagram showing an example of an image IM30 that inquires of the operator about the operation to be performed by the robot device. Image IM30 includes, for example, an operation inquiry information display area A31 and a switch display area A32. In the operation inquiry information display area A31, information is displayed for inquiring whether or not the robot device 200 needs to execute an operation content input by the operation device 330. For example, if an operation of touching the head is detected but the intention of this operation cannot be estimated, the intention estimation unit 153 displays an image showing text information such as "An operation of touching the head has been detected. Do you want the robot to perform this operation?" on the operation terminal 300, as shown in the operation inquiry information display area A31 in FIG. 11.

In the switch display area A32, for example, icons IC31 and IC32 are displayed. Icons IC31 and IC32 are, for example, GUI (Graphical User Interface) switches. When icon IC31 is selected, the management server 100 generates operation control information for causing the robot device 200 to execute the instructed operation content. When icon IC12 is selected, the HMI control unit 180 does not cause the robot device 200 to execute the instructed operation content. Furthermore, the management server 100 may execute a process to prevent the robot device from inputting the same operation even if it is input in the future. This makes it possible to suppress the robot from executing operations that are unconsciously performed due to the operator's habits, etc., and to prevent unnecessary operations.

In addition, operators around the robot device 200 may feel uneasy because they do not know who is operating the robot device 200. Therefore, the management server 100 may display information about the operator of the robot device 200 on the monitor of the robot device 200. In this case, if the robot device 200 is operated by multiple operators, the operator of each part may be presented.

12 is a diagram showing an example of an image IM40 displayed on the monitor 214 of the robot device 200. The image IM40 includes, for example, one or both of the operator information display areas A41 and A42. The operator information display area A41 displays text information related to the operator who operates the robot device 200. The operator information display area A41 may also include information related to the parts of the robot device 200 operated by the operator. In the example of FIG. 12, the operator information display area A41 displays text information such as "The right hand of the operator U2 and the left hand of the operator U1 are operating this robot." The operator information display area A42 displays an avatar image registered in the user DB. This allows surrounding operators to more accurately recognize the person remotely operating the robot and the parts they are responsible for.

[Processing sequence]
Next, the processing executed by the remote operation system in the embodiment will be described with reference to a sequence diagram. Fig. 13 is a sequence diagram for explaining the processing executed by the remote operation system 1 in the embodiment. In the example of Fig. 13, the explanation will be given using the management server 100, the robot device 200, and the operation terminals 300-1 and 300-2. Also, in Fig. 13, an example will be explained in which the robot device 200 is operated by the two operation terminals 300-1 and 300-2.

The operation terminal 300-1 and the operation terminal 300-2 each access the management server 100 to remotely operate the robot device 200 (steps S100, S102). The management server 100 performs authentication processing to determine whether or not the remote operation system 1 can be used based on authentication information and the like input from the operation terminals 300-1 and 300-2 (step S104). The following describes a case where use is permitted by authentication. If use is permitted, the management section of the management server 100 assigns the robot device 200 to be operated by the operation terminals 300-1 and 300-2 (step S106) and requests startup control of the assigned robot device (step S108).

The robot device 200 starts up the device based on a start-up request from the management server 100 (step S110) and outputs the detection results of the robot device 200's sensor, images captured by the camera, etc. to the management server 100 (step S112).

The management server 100 acquires the information transmitted from the robot device 200, and generates information (provided information) to be provided to the operation terminals 300-1 and 300-2 based on the acquired information (step S114). The management server 100 then transmits the generated provided information to the operation terminals 300-1 and 300-2, respectively (steps S116 and S118).

Each of the operation terminals 300-1 and 300-2 receives the provided information sent from the management server 100, and provides the information by outputting the received information from the HMD and the operation device, etc. (steps S120 and S122).

Next, the operation terminals 300-1 and 300-2 transmit the operation details related to the robot device 200 to the management server 100 (steps S124 and S126). The management server 100 performs intention estimation based on the operation details obtained from each of the operation terminals 300-1 and 300-2 (step S128), generates operation control information based on the respective intention estimation results (step S130), and transmits the generated operation control information to the robot device 200 (step S132).

The robot device 200 performs an operation based on the operation control information (step S134), and transmits information obtained from sensors, etc. (sensor information) and camera images, etc. during or as a result of the operation, to the management server 100 (step S136).

The management server 100 stores the received information in the storage unit 170 (step S138), generates information to be provided for transmitting the acquired information to the operation terminals 300-1 and 300-2 (step S140), and transmits the generated information to be provided to the operation terminals 300-1 and 300-2 (steps S142 and S144). Each of the operation terminals 300-1 and 300-2 outputs the acquired information and provides the information to the operator (steps S146 and S148). Thereafter, the processing of steps S124 to S148 is continued until the operation of the robot device 200 is completed.

<Modification>
In the embodiments, a part or all of the processes of the management server 100, the robot device 200, and the operation terminal 300 may be realized by AI (Artificial Intelligence) technology. In addition, a part or all of the functional units or information stored in the functional units or storage units included in the management server 100, the robot device 200, and the operation terminal 300 may be included in other devices.

In the above embodiment, the case where a single robot device 200 is operated by multiple operators has been described, but a single operator may operate multiple robot devices 200. In this case, sensor information detected by each of the multiple robot devices 200 is displayed on the image display unit 322 of the HMD 320. The management server 100 may also transmit operation history information, intent estimation information, and the like to the robot manufacturer terminal 400. This can be used for developing the robot manufacturer terminal 400 and products, updating programs, and the like. In this case, a discount or reduction service for system usage may be provided to a company that has performed eco-driving of the robot device 200 (for example, low-power driving, low-load driving, low-communication driving), or a company that has provided a robot control method suitable for an operator or telework characteristics. This can promote the provision of information from users, which can be useful for improving services.

In addition, in the above embodiment, when multiple operators operate one robot device 200, an example was shown in which each operator's operating intention and work status were displayed. However, for example, the above-mentioned intention estimation results, work status, and information about the operator may be provided between multiple robot devices 200 that exist within a specified distance, or between robot devices performing collaborative work. In this way, by using the intention estimation function between robots, it becomes possible to share awareness more effectively than between humans. In addition, since it is possible to obtain the status of surrounding robot devices 200, group work, etc. can be performed smoothly.

The embodiment described above can operate the robot more appropriately by, for example, providing a server with a communication unit that communicates via a network NW with one or more robotic devices 200 and one or more operation terminals 300 that remotely operate at least one of the one or more robotic devices 200, an acquisition unit that acquires operation content input by the operation terminal 300, an intention estimation unit that estimates the intention of the operator of the operation terminal 300 based on the operation content acquired by the acquisition unit, and an operation control unit that controls the operation of the robotic device 200 based on the intention result estimated by the intention estimation unit.

In addition, according to the above-described embodiment, when performing teleworking by remotely controlling a robot, various devices can be used more generally and can be operated by a team. Furthermore, in the past, only limited tasks could be performed with a limited number of dedicated robots even when remotely controlled, but according to this embodiment, anyone can easily perform teleworking with a highly versatile robot.

According to the embodiment, the robot information and the sensor information (information that may include corporate secrets) can be managed separately, and a server capable of handling confidential information can be provided. Remote operators can access the same server and can share the control rights and sensor information of the same robot or different robots. For example, when operating the same robot device, multiple operators can operate different arms, so that a senior and a junior can simultaneously operate the same arm of the same robot, and can add and subtract forces while sharing sensor information such as tactile sense. When operating different robots, telepathy is possible by sharing the intention estimation function and disclosing the intentions of others. Furthermore, by using the intention estimation function, a warning can be issued to prevent the robots from moving in the same direction or colliding when trying to pick up the same thing. According to the embodiment, one operator can operate two robots. In this case, for example, if two robots are to carry a large desk, one robot is brought to the edge of the desk in advance and synchronized with the other robot. Also, when the other robot is brought to the other end and an action is attempted to lift it, the robots are lifted together by synchronization processing. Furthermore, in the embodiment, more accurate work is possible than if humans were to call out to each other to perform the operation. Also, in the embodiment, when one operator has multiple robots perform the same work, the supported teleoperation function can absorb individual differences between the robots and the individual differences between the objects being worked on.

The above-described embodiment can be expressed as follows.
A server computer that communicates with a mobile object and a mobile terminal of a user of the mobile object,
A storage device storing a program;
a hardware processor;
The hardware processor executes the program stored in the storage device,
communicating with one or more robot devices and one or more operation terminals that remotely operate at least one of the one or more robot devices via a network;
Acquire operation contents inputted by the operation terminal;
Estimating the intention of the operator of the operation terminal based on the acquired operation content;
Controlling the operation of the robot device based on the estimated intention result.
The server is configured as follows:

The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made without departing from the spirit of the present invention.

1...Remote operation system, 100...Management server, 110, 202, 310...Communication unit, 120...Communication control unit, 130...Acquisition unit, 140...Management unit, 150...Control unit, 170...Memory unit, 200...Robot device, 204...Sensor, 206...Camera, 208...Microphone, 210...Drive unit, 214...Monitor, 216...Speaker, 240...Control device, 260...Memory unit, 300...Operation terminal, 320...HMD, 330...Operation device, 340...Environment sensor, 360...Control device, 380...Memory unit

Claims (11)

a communication unit that communicates with one or more robot devices and one or more operation terminals that remotely operate at least one of the one or more robot devices via a network;
an acquisition unit that acquires operation content inputted by the operation terminal;
an intention estimation unit that estimates an intention of an operator of the operation terminal based on the operation content acquired by the acquisition unit;
an operation control unit that controls an operation of the robot apparatus based on the intention result estimated by the intention estimation unit ,
the acquiring unit acquires robot control information relating to an operation of the robot device and sensor information detected by a sensor provided in the robot device;
a management unit that sets different access rights to the robot control information and the sensor information acquired by the acquisition unit, and enables acquisition of either the robot control information or the sensor information when the set access rights are confirmed,
server . a communication unit that communicates with one or more robot devices and one or more operation terminals that remotely operate at least one of the one or more robot devices via a network;
an acquisition unit that acquires operation content inputted by the operation terminal;
an intention estimation unit that estimates an intention of an operator of the operation terminal based on the operation content acquired by the acquisition unit;
an operation control unit that controls an operation of the robot apparatus based on the intention result estimated by the intention estimation unit ,
when controlling an operation of one robot device based on operation contents acquired from a plurality of operation terminals, the operation control unit generates operation control information for the robot device based on a priority of operators who operate the operation terminals;
the intention estimation unit estimates an intention of each of operators of a plurality of operation terminals that operate the one robot device,
an output control unit that outputs an intention estimation result of each of the operators estimated by the intention estimation unit to each of the multiple operation terminals;
server .
the acquiring unit acquires robot control information relating to an operation of the robot device and sensor information detected by a sensor provided in the robot device;
a management unit that sets different access rights to the robot control information and the sensor information acquired by the acquisition unit, and enables acquisition of either the robot control information or the sensor information when the set access rights are confirmed,
The server according to claim 2 . The management unit sets a strength of the access right for the sensor information to be greater than that for the robot control information.
The server of claim 1 . the operation control unit, when controlling an operation of one robot device based on operation contents acquired from a plurality of operation terminals, generates operation control information for the robot device based on priorities of operators who operate the operation terminals.
The server of claim 1 . when the acquiring unit acquires the same or similar operation content from the plurality of operation terminals, the operation control unit controls the operation of the robot device by giving priority to the operation content from the operator with a high priority.
The server according to claim 5 . A server according to any one of claims 1 to 6;
the one or more robotic devices;
The one or more operation terminals;
A remote control system comprising: The computer
communicating with one or more robot devices and one or more operation terminals that remotely operate at least one of the one or more robot devices via a network;
Acquire operation contents inputted by the operation terminal;
Estimating the intention of the operator of the operation terminal based on the acquired operation content;
Controlling the operation of the robot device based on the estimated intention result ;
acquiring robot control information relating to an operation of the robot device and sensor information detected by a sensor provided in the robot device;
a different access right is set for each of the acquired robot control information and the acquired sensor information, and when the set access right is confirmed, either the robot control information or the sensor information can be acquired;
Remote operation method. The computer
communicating with one or more robot devices and one or more operation terminals that remotely operate at least one of the one or more robot devices via a network;
Acquire operation contents inputted by the operation terminal;
Estimating the intention of the operator of the operation terminal based on the acquired operation content;
Controlling the operation of the robot device based on the estimated intention result ;
generating operation control information for the robot device based on priorities of operators who operate the operation terminals when controlling the operation of one robot device based on operation contents acquired from a plurality of operation terminals;
Estimating the intention of each of the operators of a plurality of operation terminals that operate the one robot device;
outputting the estimated intention estimation result of each of the operators to each of the plurality of operation terminals;
Remote operation method. On the computer,
communicating with one or more robot devices and one or more operation terminals for remotely operating at least one of the one or more robot devices via a network;
Acquiring operation details inputted by the operation terminal;
Inferring the intention of the operator of the operation terminal based on the acquired operation content;
Controlling the operation of the robot device based on the estimated intention result ;
acquiring robot control information relating to an operation of the robot device and sensor information detected by a sensor provided in the robot device;
A different access right is set for each of the acquired robot control information and the acquired sensor information, and when the set access right is confirmed, either the robot control information or the sensor information can be acquired.
program. On the computer,
communicating with one or more robot devices and one or more operation terminals for remotely operating at least one of the one or more robot devices via a network;
Acquiring operation details inputted by the operation terminal;
Inferring the intention of the operator of the operation terminal based on the acquired operation content;
Controlling the operation of the robot device based on the estimated intention result ;
When controlling the operation of one robot device based on operation contents acquired from a plurality of operation terminals, generating operation control information for the robot device based on the priority of operators who operate the operation terminals;
Inferring the intentions of the respective operators of a plurality of operation terminals which operate the one robot device;
outputting the estimated intention estimation result of each of the operators to each of the plurality of operation terminals;
program.

Images