WO2022199650A1 - Computer-readable storage medium, electronic device, and surgical robot system - Google Patents

Abstract

A computer-readable storage medium, an electronic device, and a surgical robot system. The computer-readable storage medium stores a program; when the program is executed, the following steps are performed: establishing a three-dimensional model according to first image information in a body of a surgery subject; planning a first path of a puncturing apparatus (10) according to a first hole (M₁) in a body surface of the surgery subject, a predetermined position (q_goal) in the body of the surgery subject, and the three-dimensional model, such that when the puncturing apparatus (10) moves along the first path, the puncturing tail end of the puncturing apparatus (10) penetrates through the body surface of the surgery subject at the first hole (M₁) and reaches the predetermined position (q_goal); and driving the puncturing apparatus (10) to move along the first path. When the computer-readable storage medium is applied to the surgical robot system and used for puncturing on the body of the surgery subject, the degree of automation of puncturing can be improved, the dependence on the experience of a doctor is reduced, and the safety of a surgery is improved.

Description

Computer-readable storage medium, electronic device, and surgical robot system technical field

The invention belongs to the technical field of medical devices, and in particular relates to a computer-readable storage medium, an electronic device and a surgical robot system.

Background technique

Surgical robots are designed to precisely perform complex surgical procedures in a minimally invasive manner. Surgical robots have been developed when traditional surgical operations face various limitations. Surgical robots break through the limitations of the human eye and can use stereo imaging technology to more clearly present the internal organs of the human body to the surgeon. And for the narrow area that some people's hands cannot reach, the surgical robot can still control the surgical instruments to complete the movement, swing, clamping and 360° rotation, and can avoid shaking, improve the accuracy of surgery, and further achieve smaller wounds and less bleeding. , The advantages of fast postoperative recovery and greatly shortening the postoperative hospital stay of the surgical subject. Therefore, surgical robots are favored by doctors and patients, and are widely used in their respective clinical operations.

Like traditional surgery, before using a surgical robot to perform surgery, it is necessary to locate the lesion, determine the perforation point according to the location of the lesion, and then perform the perforation at the perforation point to carry out the surgical operation. The punching device used for punching is usually very sharp, and the operator needs to use a lot of force to pierce the body surface of the surgical object. Therefore, the punching operation is very dependent on the experience of the operator. Excessive force is extremely used when making a hole, so that the punching device stabs the tissue after penetrating the body surface, which brings unnecessary trauma to the operation object and affects the safety of the operation.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a computer storage medium, electronic equipment and surgical robot system, which aims to automatically plan and control the tool arm to drive the punching device to complete the punching operation through the surgical robot system, and reduce the experience of the surgical punching operation on the doctor. The degree of dependence reduces the risk of surgical drilling operations and improves surgical safety.

To achieve the above object, the present invention provides a computer-readable storage medium on which a program is stored, and when the program is executed, the following steps are performed:

establishing a three-dimensional model according to the first image information in the surgical object;

The first path of the punching device is planned according to the first hole position on the body surface of the surgical object, the predetermined position in the body of the surgical object, and the three-dimensional model, so that the punching device can be punched when the punching device moves along the first path. The punching end of the device passes through the body surface of the surgical subject at the first hole position and reaches the predetermined position.

Optionally, the first path includes a first global path, and when the punching device moves along the first global path, the punching end can reach the predetermined position.

Optionally, the program performs:

When there is an obstacle on the first global path, a first partial path is planned, the first partial path is planned outside the boundary of the obstacle, and the starting point and the ending point of the first partial path are both on the first global path.

Optionally, the program performs the following steps to plan the first partial path:

An artificial potential field is established according to the three-dimensional model and the predetermined position, and the first local path is planned according to the artificial potential field.

Optionally, there is a position q in the artificial potential field, and the potential function of the position q in the artificial potential field is the sum of the attractive potential function u _att (q) and the repulsive potential function U _rep (q) :

U(q)= _Uatt (q)+ _Urep (q),

In the formula, ζ is the attraction gain; d(q, q _goal ) is the distance between the position q and the predetermined position; D(q) is the distance from the nearest obstacle to the position q; η is Repulsive force gain; Q ^* is the force threshold of the obstacle, when the distance from the obstacle to the punching end is greater than Q ^* , the obstacle will not generate a repulsive force on the punching end.

Optionally, when planning the first path, the program further performs the following steps:

performing an expansion calculation on the three-dimensional model to expand the boundary of the tissue model in the three-dimensional model to the outside by a safe distance;

The first path is planned according to the expanded three-dimensional model.

Optionally, when the punching device moves along the first path, the maximum speed is V _max1 , the acceleration is a ₁ , the safety distance is d ₁ , and the following relationship is satisfied:

d ₁ =V _max1 ² /(2a ₁ ).

Optionally, after the punching end penetrates the body surface of the surgical subject at the first hole position, the procedure further performs the following steps:

The punching state information is acquired according to the second image information of the punching end and the three-dimensional model, and guidance information is generated.

Optionally, an image acquisition device is used to penetrate the body surface from the second hole on the body surface of the operation object and enter the body of the operation object to obtain the second image information, and the program is further used to perform the following steps:

The target pose of the image acquisition device in the surgical object is planned according to the first hole position and the three-dimensional model, so that when the image acquisition device is in the target pose, the first hole position is located in the within the field of view of the image acquisition device;

Plan the motion scheme of the image acquisition device according to the initial posture of the image acquisition device in the surgical object, the three-dimensional model and the target posture, and drive the image acquisition device to move according to the motion plan and reach the target pose.

Optionally, the initial pose includes an initial position, and the target pose includes a target position; the motion scheme includes a second global path planned according to the initial position, the three-dimensional model, and the target position, The target position can be reached when the image acquisition device is moved along the second global path.

Optionally, when there is an obstacle on the second global path, the motion scheme further includes a second partial path, the second partial path is located outside the boundary of the obstacle, and the second partial path is located outside the boundary of the obstacle. Both the start point and the end point of the path are on the second global path.

Optionally, the target posture further includes a target posture, and the motion plan further includes a rotation plan planned according to the current posture when the image acquisition device reaches the target position and the target posture. The acquisition device is capable of reaching the target pose when rotated according to the rotation scheme at the target position.

Optionally, the second image information is acquired through an image acquisition device, and when acquiring the second image information, the program further performs the following steps:

Visual servoing is used to control the pose of the image capturing device, so that the punching end of the punching device is within the field of view of the image capturing device.

Optionally, the punching state information includes at least one of position information of the punching device, speed information of the punching device, and punching progress information;

The program performs at least one of the following steps to obtain the punch state information:

Obtain the position information of the punching device according to the second image information and the three-dimensional model;

Acquiring speed information of the punching device according to the position change of the punching device;

generating the punching progress information according to the current position information of the punching device and the predetermined position;

Optionally, the guidance information includes collision reminder information; the program executes the following steps to obtain the guidance information:

The collision probability is acquired according to the position information of the punching end, the speed information of the punching end, and the three-dimensional model, and collision prompt information is generated.

Optionally, the program performs the following steps to obtain the collision reminder information:

According to the position information of the punching end and the three-dimensional model, obtain the target tissue closest to the punching end, and calculate the distance between the punching end and the target tissue;

Calculate the collision occurrence time according to the speed of the punching end and the distance;

It is judged whether the collision occurrence time is greater than the set time threshold, and if not, it is judged that the collision probability is high, and the prompt information is generated.

Optionally, the program performs the following steps to determine the first hole position on the body surface of the surgical object:

establishing a first sign image model according to the first body surface information and the lesion information of the surgical object in the first state, where the first sign image model is used to plan the first pre-hole location;

establishing a second physical sign image model according to the second body surface information of the surgical object in the second state;

The second physical sign image model and the first physical sign image model are registered to obtain a first target hole position corresponding to the first pre-hole position on the second physical sign image model, and the The first target hole position is used to be guided to the body surface of the surgical object to obtain the first hole position.

To achieve the above object, the present invention also provides an electronic device, comprising a processor and the computer-readable storage medium described in any preceding item, where the processor is configured to execute a program stored on the computer-readable storage medium .

In order to achieve the above object, the present invention also provides a surgical robot system, comprising:

The tool arm is used for connecting the punching device, the punching device includes a punching end, and the punching end is used to penetrate the body surface of the surgical object from the first hole position on the body surface of the surgical object and reach the hole in the body of the surgical object. predetermined location;

an image arm for connecting to an image acquisition device, the image acquisition device for acquiring first image information inside the surgical subject; and,

A control unit, connected in communication with the tool arm, the image arm and the image acquisition device, and configured to implement the steps performed by the program as described in any preceding item.

Optionally, after the punching end penetrates the body surface of the surgical object at the first hole position, the image acquisition device further collects second image information of the punching end; the control unit further is configured to obtain punching state information and generate guide information according to the second image information and the three-dimensional model;

The surgical robot system further includes a prompting device, connected in communication with the control unit, and used for receiving and displaying the punching state information and the guidance information.

Optionally, the surgical robot system includes a first imaging device and a second imaging device, the first imaging device and the second imaging device are both connected in communication with the control unit, and the first imaging device is used for acquiring first body surface information and lesion information of the surgical object in the first state, the second imaging device is used to acquire the second body surface information of the surgical object in the second state; the control unit according to the first The body surface information and the lesion information establish a first sign image model, and a second sign image model is established according to the second body surface information, and the first sign image model and the second sign image model are used to obtain all the signs. Describe the first hole position.

Optionally, the first imaging device includes any one of MRI, X-ray device or B-ultrasound; the second imaging device includes a binocular vision camera or a structured light camera.

Compared with the prior art, the computer-readable storage medium, electronic device and surgical robot system of the present invention have the following advantages:

First, the aforementioned computer-readable storage is then stored with a program, and when the program is executed, the following steps are performed: a three-dimensional model is established according to the first image information in the body of the operation object, according to the first hole position on the body surface of the operation object, The predetermined position in the surgical object and the three-dimensional model plan the first path of the punching device, so that the punching end of the punching device is in the first hole when the punching device moves along the first path The position penetrates the body surface of the surgical object and reaches the predetermined position; and the punching device is driven to move along the first path. When the computer-readable storage medium is applied to the surgical robot system, and the surgical robot system is used to perform the punching operation, since the first path of the punching device is pre-planned, it is avoided to stab the target tissue during the punching process. The operation object brings unnecessary trauma, reduces the degree of dependence on the doctor's experience, and improves the safety of the operation.

Second, the surgical robot system also collects the second image information of the punching end entering the surgical object through an image acquisition device, and obtains punching state information according to the second image information and the three-dimensional model, The punching process can be monitored in real time to ensure the smooth execution of the punching operation and further improve the safety.

Third, the surgical robot system also plans a target pose and a motion plan of the image acquisition device, and drives the image acquisition device to move according to the motion plan to reach the target pose, so that the image The acquisition device can accurately collect the first image information, monitor the punching process, and prevent the image acquisition device from causing damage to the target tissue during the process of moving to the target pose.

Fourth, when acquiring the first hole position, first plan the first pre-hole position on the first sign image model of the surgical object in the first state, and analyze the second sign of the surgical object in the second state. The image model and the first sign image model are registered to obtain a first target hole position on the second sign image model, and then the first target hole position is directed to the surgical object in the second state The first hole position is obtained on the body surface of the machine, which reduces the problem of inaccurate punch hole position caused by changes in the body position and state of the surgical object before surgery, further reduces the degree of dependence on the doctor's experience, and improves the safety of the operation.

Description of drawings

The accompanying drawings are used for better understanding of the present invention and do not constitute an improper limitation of the present invention. in:

1 is a schematic diagram of an application scenario of a surgical robot system provided by the present invention according to an embodiment;

2 is a schematic diagram of a punching device used by a surgical robot system according to an embodiment of the present invention when performing a punching operation;

3 is a schematic structural diagram of an endoscope used by a surgical robot system according to an embodiment of the present invention when performing a drilling operation;

4 is a schematic structural diagram of an endoscope used by a surgical robot system according to another embodiment of the present invention when performing a drilling operation;

5 is a schematic diagram of a surgical robot system according to an embodiment of the present invention when performing a drilling operation;

6 is a flowchart of the surgical robot system according to an embodiment of the present invention when performing a drilling operation;

FIG. 7 is a schematic diagram of a second imaging device according to an embodiment of the present invention when the second body surface information of a surgical subject is collected;

8 is an imaging principle diagram of a binocular vision camera according to an embodiment of the present invention;

9 is a flowchart of the control unit of the surgical robot system according to an embodiment of the present invention acquiring second body surface information and establishing a second vital sign image model;

10 is a schematic diagram of establishing a mapping relationship between a control unit coordinate system, a surgical object body surface coordinate system, and a second imaging device coordinate system in a surgical robot system according to an embodiment of the present invention;

11 is a schematic diagram of a first hole position on the body surface of an operation subject provided by the present invention according to an embodiment;

12 is a schematic diagram of a second path planned by a surgical robot system in a surgical object according to an embodiment of the present invention;

13 is a flow chart of the movement of an endoscope to a target pose in a surgical robot system according to an embodiment of the present invention;

14 is a schematic diagram of a second path planned by a control unit of a surgical robot system according to an embodiment of the present invention, in which there is a safety gap between the second path and the target tissue;

15 is a schematic diagram of planning a first partial path by a control unit of a surgical robot system according to an embodiment of the present invention;

16 is a flow chart of the control unit of the surgical robot system according to an embodiment of the present invention acquiring punching process information;

17 is a schematic diagram of a control unit of a surgical robot system according to an embodiment of the present invention acquiring collision prompt information;

FIG. 18 is a flowchart when the control unit of the surgical robot system according to an embodiment of the present invention acquires collision prompt information;

FIG. 19 is a schematic diagram of the principle of visual servo control of an endoscope by a control unit of a surgical robot system according to an embodiment of the present invention.

Detailed ways

The embodiments of the present invention are described below through specific specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the drawings provided in this embodiment are only to illustrate the basic concept of the present invention in a schematic way, so the drawings only show the components related to the present invention rather than the number, shape and the number of components in actual implementation. For dimension drawing, the type, quantity and proportion of each component can be changed at will in actual implementation, and the component layout may also be more complicated.

In addition, each embodiment of the following description has one or more technical features, but this does not mean that the person using the present invention must implement all the technical features in any embodiment at the same time, or can only implement different embodiments separately. One or all of the technical features of the . In other words, under the premise of possible implementation, those skilled in the art can selectively implement some or all of the technical features in any embodiment according to the disclosure of the present invention and depending on design specifications or implementation requirements, or The combination of some or all of the technical features in the multiple embodiments is selectively implemented, thereby increasing the flexibility of the implementation of the present invention.

As used in this specification, the singular forms "a," "an," and "the" include plural referents, and the plural forms "a plurality" include two or more referents unless the content clearly dictates otherwise. As used in this specification, the term "or" is generally employed in its sense including "and/or" unless the content clearly dictates otherwise, and the terms "installed", "connected", "connected" shall be To be understood in a broad sense, for example, it may be a fixed connection, a detachable connection, or an integral connection. It can be a mechanical connection or an electrical connection. It can be directly connected, or indirectly connected through an intermediate medium, and it can be the internal communication between two elements or the interaction relationship between the two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

In order to make the objects, advantages and features of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. It should be noted that, the accompanying drawings are all in a very simplified form and in inaccurate scales, and are only used to facilitate and clearly assist the purpose of explaining the embodiments of the present invention. The same or similar reference numbers in the drawings represent the same or similar parts.

Fig. 1 shows a schematic diagram of an application scenario of a surgical robot system according to an embodiment of the present invention; Fig. 2 shows a schematic diagram of a punching device 10 used by the surgical robot system when performing a punching operation; Fig. 3 And FIG. 4 shows a schematic structural diagram of the endoscope 20 used by the surgical robot system to perform the drilling operation. FIG. 5 shows a schematic diagram when the control unit of the surgical robot system drives the tool arm 210 to move and then drives the punching device 10 to move to perform the punching operation.

Please refer to FIG. 1 to FIG. 5 , the surgical robot system includes a control end and an execution end, the control end includes a doctor console and a doctor end control device 100 arranged on the doctor console, and the execution end includes a patient Equipment such as the terminal control device, the surgical operation device 200, and the image display device 300. Wherein, the surgical operation device 200 is provided with a tool arm 210 and an image arm 220, the tool arm 210 is used for connecting the punching device 10 and the surgical instrument, and the punching device 10 has a punching end such as a cone shape The tip 11, the tapered tip 11 is used to penetrate the skin and subcutaneous fat of the surgical subject at the _first hole position M1 on the body surface of the surgical subject, and reach the predetermined position q _goal in the surgical subject to complete the drilling operation. The surgical instrument is used to enter the body of the surgical object from the _first hole position M1 to perform a surgical operation. The image arm 220 is used to connect an image acquisition device, and the image acquisition device is used to acquire the image information of the region or device of interest (for example, the first image information, the second image information, etc. The image acquisition device is, for example, the endoscope 20 described above. It should be noted that the punching operations mentioned in this article all refer to using the surgical robot system to punch holes in the first hole on the body surface of the surgical object, so that surgical instruments can enter from the first hole. Surgical subjects inside the body and perform surgical operations.

The surgical robot system further includes a control unit, which is connected in communication with the tool arm 210 , the imaging arm 220 and the endoscope 20 . The control unit may be entirely disposed at the doctor-side control device, or the patient-side control device as a whole, or a part of the doctor-side control device and the other part of the patient-side control device. , or other locations. That is to say, the present invention does not limit the specific arrangement of the control unit, as long as it can perform related functions.

Before using the surgical robot system to make a hole at the _first hole position M1 on the body surface of the operation object, the endoscope 20 enters the body of the operation object from the second hole position on the body surface of the operation object, and collects the inside of the operation object. of the first image information. The control unit is configured to establish a three-dimensional model according to the first image information, and to plan the first hole of the punching device 10 according to the first hole position M ₁ , the three-dimensional model and the predetermined position q _goal . A path, so that when the tool arm 210 drives the drilling device 10 to move along the first path, the tapered tip 11 can reach the predetermined position q _goal to complete the drilling operation. That is, the starting point of the first path is the first hole position M ₁ , and the ending point is the predetermined position q _goal . In the embodiment of the present invention, the second hole position and the first hole position may be determined at the same time.

In this embodiment, the surgical robot system is used to punch a hole at the first hole position M1 on the body surface of the surgical object, and the _first path of the punching device 10 is also pre-planned, so that the punching device 10 Move along the first path and complete the punching operation, which reduces the dependence on the operator's personal experience, reduces the possibility of causing damage to the surgical object due to the operator's inexperience, and improves the automation level of punching. The safety of the punching operation can effectively shorten the punching time and reduce the fatigue of the operator.

Preferably, the first path includes a first global path L ₁ (as shown in FIG. 5 ), and when the punching device 10 moves along the first global path L ₁ , the tapered tip 11 can reach the predetermined position q _goal . The first global path L ₁ plays a role of overall guidance, but it may pass through the target tissue or be relatively close to the target tissue in a partial road section, so that this part of the target tissue constitutes an obstacle to the movement of the punching device 10 At this time, if the punching device 10 completely moves along the _first global path L1, it may collide with the target tissue and cause damage to the target tissue. In view of this, the first path further includes a first partial path (not shown in the figure), and the first partial path is actually a correction path. When the punching device 10 follows the first partial path The movement can avoid collision with the target tissue that constitutes the obstacle. Specifically, the first partial path is located outside the obstacle, and the start point and the end point of the first partial path are both located on the _first global path L1, so that the punching device 10 The obstacle may be avoided while moving along the first local path, and after the obstacle is avoided, the first global path may be re-entered. In this embodiment, the _first global path L1 may be planned before punching starts, and the first local route is planned during the punching process.

Not only that, before punching, the control unit is also configured to determine the _first hole position M1 and the second hole position on the body surface of the operation object, which further reduces the dependence on the doctor's experience. The method for determining the _first hole position M1 and the second hole position will be described in detail below.

In addition, the endoscope 20 also collects second image information of the tapered tip 11 entering the body of the surgical object during the drilling process, and the control unit is further configured to perform according to the second image information and the three-dimensional The model obtains the punching status information and generates guidance information to monitor the punching process in real time, and stop punching when necessary, and the operator intervenes to adjust the punching status. In this embodiment, the tapered tip 11 of the punching device 10 is provided with a marker 12 , so that the endoscope 20 can identify the tapered tip 11 and collect the first 2. Image information. The marker 12 can be, for example, a brightly colored reflective body or a luminous body.

In addition, it should also be known that the surgical robot system will punch holes at a plurality of _first hole positions M1 on the body surface of the surgical object during the actual operation. The endoscope 20 is adjusted so that the endoscope 20 is in the target posture corresponding to the corresponding _first hole position M1. In the target posture, the first hole is located within the field of view of the endoscope 20, so as to ensure that once the tapered tip 11 of the punching device 10 enters the operation object, the endoscope 20 can The second image information is immediately collected and the punching operation is monitored in real time.

In this way, in a non-limiting embodiment, the process of using the surgical robot system to perform preoperative drilling on the body surface of the surgical object may be shown in FIG. 6 , including:

Step A10: determining the _first hole position M1 and the second hole position on the body surface of the surgical object;

Step A20: Insert an endoscope into the body of the operation subject and locate it at the initial position. Specifically, punching a hole at the second hole position, and inserting an endoscope into the body of the surgical subject from the second hole position and in an initial position;

Step A30: Plan the target pose and motion scheme of the endoscope, and drive the endoscope to move to the target pose. Specifically, the endoscope is set to the target pose corresponding to the current _first hole position M1. In this step, the endoscope 20 collects the first image information in the operation object, the control unit plans the target pose of the endoscope 20, and establishes a three-dimensional model according to the first image information, and according to the first image information The three-dimensional model, the initial pose, and the target pose plan a motion scheme of the endoscope 20, and drive the endoscope 20 to move along the motion scheme from the initial pose to the target pose.

Step A40: Planning a first global path of the punching device. Specifically, the control unit plans a first global path L ₁ of the punching device according to the three-dimensional model, the first hole location M ₁ and the predetermined position.

Step A50: Drive the punching device to move and perform local path correction, and supervise the punching process. Specifically, the control unit drives the tool arm to move, and then drives the tool arm to drive the punching device to move along the _first global path L1. During this process, if the punching device encounters an obstacle, the control unit also plans the first partial path, and drives the punching device 10 to move along the first partial path to temporarily deviate from the first partial path. _A global path L1 to avoid the obstacle, and after avoiding the obstacle, return to the global path _L1 until the punching operation is completed. At the same time, the control unit also acquires punching state information according to the second image information and the three-dimensional model collected by the endoscope 20, so as to monitor the punching operation in real time.

Usually, the operator needs to make holes at a plurality of _first hole positions M1 on the body surface of the operation object, and the control unit can plan a corresponding first path for each of the _first hole positions M1 ( As shown in FIG. 6 ), the planning of the first paths of all the first hole positions M ₁ can also be completed at one time, which is not limited in the present invention. After the drilling operation at _one of the first hole positions M1 is completed, the surgical robot system drives the endoscope 20 to return to the initial position, and then drives the endoscope 20 to move to the same position as the next first hole. A target pose corresponding to _a hole position M1 is used to perform the punching operation of the next _first hole position M1 until all the _first hole positions M1 are punched. Finally, the endoscope 20 can be retracted into the stamp card on the imaging arm 220, and corresponding operations can be performed according to subsequent surgical needs. It should be noted that the execution order of each step in the process shown in FIG. 6 is not static, and can be changed according to the actual situation. For example, the step A30 can be executed synchronously with the step A40, or the step A20 In the process, the target pose and the like of the endoscope 20 are planned.

Next, this article will introduce the specific implementation of each of the above steps in detail.

Please continue to refer to FIG. 6, the step A10 includes:

Step A11: Establish a first physical sign model of the patient. Specifically, the first sign image model is established according to the first body surface information and the lesion information of the surgical object in the first state.

Step A12: Planning the pre-hole location on the first sign image model. Specifically, the pre-hole positions include a first pre-hole position and a second pre-hole position.

Step A13: Establish a second physical sign model. Specifically, the second vital sign image model is established according to the second body surface information of the surgical object in the second state.

Step A14: Register the second physical sign image model and the first physical sign image model to obtain a target hole position. Specifically, the second physical sign image model and the first physical sign image model are registered to obtain a target hole position corresponding to the pre-hole position on the second physical sign image model, the target hole position The hole positions include a first target hole position and a second target hole position.

Step A15: Electrically guide the target punching hole to the body surface of the surgical subject to obtain the actual hole position. Specifically, any suitable method is used to guide the target hole position to the body surface of the operation object to obtain the actual hole position M (as shown in FIG. 11 ), wherein the first target hole position is guided to the body surface of the operation object to obtain the The first hole position M ₁ , the second target hole position is guided to the body surface of the surgical object to obtain the second hole position. How to guide the target hole position to the body surface of the surgical object is known to those skilled in the art, and will not be described in detail here.

In this embodiment, taking laparoscopic surgery as an example, the first state refers to a state in which the operation object is before pneumoperitoneum, and the second state refers to a state in which the operation object has established pneumoperitoneum. The first body surface information and lesion information when the surgical object is in the first state are acquired by a first imaging device (not shown in the figure), where the first imaging device includes but is not limited to MRI, CT or other X-ray devices , or B-ultrasound, as long as it can perform three-dimensional scanning on the body surface and the body of the surgical target at the same time. The control unit performs three-dimensional reconstruction according to the first body surface information and the lesion information acquired by the first imaging device, obtains the first sign image model, and associates the first sign image model with the surgery. The parameters of the robot system (mainly the parameters of the tool arm 210) are matched, and then the pre-hole position is planned on the first sign image model by means of three-dimensional simulation drilling.

The second body surface information of the surgical object in the second state is collected by a second imaging device 30 (as shown in FIG. 7 ). In an implementation manner, the second imaging device 30 is a binocular vision camera At the same time, a plurality of feature points are set on the body surface of the surgical object, for example, a plurality of target pens 40 are set on the body surface of the surgical object to represent the feature points. Then, the binocular vision camera recognizes the target pen 40 to obtain image information of the target pen 40 , and the image information of the target pen 40 is used to obtain the second body surface information. This embodiment does not specifically limit the distribution of the feature points, which can be reasonably set by the operator according to the actual situation.

The binocular vision camera includes a first camera 31 and a second camera 32 (as shown in FIG. 8 ), and the image information of the target pen 40 includes the first sub-image information captured by the first camera 31 and the first sub-image information captured by the first camera 31 . The second sub-image information captured by the two cameras 32 . Fig. 8 shows the imaging principle of the binocular vision camera. In the figure, f is the focal length of the camera, b is the baseline of the first camera and the second camera, and P(x, y, z) is the captured image. of any one of the coordinates of the target pen 40 . Then, f, b and P(x, y, z) satisfy the following relationship:

Thus, the control unit can acquire the second body surface information and establish the second vital sign image model according to the method shown in FIG. 9 , including:

Step S1: Extract feature points from the first sub-image information and the second sub-image information. Specifically, the control unit extracts the feature points from the first sub-image information and the second sub-image information of the target pen;

Step S2: Pairing feature points to obtain feature point pairs. Specifically, the control unit pairs the feature points on the first sub-image information and the second sub-image information to form a feature point pair.

Step S3: Perform spatial positioning on the feature point pair to obtain a point cloud model, which is used as the second body surface data. Specifically, the control unit locates the feature point pair to a three-dimensional space position in the binocular vision camera coordinate system according to epipolar constraints. After locating all feature point pairs, a point cloud model of the feature points is obtained as the second physical sign data.

Step S4: performing three-dimensional surface reconstruction according to the second body surface data to obtain a second physical sign model. Specifically, the control unit performs three-dimensional surface reconstruction according to the second body surface information to obtain the second vital sign image model.

In an alternative implementation manner, a reflective sphere may also be used to represent the feature points, and the second imaging device 30 may also be a 3D structured light camera or a laser sensor.

After that, the control unit may use an iterative closest point method (ICP) to register the second vital sign image model and the first vital sign image model, so as to solve the problem between the second vital sign image model and the first vital sign image model. The coordinate transformation matrix of the physical sign image model, and the coordinates of the target hole position on the second physical sign image model are obtained by changing the matrix and the coordinates of the pre-hole position on the first physical sign image model accordingly.

Next, the surgeon establishes a mapping relationship between the control unit coordinate system F ₁ (X ₁ , Y ₁ , Z ₁ ) and the surgical object body surface coordinate system F ₂ (X ₂ , Y ₂ , Z ₂ ), and according to the The mapping relationship indicates the target hole position on the body surface of the surgical object to obtain the actual punching point M (as shown in FIG. 11 ). When establishing the mapping relationship, as shown in FIG. 10 , through the coordinate system F ₃ (X ₃ , Y ₃ , Z ₃ ) of the second imaging device 30 in the world coordinate system F ₀ (X ₀ , Y ₀ , Z ) ₀ ) and the control unit coordinate system F ₁ (X ₁ , Y ₁ , Z ₁ ) to establish a mapping relationship (in FIG. 10 , the surgical object-side control device 400 of the surgical robot system is taken as the control unit as an example). When the second body surface information is collected, the body surface coordinate system F ₂ (X ₂ , Y ₂ , Z ₂ ) of the surgical object and the coordinate system F ₃ (X ₃ , Y ₃ of the second imaging device) have been established , Z ₃ ), from which the coordinate system F ₁ (X ₁ , Y ₁ , Z ₁ ) of the control unit and the coordinate system F ₂ (X ₂ , Y ₂ ) of the body surface of the surgical object can be obtained, The mapping relationship between Z ₂ ).

In view of the above introduction, in the process of determining the _first hole position M1 and the second hole position on the body surface of the surgical object, the control unit is used to establish a pre-determined image model of the first physical sign of the surgical object. the hole position, and then register the first physical sign image model with the second physical sign image model in the second state during actual drilling to obtain the target hole position on the second physical sign image model, and then obtain the actual hole position. The hole position M (including the first hole position M ₁ and the second hole position), avoids the problem that the actual hole position M is inaccurate due to the different physical status of the surgical object, and is used for the subsequent punching operation. and surgical operation to lay a good foundation.

In the step A20, the operator can use any suitable method to make a hole at the second hole, and insert the endoscope 20 into the body of the subject from the second hole and place the endoscope 20 in the second hole. initial pose. The initial posture includes an initial position and an initial posture, and the initial position and the initial posture may be predetermined by the operator, or may be random, which is not limited in the present invention.

In the step A30, the target posture of the endoscope 20 includes a target position and a target posture. The motion regimen includes a second path. Similar to the first path, the second path preferably includes a second global path L ₂ and a second local path L ₃ (as shown in FIG. 12 ), wherein the control unit is configured to The initial position of the endoscope 20, the three-dimensional model and the target position plan the _second global path L2 so that when the endoscope 20 moves along the _second global path L2, the internal The scope 20 can reach the target position. The control unit is configured to plan the second global path L ₂ when there is an obstacle in the second global path L 2 (ie, target tissue that may hinder the movement of the endoscope 20 along the second global path L ₂ ) Two local paths L ₃ , the second local path L ₃ may be located outside the obstacle, and both the start point and the end point of the second local path L ₃ are located on the second global path L ₂ . In this embodiment, the second global path L ₂ adopts any one of Dijkstra algorithm, A ^* (A-Star) algorithm and random forest algorithm before the endoscope 20 moves. Plan. The second partial path L ₃ can be planned by using a dynamic window method according to the surrounding environment of the endoscope 20 during the movement of the endoscope 20 .

If the current posture of the endoscope 20 when it reaches the target position is not the target posture, the control unit is further configured to plan a rotation scheme according to the current posture and the target posture, and drive the The endoscope 20 is rotated to the target posture according to the rotation scheme. The rotation scheme includes the rotation direction and rotation angle of the endoscope 20 around the second hole.

Therefore, as shown in FIG. 13 , the step A30 specifically includes:

Step A31: Plan a second global path. Specifically, the second global path L ₂ is planned according to the three-dimensional model, the initial position of the endoscope and the target position.

Step A32: Drive the imaging arm to move, and then drive the endoscope to move along the _second global path L2. During this process, if the endoscope does not encounter an obstacle, the endoscope moves completely along the _second global path L2 until it reaches the target position. If the endoscope encounters an obstacle while moving along the _second global path L2, the control unit also plans a second partial path _L3 and drives the endoscope along the second partial path _L3 moves to avoid the obstacle, and returns to the _second global path L2 after avoiding the obstacle, so as to realize the automatic navigation control of the moving process of the endoscope until the endoscope The endoscope reaches the target position, that is, the endoscope can automatically avoid obstacles and reach the target position.

Step A33: Plan the rotation scheme and drive the endoscope to rotate to the target posture. Specifically, the rotation scheme is planned and the imaging arm is driven to move, so as to drive the endoscope to rotate to the target posture according to the rotation scheme.

Preferably, when the control unit executes the step A30, the endoscope 20 collects the first image information in real time, and the control unit updates the three-dimensional model at predetermined intervals, and then updates the The second global path L ₂ and the second local path L ₃ ensure that the endoscope 20 can avoid the obstacle in real time.

Those skilled in the art can understand that the imaging arm 220 includes at least one joint, and when the control unit plans the second path, the control unit further imposes time constraints on the second path to obtain the endoscope 20's pose (mainly position) versus time. Then, the control unit obtains the acceleration, speed and position of the joints on the image arm 220 through the inverse kinematics of the robot, so that the control unit can drive the joints on the image arm 220 according to the acceleration, The speed moves to a position corresponding to it, so as to drive the endoscope 20 to move according to the second path.

Further, when planning the second path (ie planning the second global path L ₂ and the second local path L ₃ ), the control unit is further configured to perform a dilation operation on the three-dimensional model, so that the The tissue model S in the three-dimensional model is expanded to the outside by a safe distance to obtain the expansion boundary S ₁ (as shown in FIG. 14 ). In this way, the control unit is configured to plan the second path according to the expanded three-dimensional model, so that there is a safe gap between the second path and the target tissue, and further prevent the endoscope 20 from moving During the process, it collides with the target tissue and damages the tissue, improving safety. As used herein, "outside" refers to the side toward the outside of the tissue model.

Optionally, the maximum speed of the endoscope 20 during the movement along the second path is V _max2 , the acceleration is a ₂ , the safety distance is d ₂ , and the following relationship is satisfied: d ₂ =V _max2 ² /(2a ₂ ).

In the step A33, if the current posture of the endoscope 20 when it reaches the target position coincides with the target posture, the rotation speed and the rotation angle of the endoscope 20 in the rotation scheme are both. zero.

In the step A40, the starting point of the first global path L ₁ may be the first hole position M ₁ , and the ending point is the predetermined position q _goal .

In the step A50, both the start point and the end point of the first partial path can be on the _first global path L1, so that the punching device 10 moves along the first partial path and Return to the first global path L ₁ after avoiding the obstacle.

In this step, as shown in FIG. 15 , the control unit is configured to establish an artificial potential field according to the three-dimensional model and the predetermined position q _goal , and plan the first local path according to the artificial potential field. It should be understood that when establishing the artificial potential field, an area within a certain range around the predetermined position q _goal should be within the artificial potential field, and the distance from any point in this area to the body surface of the surgical object. It is not less than the distance from the predetermined position q _goal to the first hole position M ₁ .

The artificial potential field has a position q, and the potential function of the position q in the artificial potential field is the sum of the attractive potential function U _att (q) and the repulsive force potential function U _rep (q):

U(q)= _Uatt (q)+ _Urep (q),

In the formula, ζ is the attraction gain; d(q, q _goal ) is the distance between the position q and the predetermined position q _goal ; D(q) is the closest obstacle to the position q (possibly the same as The distance of the target tissue that the punching device collides with); η is the repulsive force gain; Q ^* is the force threshold of the obstacle, when the distance from the obstacle to the tapered tip 11 of the punching device 10 When greater than Q ^* , the obstacle will not generate a repulsive force on the tapered tip 11 .

The control unit can calculate the force generated by the artificial potential field on the position q when the tapered tip 11 moves to the position q according to the potential function of the position q, and the force acts on the position q. on the tapered tip 11, so that the tapered tip 11 generates an acceleration component for avoiding obstacles (as shown by the arrow F in FIG. 15). Thus, the control unit can calculate the force that the tapered tip 11 is subjected to at any position on the _first global path L1, and plan the first local path accordingly to The _first global path L1 performs local corrections and avoids the obstacles. Those skilled in the art can understand that the position of the punching device 10 in the artificial potential field can be calculated by a robot kinematics method.

The control unit adopts scientific and reasonable calculations on the basis of the three-dimensional model to locally correct the _first global path L1, so as to prevent the punching device 10 from colliding with the target tissue during the punching process, Improve security.

Similar to the second path, please refer back to FIG. 5 , when the control unit is planning the first path (ie planning the first global path L1 and the _first local path), the control unit The expansion calculation is performed on the three-dimensional model to expand the tissue model in the three-dimensional model outward by a safe distance to obtain the expanded boundary S ₁ . Meanwhile, the control unit plans the first path according to the expanded three-dimensional model, so that there is a safety gap between the first path and the target tissue. In this embodiment, the maximum movement speed of the punching device 10 when moving along the first movement path is V _max1 , the acceleration is a ₁ , the safety distance is d ₁ , and the following relationship is satisfied: d ₁ =V _max1 ² /(2a ₁ ).

Likewise, when planning the first path (ie, the first global path L1 and the _first local path), the control unit further imposes time constraints on the first path to obtain the punching The relationship between the position of the device 10 and the time, and the inverse kinematics of the robot is used to obtain the acceleration, speed and position of the joints on the tool arm 210, so that the control unit can drive the joints on the tool arm 210. Move to the corresponding position according to the acceleration and the speed, so as to drive the punching device 10 to move according to the first path.

The punching state information includes the position of the punching device 10, the punching progress information, the moving speed of the punching device 10, and the collision prompt information. Therefore, when acquiring the punching state information, the control unit is configured to acquire the position information of the punching device 10 according to the second image information and the three-dimensional model. The punching progress information is generated according to the position information of the punching device 10 and the predetermined position q _goal . The speed information of the punching device 10 is acquired according to the position change of the punching device 10 . The collision probability between the punching device 10 and the target tissue is obtained according to the position information of the punching device 10, the speed information of the punching device 10 and the three-dimensional model, and the collision prompt information is generated.

The control unit may acquire the position of the punching device 10 in the human body according to the second image information and the structural model of the punching device 10 pre-stored in the control unit. Or, in an alternative implementation manner, all surfaces of the punching device 10 are provided with the markers, so that all the structures that the punching device 10 enters into the human body can be viewed by the endoscopy The mirror 20 is identified, so that the control unit can directly determine its position according to the image information of the punching device 10 .

And, as shown in FIG. 16 , the method for the control unit to acquire the punching progress information includes:

Step S11: Obtain the punching depth according to the current position of the punching device. Specifically, the control unit acquires the current punching depth z ₁ according to the current position of the punching device. The current drilling depth z ₁ is the distance from the current position of the tapered tip of the drilling device to the first hole position M ₁ .

Step S12: Compare the current punching depth with the expected punching depth, and use the ratio of the two as punching progress information. Specifically, the control unit compares the current punching depth z ₁ with the expected punching depth z ₀ , and obtains a ratio between the two, which is used as the punching progress information. The expected drilling depth z ₀ is the distance from the predetermined position q _goal to the first hole position M ₁ . And, when the ratio of the current drilling depth z ₁ to the expected drilling depth z ₀ is 1, the control unit determines that the drilling is completed.

As shown in FIG. 17 , when acquiring the collision prompt information, the control unit is configured to obtain a target closest to the punching device 10 according to the position of the punching device 10 and the first three-dimensional model tissue, and calculate the distance between the target tissue and the punching device 10 (mainly the tapered tip 11). Then calculate the collision occurrence time according to the moving speed of the punching device 10 and the distance, and judge whether the collision occurrence time is greater than the preset time threshold, if not, judge that the collision probability is high, and generate the collision prompt information, if yes, it is determined that the collision probability is small, and the collision prompt information is not generated or other information different from the prompt information is generated.

Therefore, as shown in FIG. 18 , the method for obtaining the collision prompt information includes the following steps:

Step S21: Determine the target tissue closest to the punching device. Specifically, the control unit determines the target tissue closest to the punching device according to the position of the punching device and the first three-dimensional model.

Step S22: Calculate the distance between the target tissue and the punching device. Specifically, the control unit calculates the distance between the target tissue and the punching device.

Step S23: Calculate the collision occurrence time. Specifically, the control unit calculates the collision occurrence time t according to the speed of the conical tip and the distance. In this embodiment, the speed of the conical tip is v ₁ and the distance is D, then the The collision occurrence time t satisfies: t=D/v ₁ .

Step S24: Determine whether the collision occurrence time is greater than a preset time threshold. Specifically, the control unit judges whether the time t is greater than the set time threshold t ₀ , if not, judges that the collision probability is high, and generates the collision prompt information, and if so, judges that the collision probability is small and does not generate The collision prompt information or other information is generated.

Further, the surgical robot system further includes a prompting device, and the prompting device is configured to communicate with the control unit, so as to receive the punching state information and give a prompt. The prompting device may have various options. For example, the prompting device may include a buzzer alarm, which prompts the collision prompt information through a buzzer alarm. The prompting device may further include a voice prompting device for broadcasting the collision prompting information and the punching progress information. The prompting device may further include a display device for displaying the position of the punching device, the speed of the punching device, the collision prompt information, the punching progress information, etc. through text and images. Furthermore, the control unit can control the tool arm 210 to stop moving in time before the collision, so as to stop the drilling operation, so that the operator can intervene to adjust the drilling direction manually.

In this embodiment, in the process of automatically performing the punching by the surgical robot system, the punching process is also monitored in real time in combination with the second image information collected by the endoscope 20, so that human intervention can be carried out at any time, and the punching process is further avoided. The hole manipulation causes damage to the target tissue.

In addition, in order to ensure that the tapered tip 11 of the punching device 10 is located within the field of view of the endoscope 20 when collecting the second image information, the control unit is further configured to use visual servoing The pose of the endoscope 20 is controlled so that the tapered tip 11 is always within the field of view of the endoscope 20 , preferably at the center of the field of view of the endoscope 20 .

Visual servo control is to compare the image information obtained by real-time measurement with the given image information, and use the obtained image error to feedback to form a closed-loop control, so that the controlled object is in a given orientation. In this embodiment, the given image information is that the tapered tip 11 is at the center of the field of view of the endoscope 20 . FIG. 19 is a schematic diagram showing the principle of the control unit performing visual servo control on the endoscope 20 . As shown in FIG. 19 , the control unit includes a visual servo controller 401 and an image arm joint controller 402 . A joint sensor 221 is provided on the joint of the imaging arm 220 . The process of visual servo control is as follows:

The endoscope 20 collects the second image information as actual image information, and sends it to the servo controller 401 .

The visual servo controller 401 judges, according to the actual image information, whether the tapered tip 11 that has entered the surgical object is within the field of view/center of the field of view of the endoscope 20, and if not, the visual servo control The controller 401 extracts the actual pose of the endoscope 20 according to the error between the actual image information and the given image information, and obtains the movement of the endoscope 20 from the actual pose to the given pose Movement information such as movement speed, movement direction, etc.

According to the motion information of the endoscope 20 and the inverse kinematics of the robot, the motion information of the joints of the image arm 220 is calculated and sent to the image arm joint controller 402 .

The image arm joint controller 402 drives relevant joint motions on the image arm 220 according to the calculated motion information of the joints of the image arm 220, and the joint sensor 221 feeds back the joint information in real time until the image The joint of the arm 220 drives the endoscope 20 to move to the given position.

During the movement process of the endoscope 20 (the movement process along the second path, the rotation process after reaching the target position, and the servo control process), according to the posture of the endoscope 20 , the The endoscope produces a bending motion. Here, the endoscope 20 used in this embodiment is a flexible endoscope, please refer back to FIG. 3 and FIG. 4 , the endoscope 20 includes an image acquisition element 21 and a mirror arm 22 , and the mirror arm 22 It includes a first rigid section 22a, a controllable bending section 22b and a second rigid section 22c sequentially connected from the proximal end to the distal end, wherein the controllable bending section 22b includes a corrugated pipe (as shown in FIG. 3 ), or The controllable bending section 22b includes a snake bone (as shown in FIG. 4 ). The image acquisition element 21 is disposed on the second rigid segment 22c, and the image acquisition element 21 may be a binocular vision camera. For the process of obtaining the feature points of the target tissue in the second target region from the three-image information, and establishing the three-dimensional model, reference may be made to the foregoing introduction. It should be noted that, when the feature points of the human body tissue are positioned on the coordinate system of the image acquisition element 21 of the endoscope 20, the coordinate system of the image acquisition element 21 is in the coordinate system of the control unit. The position under the system can be obtained according to the forward kinematics of the robot and the pre-calibrated coordinate system parameters of the image acquisition element 21 .

In addition, the endoscope 20 also includes components such as a pull cord (not shown in the figure), a light source (not shown in the figure), and the like, wherein the pull cord is arranged in the pull cord hole of the mirror arm 22 (the figure is not shown in the figure). (not shown), the bending or straightening of the controllable bending section 22b is realized by tightening or loosening the pulling rope, which is specifically set as a conventional technical means in the field, and will not be described in detail here. The light source is disposed on the second rigid section 22c for providing illumination for the image capturing element 21 .

Further, an embodiment of the present invention also provides a computer-readable storage medium on which a program is stored, and when the program is executed, all operations performed by the aforementioned control unit are performed.

Further, an embodiment of the present invention further provides an electronic device, where the electronic device includes a processor and the computer-readable storage medium, where the processor is configured to execute a program stored on the computer-readable storage medium.

Still further, an embodiment of the present invention also provides a path planning method, which is at least used to plan the first motion path of the punching device, that is, including the planning of the first motion path performed by the aforementioned control unit. step. Not only that, the path planning method also includes a step performed by the control unit to plan a second global path and a second local path of the image acquisition device.

Although the present invention is disclosed above, it is not limited thereto. Various modifications and variations can be made in the present invention by those skilled in the art without departing from the spirit and scope of the invention. Thus, provided that these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims (22)

1. A computer-readable storage medium on which a program is stored, characterized in that, when the program is executed, the following steps are performed:

   establishing a three-dimensional model according to the first image information in the surgical object;

   The first path of the punching device is planned according to the first hole position on the body surface of the surgical object, the predetermined position in the body of the surgical object, and the three-dimensional model, so that the punching device can be punched when the punching device moves along the first path. The punching end of the device passes through the body surface of the surgical subject at the first hole position and reaches the predetermined position.
2. The computer-readable storage medium of claim 1, wherein the first path comprises a first global path, and when the punching device moves along the first global path, the punching end is capable of arrive at the predetermined location.
3. The computer-readable storage medium of claim 2, wherein the program executes:

   When there is an obstacle on the first global path, a first partial path is planned, the first partial path is planned outside the boundary of the obstacle, and the starting point and the ending point of the first partial path are both on the first global path.
4. The computer-readable storage medium of claim 3, wherein the program performs the following steps to plan the first partial path:

   An artificial potential field is established according to the three-dimensional model and the predetermined position, and the first local path is planned according to the artificial potential field.
5. The computer-readable storage medium according to claim 4, wherein the artificial potential field has a position q, and the potential function of the position q in the artificial potential field is an attractive potential function U _att (q ) and the sum of the repulsive force potential function U _rep (q):

   U(q)= _Uatt (q)+ _Urep (q),

   

   

   In the formula, ζ is the attraction gain; d(q, q _goal ) is the distance between the position q and the predetermined position; D(q) is the distance from the nearest obstacle to the position q; η is Repulsive force gain; Q ^* is the force threshold of the obstacle, when the distance from the obstacle to the punching end is greater than Q ^* , the obstacle will not generate a repulsive force on the punching end.
6. The computer-readable storage medium according to claim 1, wherein when planning the first path, the program further executes the following steps:

   performing an expansion calculation on the three-dimensional model to expand the boundary of the tissue model in the three-dimensional model to the outside by a safe distance;

   The first path is planned according to the expanded three-dimensional model.
7. The computer-readable storage medium according to claim 6, wherein the maximum speed of the punching device when moving along the first path is V _max1 , the acceleration is a ₁ , the safety distance is d ₁ , and satisfy the following relationship:

   d ₁ =V _max1 ² /(2a ₁ ).
8. The computer-readable storage medium according to claim 1, wherein after the punching end penetrates the body surface of the surgical subject at the first hole position, the program further performs the following steps:

   The punching state information is acquired according to the second image information of the punching end and the three-dimensional model, and guidance information is generated.
9. The computer-readable storage medium according to claim 8, wherein an image acquisition device is used to penetrate the body surface from the second hole on the body surface of the surgical subject and enter the body of the surgical subject to obtain the second image information, the program is also used to perform the following steps:

   The target pose of the image acquisition device in the surgical object is planned according to the first hole position and the three-dimensional model, so that when the image acquisition device is in the target pose, the first hole position is located in the within the field of view of the image acquisition device;

   Plan the motion scheme of the image acquisition device according to the initial posture of the image acquisition device in the surgical object, the three-dimensional model and the target posture, and drive the image acquisition device to move according to the motion plan and reach the target pose.
10. The computer-readable storage medium according to claim 9, wherein the initial pose comprises an initial position, and the target pose comprises a target position; the motion scheme comprises: according to the initial position, the three-dimensional model and a second global path planned by the target position, when the image acquisition device moves along the second global path, the target position can be reached.
11. 11. The computer-readable storage medium of claim 10, wherein when there is an obstacle on the second global path, the motion scheme further comprises a second partial path, the second partial path is located in the outside the boundary of the obstacle, and both the start point and the end point of the second local path are on the second global path.
12. The computer-readable storage medium according to claim 10 or 11, wherein the target pose further comprises a target pose, and the motion scheme further comprises a current pose when the image acquisition device reaches the target position and the rotation scheme planned by the target posture, when the image acquisition device rotates at the target position according to the rotation scheme, the target posture can be reached.
13. The computer-readable storage medium according to claim 8 or 9, wherein the second image information is acquired by an image acquisition device, and when acquiring the second image information, the program further executes the following steps:

    Visual servoing is used to control the pose of the image capturing device, so that the punching end of the punching device is within the field of view of the image capturing device.
14. The computer-readable storage medium according to claim 8, wherein the punching state information comprises at least one of position information of the punching device, speed information of the punching device, and punching progress information By;

    The program performs at least one of the following steps to obtain the punch state information:

    Obtain the position information of the punching device according to the second image information and the three-dimensional model;

    Acquiring speed information of the punching device according to the position change of the punching device;

    The punching progress information is generated according to the current position information of the punching device and the predetermined position.
15. The computer-readable storage medium according to claim 14, wherein the guidance information includes collision reminder information; and the program executes the following steps to obtain the guidance information:

    The collision probability is acquired according to the position information of the punching end, the speed information of the punching end, and the three-dimensional model, and collision prompt information is generated.
16. The computer-readable storage medium of claim 15, wherein the program executes the following steps to obtain the collision reminder information:

    According to the position information of the punching end and the three-dimensional model, obtain the target tissue closest to the punching end, and calculate the distance between the punching end and the target tissue;

    Calculate the collision occurrence time according to the speed of the punching end and the distance;

    It is judged whether the collision occurrence time is greater than the set time threshold, and if not, it is judged that the collision probability is high, and the prompt information is generated.
17. The computer-readable storage medium of claim 1, wherein the program performs the following steps to determine the first hole position on the body surface of the surgical subject:

    establishing a first sign image model according to the first body surface information and the lesion information of the surgical object in the first state, where the first sign image model is used to plan the first pre-hole location;

    establishing a second physical sign image model according to the second body surface information of the surgical object in the second state;

    The second physical sign image model and the first physical sign image model are registered to obtain a first target hole position corresponding to the first pre-hole position on the second physical sign image model, and the The first target hole position is used to be guided to the body surface of the surgical object to obtain the first hole position.
18. An electronic device comprising a processor and a computer-readable storage medium according to any one of claims 1-17, the processor being configured to execute a program stored on the computer-readable storage medium.
19. A surgical robot system, comprising:

    The tool arm is used to connect a punching device, the punching device includes a punching end, and the punching end is used to penetrate the body surface of the surgical object from the first hole position on the body surface of the surgical object and reach the hole in the body of the surgical object. predetermined location;

    an image arm for connecting to an image acquisition device, the image acquisition device for acquiring first image information inside the surgical subject; and,

    A control unit, connected in communication with the tool arm, the image arm and the image acquisition device, and configured to implement the steps performed by the program of any one of claims 1-17.
20. The surgical robot system according to claim 19, wherein after the punching end penetrates the body surface of the surgical object at the first hole position, the image acquisition device further captures the punching end the second image information; the control unit is further configured to obtain punching state information and generate guide information according to the second image information and the three-dimensional model;

    The surgical robot system further includes a prompting device, connected in communication with the control unit, and used for receiving and displaying the punching state information and the guidance information.
21. The surgical robot system according to claim 19, wherein the surgical robot system comprises a first imaging device and a second imaging device, and both the first imaging device and the second imaging device are connected to the control unit Communication connection, the first imaging device is used for acquiring first body surface information and lesion information of the surgical object in the first state, and the second imaging device is used for acquiring the second body surface of the surgical object in the second state information; the control unit establishes a first sign image model according to the first body surface information and the lesion information, and establishes a second sign image model according to the second body surface information, the first sign image model and The second physical sign image model is used to obtain the first hole position.
22. The surgical robot system according to claim 21, wherein the first imaging device comprises any one of MRI, X-ray device or B-ultrasound; the second imaging device comprises a binocular vision camera or structured light camera.

Images