Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the term “and/or” includes all combinations of one or more of the associated listed items.

Referring to FIG. 1 to FIG. 4, schematic structure diagrams of a surgical robot according to an embodiment and partial schematic diagrams thereof are illustrated.

A surgical robot includes a slave operating device 100 and a master control console 200 controlling the slave operating device 100.

The slave operating device 100 includes a main arm 110, an adjusting arm 120, and a manipulator 130, which are sequentially connected. The number of the adjusting arm 120 may be two or more, for example, may be four, and the number of the manipulator 130 may be two or more, for example, may be four. The distal end of the main arm 110 includes an orienting platform, the proximal end of each of the adjusting arms 120 is connected to the orienting platform, and the proximal end of each of the manipulators 130 is connected to the distal end of each of the adjusting arms 120. The manipulator 130 may be configured to be detachably connected to an operating arm 150, the manipulator 130 includes a plurality of joint assemblies. The operating arm 150 includes an actuating device 160, a link 170, a connecting assembly 180, and an end effector 190, which are sequentially connected. The joint assemblies may include the link 170 and the connecting assembly 180, or may include only the connecting assembly 180. The link 170 may be a hollow structure for actuating cables to pass through. The actuating device 160 may actuate the end effector 190 to move by manipulating the joint assemblies with the actuating cables. The end effector 190 includes an image end effector 190 and an operation end effector 190.

During a surgical process, the manipulator 130 may be positioned outside the patient's body, the distal end of the operating arm 150 may insert into the patient's body with a puncture device 300 mounted to the manipulator 130. The doctor may control the manipulator 130 and the operating arm 150 to move coordinatively so as to adjust the pose of the end effector 190. The joint assemblies of the manipulator 130 may be mainly configured for positioning, the joint assemblies of the operating arm 150 may be mainly configured for orientation. The manipulator 130 may adjust the position and the pose of the end effector 190 by manipulating the operating arm 150 to move.

The master control console 200 includes a handle 210 and a display. The doctor may send control instructions to the slave operating device 100 by operating the handle 210, so as to cause the slave operating device 100 to perform operations in response to the control instructions, and the doctor may observe the surgical site captured by the image end effector 190 on the display.

Referring to FIG. 5, the manipulator 130 includes two rotary joint assemblies and one prismatic joint assembly. The second rotary joint assembly from the rotary joint assemblies may actuate a parallelogram-based remote center of motion mechanism to transfer a rotation to the remote center, so as to realize the axes of three DoFs (degrees of freedom) to intersect at a point (i.e., the remote center) in space. The prismatic joint assembly may be located at the distal end of the manipulator 130. The prismatic joint assembly includes an actuating device 140, and the actuating device 140 of the prismatic joint assembly may engage with the actuating device 160 of the operating arm 150, so as to allow the transfer of the actuating force.

In some embodiments, please refer to FIGS. 6 and 7 in combination, an additional rotary joint assembly may be arranged at the proximal end of the first rotary joint assembly, so as to increase the range of motion of the manipulator 130. The axis of the rotary joint assembly may intersect at the remote center but not coincident with the axis of the first rotary joint assembly, so as to ensure the distal end of the manipulator 130 may still rotate around the remote center.

The operating arm 150 includes three rotary joint assemblies, for example, the link 170 may roll with respect to the actuating device 160, and the connecting assembly 180 may yaw and pitch with respect to the link 170. For another example, the link 170 may be fixed with respect to the actuating device 160, and roll, yaw and pitch with respect to the link 170 through the connecting assembly 180. Thereby the three basic DoFs of a wrist, i.e., the yaw, the pitch, and the roll, can be realized. The three DoFs of a wrist may be mainly configured for orientation, since the three DoFs have little perturbation to the position. The connecting assembly 180 may further have an open/close-related DoF to actuate the end effector 190 to open and close.

The manipulator 130 and the operating arm 150 thereon may move coordinately, that is, they may move with six DoFs (three position-related DoFs and three pose-related DoFs) to satisfy the needs of the surgery.

The surgical robot may further include a controller. The controller may be integrated in the master control console 200 or integrated in the slave operating device 100. Or, the controller may be independent from the master control console 200 and the slave operating device 100, for example, the controller may be deployed locally, or deployed in the cloud. The controller may include at least one processor.

In some embodiments, each of the joint assemblies of the manipulator and the operating arm of the slave operating device 200 includes a sensor configured to sense the state of the joints. The sensors include angle sensors and displacement sensors, the angle sensors are configured to sense the rotational motion of the joint assemblies, and the displacement sensors are configured to sense the linear motion of the joint assemblies. Specifically, appropriate sensors can be configured according to the type of the joint assemblies. The controller may couple to the sensors and to the display of the master control console.

In some embodiments, a graphical display method of the surgical robot is provided, which can be executed by the controller mentioned above. Referring to FIG. 8, the graphical display method includes the following steps:

Step S1, detecting the presence of a first manipulator from a plurality of manipulators which is in an operational mode.

The manipulator being in the operational mode may be simply understood as that the manipulator is provided with an operating arm, and more precisely, be understood as that the manipulator is provided with the operating arm and the operating arm is inserted into the patient's body by the guiding and the sealing of a puncture device 300. Generally, the operating arm mounted to the manipulator may be inserted into the patient's body through the puncture device 300, and a manipulator without an operating arm may be located at a relatively distant position when the doctor arranges the operation, so that the manipulator being in the operational mode can be simply regarded as the manipulator being provided with the operating arm. Opposite to the concept of the operational mode, there is an idle mode for the manipulator, and the manipulator without the operating arm can be regarded as that the manipulator is in the idle mode. Generally, the manipulator in the idle mode may be in a static state. Herein, the first manipulator generally refers to a type of manipulator, and should not be understood as that it only refers to a specific manipulator, and it may represent at least one manipulator.

A detection device can be provided to each of the manipulator, which is configured to detect whether an operating arm is mounted thereon. For example, the detection device may include a proximity sensor, a pressure sensor or a photoelectric sensor, etc. In other embodiments, whether the manipulator is in the operational mode may be determined by the manual input from the doctor, in this way, the manipulator can be set to be in the operational mode even if the manipulator is not provided with an operating arm.

Also, a memory can be provided to the operating arm to store the basic information of the operating arm, such as the link parameters and the types of e the operating arm, etc. And another detection device can be provided to the manipulator to read the basic information of the operating arm.

Proceed to step S2 if the presence of the first manipulator is detected in Step S1. Otherwise, continue with Step S1.

Step S2, obtaining the state information of the joint assemblies sensed by the sensors of the first manipulator.

The sensed state information of the joint assemblies includes angle information and/or displacement information.

Step S3, obtaining a kinematic model of the first manipulator.

For example, the kinematic model can be pre-built. For another example, the kinematic model can be built in real time, such as by combining the state information of the first manipulator and the link parameters and using forward kinematics to obtain the kinematic model in the reference frame For example, the reference frame may be the base coordinate system of the proximal end of the main arm, or may be the tool coordinate system of the distal end of the main arm (i.e., the tool coordinate system of the orienting platform).

Step S4, obtaining configuration parameters of a virtual camera.

As its name implies, the virtual camera may be a camera that does not actual exist. The virtual camera does not actually capture the images of the objects, and only represents a concept about at least one viewing point. Even so, it is still possible to set the parameters of the virtual camera.

It is possible to configure parameters of the virtual camera. The parameters (i.e., the configuration parameters) of the virtual camera may at least include (virtual) poses, camera parameters corresponding to those of an actual camera such as the focal length and/or the aperture, the virtual camera parameters may also include the virtual focal length and/or the virtual aperture. Generally, the (virtual) focal length may correspondingly adjust the field angle of the (virtual) camera, and the (virtual) aperture may correspondingly adjust the depth of the field. In some embodiments, it is also possible to describe the virtual camera parameters including the field angle and the depth of the field. With respect to the virtual camera, the field angle and/or the depth of the field are virtual. Even if the focal length and the aperture of the camera are virtual, it is also possible to combine, for example the same imaging principle of the actual camera, to realize the purpose of the present disclosure. Different virtual camera parameters can realize different imaging effects.

The virtual camera parameters can be stored in the system configuration files of the memory of the surgical robot, and obtained by reading the system configuration files. The virtual camera parameters may also be manually set by the doctor before the operation process or during the operation process as needed through an input device coupled to the controller. such setting is on demand, for example, the virtual camera parameters can be obtained by inputting relative data(s) into a text control, and for example, the virtual camera parameters can be obtained by selecting from a tab control.

The pose of the virtual camera can be the same as the pose of the actual camera (i.e., the image end effector), so as to observe the operating arm from the same viewing point as the actual camera. The pose of the virtual camera also can be different from the pose of the actual camera, so as to observe the operating arm from a different viewing point from the actual camera. Generally, it is possible to select the pose of the virtual camera different from that of the pose the actual camera to observe, this may help obtain more comprehensive information of the operating arm. For example, the operating arm also can be a camera arm, and observations can be made through the virtual camera.

Step S5, generating an image model of the first manipulator from the viewing point of the virtual camera by combining the configuration parameters of the virtual camera, the kinematic model of the first manipulator and the state information of each joint assembly of the first manipulator.

For example, the image model of the first manipulator can be a computer model substantially embodying the structural features of the first manipulator. And for another example, the image model of the first manipulator can be a projective model substantially embodying the feature point projections of the first manipulator. Each of the two image models of the first manipulator can display the image model from the viewing point of the virtual camera, each of the two image models can present an effect of a plane graph and can follow the movement of the first manipulator to present a dynamic effect.

Step S6, displaying the image model of the first manipulator in a first display window of the display.

Since the display also displays the actual image of the lesion captured by the image end effector, the first display window can be arranged at an edge area, such as the bottom, the sides, and/or the top, of the display in order not to affect the operation However, it is also possible to arrange the first display window at the center area of the display if necessary. The image model of the first manipulator displayed on the first display window may be generated as a perspective image model in order to not prevent the doctor from observing the actual image of the focus.

By observing the first manipulator positioned outside the patient's body through the virtual camera, generating the image model of the first manipulator from the viewing point of the virtual camera, and displaying the image model of the first manipulator in the first display window of the display, the above steps S1-S6 in one aspect can help the doctor to observe the pose of the first manipulator to facilitate performing surgery, for example, it can avoid the collisions between the first manipulators, in another aspect, the steps save the cost required for setting an actual camera, in a further aspect, the steps eliminate the interference of the unnecessary images caused by the actual camera, in a further aspect, the steps realize the observation of the first manipulator having the operating arm with the image end effector.

In some embodiments, referring to FIG. 9, the surgical robot may include the manipulators 130a, 130b, 130c, and 130d. Each of the manipulators 130b, 130c, and 130d may be provided with an operating arm and may be regarded as a first manipulator. Since the manipulator 130a is not provided with an operating arm and may be regarded as a second manipulator. According to the method of the steps S1-S6, image models of the manipulators 130b, 130c, and 130d are shown in the FIG. 10, each of the models is a computer model corresponding to the first manipulator. In fact, when generating the computer model, it can be generated to substantially represent the structural features of the corresponding manipulator, in other words, the computer model can be simplified to some extent. For example, a simplified computer model basically as shown in FIG. 11 can be generated corresponding to any of the manipulators when no concern is made on the poses and the joint states of a certain manipulator in the reference frame.

In some embodiments, the first manipulators in the first display window can be numbered based on the numbers, such as the numbers 1-4, of the manipulators 130a-130d in the system configuration files, the identifiers. In the embodiment shown in FIG. 9, also referring to FIG. 10, the image models of the manipulators 130b, 130c, and 130d can be numbered, the one corresponding to the manipulator 130b can be numbered 1, the one corresponding to the manipulator 130c can be numbered 2, the one corresponding to the manipulator 130d can be numbered 3. Each number can be displayed at the bottom of the image model of the first manipulator, or displayed at other associated locations that can help the doctor to know the relationship between the external actual manipulator and the image model of the first manipulator displayed on the display, for example, displayed at the proximal end of image model of the manipulator shown in FIG. 10.

In some embodiments, referring to FIG. 12, the graphical display method also may include the following steps:

Step S711, detecting the presence of a second manipulator from a plurality of manipulators, which is in an idle mode.

Proceed to step S712 if the presence of the second manipulator is detected in step S711. Otherwise, continue with step S711.

Step S712, obtaining the state information of the joint assemblies sensed by the sensors of the second manipulator.

Step S713, obtaining a kinematic model of the second manipulator based on the state information of the second manipulator.

The kinematic model of the second manipulator and the kinematic model of the first manipulator can be in a same reference frame directly or through transformation.

Step S714, generating an image model of the second manipulator from the viewing point of the virtual camera by combining the configuration parameters of the virtual camera and the kinematic model of the second manipulator.

Step S715, displaying the image model of the second manipulator in the first display window of the display, or displaying the image model of the first manipulator and the image model of the second manipulator in a second display window of the display.

In step S715, referring to FIG. 13 and FIG. 14, the image model of the second manipulator is displayed in the first display window, that is, the image models of the first manipulator and of the second manipulator are simultaneously displayed in the first display window.

In step S715, referring to FIG. 15, the image model of the second manipulator is displayed in the second display window, and the image models of the first manipulator and of the second manipulator are simultaneously displayed in the second display window. Both of the two methods can show the doctor the poses between all manipulators. The sizes of the first display window and the second display window may be the same or different. For example, the size of the first display window may be made larger to serve as a main auxiliary display window and the second display window serves as a secondary auxiliary display window. For example, the contents displayed in the first display window and in the second display window can be exchanged according to an operating instruction. and the contents displayed in each display window can be scaled proportionally when the sizes of the first display window and the second display window are different.

Generally, the above steps S711-S715 should be executed when the presence of the first manipulator is detected in step S1, since all the manipulators may be regarded as the second manipulators if there is no first manipulator. And since these second manipulators each may be in the idle mode, all of them may be positioned outside the patient's body and kept static, so that it is unnecessarily to display them graphically. The order of steps S1-S6 and steps S711-S715 is not particularly limited herein.

In some embodiments, also referring to FIG. 13-FIG. 15, the image model of the first manipulator and the image model of the second manipulator can be identified in the first display window or the second display window, so as to distinguish the operational modes of the first manipulator and the second manipulator. For example, the colors of the image models of the first manipulator and the second manipulator may be different. For another example, as shown in FIG. 13, different line types may be used to distinguish the image models of the first manipulator and the second manipulator in the first display window, for example, the image model of the first manipulator may be presented in solid lines, and the image model of the second manipulator may be in dotted lines. For another example, as shown in FIG. 14, a box, such as a dashed box, may be put outside of the image model of the second manipulator in the first display window to identify the second manipulator. For another example, as shown in FIG. 15, the image models of the first manipulator and the second manipulator can be identified in the second display window, for instance, the image model of the first manipulator may be presented in solid lines, and the image model of the second manipulator may be in dotted lines. For another example, the image models of the first manipulator and the second manipulator can be identified with numbers in the first display window or in the second display window, for instance the number of the image model of the first manipulator may be presented in solid lines, and the number of the image model of the second manipulator may be in dotted lines.

By providing the image model of the second manipulator in the first display window or in the second display window, the above steps S711-S715 can help the doctor to observe the poses of all the manipulators entirely or comprehensively. On an aspect, these steps can help to avoid the collisions between any two manipulators by knowing the poses of all the manipulators.

In some embodiments, the step S714 may be executed after determining the possibility of collisions between the first manipulator and the second manipulator. This is because the doctor does not have to or need to observe the image model of the second manipulator when it is determined that there is no possibility of collisions between the first manipulator and the second manipulator. Therefore, the first display window can be more concise and no useless information will be introduced to disturb the doctor's observation, by displaying the image model of the first manipulator and not displaying the image model of the second manipulator.

For example, the possibility of collisions between the first manipulator and the second manipulator can be determined by such a detecting method as shown in FIG. 16, the method includes:

Step S7131, calculating a reachable workspace of the first manipulator.

For example, the reachable workspace can be calculated by combining the link parameters of the first manipulator, the range of motion of each joint assembly of the first manipulator and using the forward kinematics.

Step S7132, determining whether the reachable workspace of the first manipulator covers at least part of the second manipulator.

In Step S7132, determining whether the reachable workspace of the first manipulator covers at least part of the second manipulator can be realized by: building a geometric model by combining the kinematic model and the structural features of the second manipulator, obtaining a point set of external information by discretizing the geometric model of the second manipulator, and further determining whether the reachable workspace of the first manipulator covers at least part of the point set of external information of the second manipulator (for example by calculating the intersection).

Proceed to step S714 if the reachable workspace of the first manipulator covers at least part of the second manipulator, i.e., the possibility of collisions between the first manipulator and the second manipulator is determined in step S7132. Otherwise, continue with step S7131, as the origin of the first manipulator may be changed subsequently and there may be possibility of collisions between the first manipulator and the second manipulator at other times.

In some embodiments, it is also possible to generate an image model of the reachable workspace of the first manipulator from the viewing point of the virtual camera by combining the configuration parameters of the virtual camera and the reachable workspace of the first manipulator in step S7131, i.e., on the basis of the calculation of the reachable workspace of the first manipulator, and then to display the image model of the reachable workspace in the first display window or in the second display window. For example, the image model of the reachable workspace may be a projective model or a computer model of the reachable workspace. From the viewing point of the virtual camera, the image model of the reachable workspace may be presented like a closed figure.

For example, the origin of the image model of the reachable workspace may be coincident with the motion center of the first manipulator. For another example, the origin of the image model of the reachable workspace may be on the axis of the joint assembly at the proximal end of the first manipulator. By displaying the image model of the reachable workspace in the first display window or in the second display window, an indication or a reference on the range of motion of the first manipulator can be provided to the doctor, which may help to avoid the collisions between the first manipulators, and/or between the first manipulator and the second manipulator.

In some embodiments, at least one level of safe motion space can be set in the reachable workspace of the first manipulator, more particularly, the safe motion space can be defined based on the range of motion of each joint assembly of the first manipulator. An image model of the safe motion space from the viewing point of the virtual camera can be generated by combining the configuration parameters of the virtual camera and the safe motion space of the first manipulator, and the image model of the safe motion space can be displayed in the first display window or in the second display window. By displaying the image model of the safe motion space in the first display window or in the second display window, an indication or a reference of a recommended range of motion of the first manipulator can be provided to the doctor more clearly.

In addition, when the image model of the first manipulator moves over the image model of the safe motion space to the image model of the reachable workspace, the image model of the first manipulator, the image model of the safe motion space and/or the image model of the reachable space can be identified in the first display window or in the second display window to remind the doctor to pay attention. For example, reminders can be realized by make at least one of the above image models to flash or to change color.

In case of two or above levels of safe motion spaces, the image model of the first manipulator can be identified differently, such as in different colors or line types, and etc., when the image model of the first manipulator moves to different safe motion space in response to the movement of the first manipulator.

In some embodiments, any image model of the manipulators can be hidden or displayed in the first display window or in the second display window according to an instruction. For example, the image model of the first manipulator can be hidden or displayed. For example, the image model of the second manipulator can be hidden or displayed. For example, any joint assembly in the image model of the first manipulator or the second manipulator can be hidden or displayed. For another example, at least one of the links, the joint assemblies, and the end effector in the image model of the operating arm can be hidden or displayed. In one aspect, the above enables the observation on the poses of the manipulator, which would be otherwise difficult to be observed, by hiding the image model of a corresponding manipulator, when some poses of all the displayed image models of the manipulators overlap with each other from the viewing point of the current virtual camera and affect the doctor's observation on the manipulators. In another aspect, the above is helpful to provide a concise auxiliary view to the doctor.

In some embodiments, the graphical display method mentioned above may further include:

Performing identification on at least part of an image model of a third manipulator in the manipulators in the first display window or in the second display window when the third manipulator reaches a threshold of an event.

The third manipulator may generally refer to a type of manipulator, and should not be understood as only referring to a specific manipulator. The third manipulator may be from the first manipulator and/or the second manipulator mentioned above. The threshold may be a warning threshold, the event may be a situation to be avoided.

In some specific embodiments, the warning threshold is based on the distance between the third manipulator and a fourth manipulator from the manipulators, for example, the warning threshold can be a value. The situation to be avoided can be the collision between the third manipulator and the fourth manipulator, for example, the situation to be avoided can be a value. The fourth manipulator may generally refer to a type of manipulator, and should not be understood as only referring to a specific manipulator. The fourth manipulator may be from the first manipulator and/or the second manipulator. For example, as shown in FIG. 17, the method can be realized with the following steps:

Step S721, obtaining the minimum distance between the third manipulator and the fourth manipulator.

The step S721 may be executed in real time.

Step S722, determining the relationship between the minimum distance and the warning threshold and the situation to be avoided.

Each of the warning threshold and the situation to be avoided may be represented with a value. When the situation to be avoided is a collision between the third manipulator and the fourth manipulator, the value d_limrepresenting the warning threshold should be larger than the value d_minrepresenting the situation to be avoided, that is d_lim>d_min, where the value d represents the minimum distance between the third manipulator and the fourth manipulator. In some embodiments, d_min=0 represents that the collision has occurred.

Proceed to step S721 if d>d_limin step S722, that is, the minimum distance does not reach the warning threshold. Proceed to step S723 if d_min<d≤d_lim, that is, the minimum distance has reached the warning threshold but not reached to the situation to be avoided. Proceed to step S724 if d=d_min, that is, the minimum distance has exceeded the warning threshold and reached to the situation to be avoided.

Step S723, performing a first identification on the minimum distance points on the image models of the third manipulator and the fourth manipulator in the first display window or in the second display window.

As shown in FIG. 18, the minimum distance between the manipulator 130c and the manipulator 130d has reached the warning threshold, then in step S723, the minimum distance points P1, P2 in the image models of the third manipulator and the fourth manipulator may be identified with color or a graphic frame such as a circle, as shown in FIG. 19. When a redetected minimum distance does not reach the warning threshold, the identifiers on the minimum distance points in the image models of the third manipulator and the fourth manipulator may be eliminated from the first display window or the second display window, as shown in FIG. 10. Proceed to step S724, i.e., perform a second identification if the redetected minimum distance has reached the situation to be avoided.

In addition, in the process of performing the first identification, i.e., when the condition d is satisfied, the first identifier may be changed with the gradually decreasing or increasing minimum distance. For example, the color may be changed gradually, to be different from the color when d=d_lim. For example, the first identifier may be made to flash, but differently from when d=d_min.

Step S724, performing a second identification on the minimum distance points on the image models of the third manipulator and the fourth manipulator in the first display window or in the second display window.

The first identification may be different from the second identification. In step S724, for example, the identifier of the minimum distance points P1, P2 in the image models of the third manipulator and the fourth manipulator may be enhanced, for instance the color be deepened. For example, the identifier of the minimum distance points P1, P2 in the image models of the third manipulator and the fourth manipulator may flash. For example, the type of the identifier of the minimum distance points P1, P2 in the image models of the third manipulator and the fourth manipulator may be changed, for instance the type of the graphic frame be changed, as shown in FIG. 20. The solid circle shown in FIG. 19 is replaced by the dotted circle shown in FIG. 20. Proceed to step S723, i.e., perform a first identification when the redetected minimum distance has reached the warning threshold but not reached the situation to be avoided.

Steps S721-S724 can help the doctor to know the collision position between the manipulators.

More particularly, as shown in FIG. 21, the step S721 mentioned above may be realized with the following steps:

Step S7211, building the corresponding geometric models of the third manipulator and of the fourth manipulator based on the respective kinematic models and the structural features of the third manipulator and of the fourth manipulator.

In step S7211, it is generally possible to use a slightly lager basic geometry to replace the actual model to perform the interference analysis, so as to improve the detection efficiency. The respective geometric models of the third manipulator and the fourth manipulator can be simplified to be a sphere, a cylinder, a cuboid, a convex polyhedron, or a combination of two or more thereof.

Step S7212, obtaining the respective point sets of external information of the third manipulator and the fourth manipulator in the reference frame by discretizing the respective geometric models of the third manipulator and the third manipulator.

In step S7212, the point sets of external information of the third manipulator and the fourth manipulator can be obtained by performing datafication to the respective geometric models of the third manipulator and the fourth manipulator.

Step S7213, determining the minimum distance between the third manipulator and the fourth manipulator based on the respective point sets of external information of the third manipulator and the fourth manipulator.

In step S7213, the minimum distance between the third manipulator and the fourth manipulator can be determined by using a distance tracking method. More particularly, the minimum distance between the third manipulator and the fourth manipulator can be determined from the respective point sets of external information of the third manipulator and the fourth manipulator by using a traversal algorithm.

More particularly, as shown in FIG. 22, the step S723 mentioned above maybe realized with the following steps:

Step S7231, determining the minimum distance points in the image models of the third manipulator and the fourth manipulator, which corresponds to the minimum distance between the third manipulator and the fourth manipulator.

Step S7232, performing a first identification on the minimum distance points on the image models of the third manipulator and the fourth manipulator.

In some embodiments, as shown in FIG. 23, when the minimum distance reaches the warning threshold, the graphical display method may further include the following steps:

Step S7233, determining a direction of collision in the reference frame based on the positions of the minimum distance points on the image models of the third manipulator and the fourth manipulator.

Step S7234, performing identification on the direction of the collision between the third manipulator and the fourth manipulator in the first display window or in the second display window.