Shared Control for Bimanual Telesurgery with Optimized Robotic Partner

07/12/2021 ∙ by Ziwei Wang, et al. ∙ Imperial College London 0

Traditional telesurgery relies on the surgeon's full control of the robot on the patient's side, which tends to increase surgeon fatigue and may reduce the efficiency of the operation. This paper introduces a Robotic Partner (RP) to facilitate intuitive bimanual telesurgery, aiming at reducing the surgeon workload and enhancing surgeon-assisted capability. An interval type-2 polynomial fuzzy-model-based learning algorithm is employed to extract expert domain knowledge from surgeons and reflect environmental interaction information. Based on this, a bimanual shared control is developed to interact with the other robot teleoperated by the surgeon, understanding their control and providing assistance. As prior information of the environment model is not required, it reduces reliance on force sensors in control design. Experimental results on the DaVinci Surgical System show that the RP could assist peg-transfer tasks and reduce the surgeon's workload by 51% in force-sensor-free scenarios.



There are no comments yet.


page 2

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Robotic technology and its integration with surgical procedures have largely improved the operation intuitiveness for the surgeons [10]. Most of the current surgical robotic systems are conducted in master-slave mode, in which the surgeon could comfortably sit at a master console, remotely controlling the slave robotic arm and holding surgical tools through master interfaces and high-definition camera view [5, 13]. These teleoperation systems tend to simplify the surgeon operation to some extent. However, the surgeon’s performance would suffer from fatigue during long time operation. Also, the surgeon’s performance is limited by delayed visual and tactile information from the interaction environment due to the communication channel. Therefore, sharing the surgical robot with a degree of autonomy could facilitate the telesurgery, easing both physical and mental workloads for surgeons at the master console.

To this end, the slave surgical robot needs to perceive interaction forces in terms of telepresence. Nevertheless, it is challenging to obtain interactive forces in contact with patient tissue directly due to the mounting position of the force sensors conflicting with the surgical tool. Some studies [17, 16] integrated the environment information into the control design by force sensors or model calculation to enable the robot to conduct force interaction with environment automatically. These methods require accurate model structures of remote interactive dynamics or force sensors on the end-effector of the robot. It is revealed in [15] that accurate dynamics model is difficult to build in practical teleoperation subject to uncertainties, which may limit the merits of the above methods in telesurgery.

Another critical issue to realise the target is how the surgical robot can predict the human intention and give assistance, thus reducing the workload. The current methods mostly focus on training the robot to carry out repetitive tasks [4, 7], which lack the control flexibility to reflect on different scenarios. To assist the surgeon better, the surgical robot is required to adapt to the cooperated surgeon, react to the interacted environment and even improve the performance of the surgeon through sensory augmentation [12].

Based on this idea, we propose an intelligent Robotic Partner (RP) in bimanual shared control scheme to solve the two issues mentioned above. The RP is trained by interval type-2 polynomial fuzzy-model-based learning (IT2 PFMBL) algorithm according to human-human demonstration and robot-environment interaction, which enables the learning speed with few-shot data through virtue of the fitting ability of polynomials and the nonlinear expression ability of fuzzy models. The RP could then estimate interaction force without the exact physical model nor force sensors. Moreover, the proposed RP provides intuitive assistance to perform a cooperation task (e.g. peg-transfer) with human operator.

Ii Method

IT2 PFMBL algorithm is designed to estimate the human motions (i.e., movement trajectories and gesture actions) and interactive force with environment simultaneously based on the human-robot and robot-environment interaction data.

Let collected input-output pair data be , where stands for human movement trajectories, gesture actions or interactive forces with environment, respectively. are position, velocity and the associated output in Cartesian space at the th sampling. is the number of sampling pairs. Fuzzy C-Means clustering [1] and subtractive clustering [3] are utilised to obtain the number of fuzzy sets , the centre of each fuzzy set and the corresponding membership function , where , and is the premise variable. The membership function describes the degree to which the input data matches the fuzzy set.

With the help of the obtained fuzzy sets, the inner relationship between the input and output can be represented by the following polynomial discrete-time fuzzy logic system


where is the sampling time; is a polynomial fuzzy set of Rule corresponding to the premise variable , ; , , and are the polynomial system matrices, where each element is a polynomial in . For example, in the case of , the first three system matrices donate the equivalent inertia, damping and stiffness ones while the last the bounded exogenous force concerned with the position of the contact point. Robust linear least-square approach is employed in each Rule to identify the polynomial coefficients of the associated system matrices.

Considering the measurement noise in and the system uncertainty, the firing strength of the Rule is of the following interval set


where , , in which and denote the lower and upper membership function such that ; and are the lower and upper grade of membership, respectively, satisfying . Hence, the data-driven IT2 PFMBL system is described by




with , such that .

Remark 1. A unified data-driven learning paradigm is developed without the prior information (i.e., deterministic model structure) in [11, 9]. For , represents the human motion trajectories in Cartesian space, gripper angles and interactive forces with the environment, leading to different system matrices for different tasks.

Iii Experimental validation and results

In this section, we have verified the proposed method by the following experiments, which provides implementation foundation for the RP.

Iii-a Prediction of robot-environment interaction force

Fig. 1: H-Man planar robot-environment interaction platform with robotic finger.

We firstly apply the proposed IT2 PFMBL method to predict the interaction force in a planar robot-environment interaction platform using modified H-Man system [2], which is shown in Fig. 1. The environment is selected as hard silicone having similar mechanical characteristics with the organ tissues. The robotic finger equipped with an ATI 6-axis force/torque sensor will press the silicon at three different levels of depth: shallow, medium and deep, and scan the whole environment surface to precept the overall material properties. Interaction force is collected during 50 interactive trials for each type of pressing, totalling 150 trials for each sampling point. The proposed IT2 PFMBL method is advantageous at producing high-prediction accuracy with a small training set by introducing a type-2 fuzzy rule. We randomly allocate available data into 10/90 proportions of training and testing sets. Fig. 2a shows that the root-mean-square-error (RMSE) of the IT2 PFMBL method decreases rapidly with few-shot training data.

Fig. 2: (a) RMSE of interactive force estimation using IT2 PFMBL algorithm. (b) Performance comparison in terms of SRMSE and SAME (The smaller these two indicators are, the better).
Fig. 3: Experimental setup. (a) Master interfaces: dual Omega 7 devices with visual feedback. (b) Slave robot: surgical robotic manipulators. (c) Task: peg transfer.

To validate the merits of the IT2 PFMBL method, we compare its performances with other three common interaction force models, namely linear Kelvin-Voigt (LKV) model [8], T-S FMB [14] and PFMB model. Due to the improved non-linear representation of material stiffness properties, Fig. 2b indicates the force estimation of IT2 PFMBL surpasses the above models on the sum of RMSE (SRMSE) and mean absolute error (SMAE) in testing trials.

Iii-B Human-Robot shared control using IT2 PFMBL

To further assess the performance of the IT2 PFMBL method, we implemented it in a surgical robotic teleoperation system to train the RP to cooperate with a human operator. The setup of the surgical robotic system is shown in Fig. 3. The master console (Fig. 3a) consists of two hand haptic interfaces (Force Dimension Inc., Switzerland) and a monitor providing visual feedback of the patient side. Two patient-side manipulators of Da Vinci robotic system (Intuitive Surgical, USA) are used as slave robotic arms as shown in Fig. 3b.

The end-effectors of the robotic arms are connected to surgical grippers to perform a peg-transfer task (Fig. 3c), which is a standard laparoscopic training task from Fundamentals of Laparoscopic Surgery (FLS) [6]. The left gripper is controlled by a human operator and the right gripper by a RP. During the task, the human operator navigates the left gripper to the initial peg with an orange block, grasps the block and moves it to conduct a handover to the RP controlled gripper. After the handover, the RP moves the block to the target peg (Fig. 3c) (see supplementary video 1). The manipulation data of two operators were used as the input to the IT2 PFMBL method, resulting in a final interaction environment model including the motion trajectories, and gripper movements. After training, the RP could imitate the human partner and cooperate intelligently with another human operator.

Fig. 4: Experimental results from (a-b) human-human training to (c-d) human-robot testing. Motion learning from human operator and human partner:(a) Cartesian trajectories in human-human telesurgery (x, y and z denote the right hand movement along the associated direction; the dashed lines are the estimate with respect to the associated direction using IT2 PFMBL). (b) Gripper grasping : the dashed line stands for the estimate of gripper movement. and represent the angle at which the left and right hand grippers open. Motion learning from human operator and RP: (c) Cartesian trajectories in human-robot telesurgery (x, y and z denote the RP movement along the associated direction). (d) Assistive gripper grasping: is the angle of RP gripper.

Fig. 4 a&b show the comparison of motion trajectories between human-human manipulation and our training results from IT2 PFMBL method. It indicates that the derived models could effectively represent the human partner’s behavior with minimal errors. We implemented these models in the surgical robotic system to validate their performance. The results (Fig. 4c,d) show that the trained RP could successfully complete the transfer task with closely cooperating with a human partner (see supplementary video 2). The task completion of human operator with RP is within 27 seconds.

Iv Discussion and Conclusion

An innovative Robotic Partner concept is proposed in shared control for bimanual telesurgery to reduce the workload of the surgeon. A novel interval type-2 polynomial fuzzy model-based method is designed to realize precise estimation of contact force and few-shot training for intelligent surgeon assistance. The experimental results illustrate that the proposed IT2 PFMBL method achieves superior nonlinear representation capability and fitting performance in terms of SRMSE and SAME. Compared to T-S FMB method, PFMBL method can better characterise the nonlinearities of interaction force because its model parameters draw on the ability to fit polynomials. Besides, IT2 fuzzy sets are effective to capture the noise and system uncertainty by using the upper and lower bound of membership functions, thereby demonstrating satisfactory estimation robustness. The IT2 PFMBL is therefore able to estimate force accurately and provide haptic feedback to telesurgery system, avoiding installing force sensors on delicate surgical tools.

Furthermore, the experiment results showed that the RP could reduce the intensity of the surgeon’s manipulation effectively with the implemented IT2 PFMBL method, by assisting and adapting to surgeon behaviour autonomously. Trained by trajectories and gripper movements of two human operators, the intuitive human-robot cooperation is demonstrated in a peg-transfer task with self-determining movements from the RP. With the assistance of the RP, the surgeon could teleoperate the other surgical robot in the peg-transfer task alone avoiding the communication errors of two surgeons and saving the manpower.


  • [1] J. C. Bezdek (2013) Pattern recognition with fuzzy objective function algorithms. Springer Science & Business Media. Cited by: §II.
  • [2] D. Campolo, P. Tommasino, K. Gamage, J. Klein, C. M.L. Hughes, and L. Masia (2014) H-man: a planar, h-shape cabled differential robotic manipulandum for experiments on human motor control. Journal of Neuroscience Methods 235, pp. 285–297. Cited by: §III-A.
  • [3] S. L. Chiu (1994) Fuzzy model identification based on cluster estimation. Journal of Intelligent & Fuzzy Systems 2 (3), pp. 267–278. Cited by: §II.
  • [4] G. A. Fontanelli, G. Yang, and B. Siciliano (2018) A comparison of assistive methods for suturing in MIRS. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vol. , pp. 4389–4395. External Links: Document Cited by: §I.
  • [5] Y. Huang, W. Lai, L. Cao, J. Liu, E. Burdet, and S. J. Phee (2021) A three-limb teleoperated robotic system with foot control for flexible endoscopic surgery. Annals of Biomedical Engineering. External Links: Document Cited by: §I.
  • [6] H. Hur, D. Arden, L. E. Dodge, B. Zheng, and H. A. Ricciotti (2011) Fundamentals of laparoscopic surgery: a surgical skills assessment tool in gynecology. JSLS Journal of the Society of Laparoscopic & Robotic Surgeons 15 (1), pp. 21–26. External Links: Document Cited by: §III-B.
  • [7] H. Ishida, M. M. Marinho, K. Harada, J. Gao, and M. Mitsuishi (2020)

    Virtual-fixtures for robotic-assisted bi-manual cutting using vector-field inequalities

    In 2020 IEEE/SICE International Symposium on System Integration (SII), Vol. , pp. 395–400. External Links: Document Cited by: §I.
  • [8] H. Li and K. Kawashima (2014) Achieving stable tracking in wave-variable-based teleoperation. IEEE/ASME Transactions on Mechatronics 19 (5), pp. 1574–1582. External Links: Document Cited by: §III-A.
  • [9] Z. Lu, P. Huang, and Z. Liu (2020) High-gain nonlinear observer-based impedance control for deformable object cooperative teleoperation with nonlinear contact model. International Journal of Robust and Nonlinear Control 30 (4), pp. 1329–1350. Cited by: §II.
  • [10] J. Reynoso, A. Meyer, J. Unnirevi, and D. Oleynikov (2012) Robotics for minimally invasive surgery (mis) and natural orifice transluminal endoscopic surgery (notes). In Medical Robotics, pp. 210–223. Cited by: §I.
  • [11] A. Shahdi and S. Sirouspour (2009) Adaptive/robust control for time-delay teleoperation. IEEE Transactions on Robotics 25 (1), pp. 196–205. Cited by: §II.
  • [12] A. Takagi, G. Ganesh, T. Yoshioka, M. Kawato, and E. Burdet (2017) Physically interacting individuals estimate the partner’s goal to enhance their movements. Nature Human Behaviour 1 (3), pp. 1–6. Cited by: §I.
  • [13] S. Treratanakulchai and F. Baena (2021) A passive decoupling mechanism for misalignment compensation in master-slave teleoperation. IEEE Transactions on Medical Robotics and Bionics 3, pp. 285–288. Cited by: §I.
  • [14] Z. Wang, H. Lam, Z. Chen, B. Liang, and T. Zhang (2020) Event-triggered interval type-2 fuzzy control for uncertain space teleoperation systems with state constraints. In 2020 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), Vol. , pp. 1–8. External Links: Document Cited by: §III-A.
  • [15] Z. Wang, H. Lam, B. Xiao, Z. Chen, B. Liang, and T. Zhang (2020) Event-triggered prescribed-time fuzzy control for space teleoperation systems subject to multiple constraints and uncertainties. IEEE Transactions on Fuzzy Systems (), pp. 1–1. External Links: Document Cited by: §I.
  • [16] J. Xu, M. Ge, Z. Liu, W. Zhang, and W. Wei (2021) Force-reflecting hierarchical approach for human-aided teleoperation of nrs with event-triggered local communication. IEEE Transactions on Industrial Electronics (), pp. 1–1. External Links: Document Cited by: §I.
  • [17] C. Yang, G. Peng, L. Cheng, J. Na, and Z. Li (2021)

    Force sensorless admittance control for teleoperation of uncertain robot manipulator using neural networks

    IEEE Transactions on Systems, Man, and Cybernetics: Systems 51 (5), pp. 3282–3292. External Links: Document Cited by: §I.