Teleoperational control allows the operator to complete tasks at a safe, remote location. This is accomplished by having a human operator send control signals from a remote console, traditionally known as master, to the robot performing the task, traditionally known as slave. The slave then returns feedback to the master through sensors, such as cameras and haptic devices, which gives the operator the feeling of telepresence through displays and force feedback. Through improvement of telepresence and robotic systems, teleoperational systems are becoming common place in a wide range of applications such as underwater exploration, space robotics, mobile robots, and telesurgery [1, 2] .
With the advent of surgical robots such as the da Vinci® Surgical System, attempts have been made to investigate the feasibility of remote telesurgery. Anvari et al. in 2005 was first to establish a remote telesurgical service and reported on 21 remote laparoscopic surgeries with an observed delay of 135-140 msec over the 400 km . The signal latency grows as the distance increases, and prior reports have cited 300 msec as the maximum time delay where surgeons began to consider the operation unsafe [4, 5]. When using satellite communication between London and Toronto, a report measured a delay of msec . This unavoidable time delay is the greatest obstacle for safe and effective remote telesurgery because it leads to overshoot and oscillations. These undesirable effects have been observed with as little as 50 msec of time delay in surgical tasks [7, 8]. The challenges of teleoperating under delay has been investigated for over 50 years, and the field is too expansive to cover in this paper. A more broad look can be found in Hokayem and Spong’s historical survey .
The first study on the effects of delay in teleoperational system were by Ferrell in 1965 and found that experienced operators will use a strategy called move and wait . A stable system is realized by inputting a new control command and then waiting to see the effect. Ferrell and Sheridan then developed supervisory control to address the problem of delay . This gives a level of autonomy to the system such that the operator supervises rather than explicitly inputting trajectory motions. Therefore, supervisory control can only be implemented practically at this time in structured environments.
In the case of teleoperating with haptic feedback, also known as bilateral teleoperation, delay has been experimentally and theoretically shown to create instability within the system [11, 12]. Techniques such as wave variables have been used to dampen the unstable overshoot in bilateral control under delay . However, they have been shown to increase time to complete task [14, 15], and therefore teleoperating without haptic feedback can often be a better alternative.
Predictive displays circumvent the deleterious effects from teleoperating under delay by giving the operator immediate feedback through a virtual prediction. Early predictive displays focused on space robotics where delays can reach up to 7sec  . Winck et al. recently applied this approach by creating an entire predicted virtual environment that is displayed to the operator and is also used to generate haptic feedback . Currently this is not applicable to telesurgery because of the unique 3D geometry found in a surgical environment. The scale of the operation requires high precision tracking to create reliable predictions, and obstacles such as tissue cannot be modeled as rigid bodies nor accurately predicted.
In this paper, we present new motion scaling solutions for conducting telesurgery under delay that:
reduce errors to improve patient safety,
have intuitive control since the delay can change from one operation to another due to variable distances, and
have simple implementation so it can be easily deployed to any teleoperated system.
To show the performance of the solutions with statistical significance, we conducted a user study on the da Vinci® Surgical System. The user study was designed such that there is independence from unwanted variables such as participant exhaustion and experience and that the best performing solution has a short learning curve, therefore intuitive. Both time to complete task and an error metric were used to evaluate task performance, and we measured that our proposed solutions decreased the number of errors at the cost of time. In fact, 16 out of the 17 participants from the user study performed best, with regards to error rate, using our proposed solutions.
Teleoperation systems utilizes a scaling factor between master arms movements and the slave arms movements to adapt the teleoperator to the slave workspace. It can be expressed as a general teleoperation system with delay, shown in Fig. 1, as:
where is the scaling factor previously described. It is worth noting that teleoperation systems described in this manner using position control, such as the da Vinci® Surgical System, only apply this scaling to position and not orientation, rather mapping orientation commands one-to-one so there is no obvious mismatch between user wrist orientation and robot manipulator orientations. The solutions we present are all modifications of Equations (1) and (2).
Ii-a Constant Scaling Solution
The first solution simply reduces the scaling factor in Equation (1). Lowering the scaling factor in robotic surgery has been shown to increase task completion time and improve accuracy [19, 20]. We extend this to teleoperating under delay and hypothesize that decreasing the constant scale will reduce the number of errors at the cost of time when operating under delay. This idea is similar to the move and wait strategy, creating stability by slowing down the motions. However, here the motions are slowed continuously, while the move and wait strategy discretizes the motions and requires the operator to have experience working under the delay.
Ii-B Positional Scaling Solution
Positional scaling builds on the constant scaling solution by decreasing the scaling as the slave arms move towards an obstacle. Therefore, reducing the cost of time from the constant scaling solution by slowing down the operators motions only in areas of the work space that require higher precision. Since this deals with obstacle detection, the positional scaling will be implemented on the slave side through the following equations:
where is the distance from to the nearest obstacle, is the rate changes, and and set the lower and upper bounds respectively of . This is paired with Equation (1), and can be thought of as secondary layer of scaling based on proximity to an obstacle.
To implement this scaling method, the distance requires the location of the obstacles in the environment. Methods such as feature-based 3D tissue reconstruction from  or dense pixel-based depth reconstruction can be used. In the presented user study, the task environment is known and distances are found analytically.
Ii-C Velocity Scaling Solution
Velocity scaling is founded on the idea that an operator will naturally move slower when higher accuracy is required. This is inspired by the software controlling computer mice and trackpads, which natively utilize this strategy by default (unless turned off by the owner). Combining velocity scaling with the constant scaling solutions hypothesis, the scaling factor in Equation (1) is increased proportionally to the input translational velocity as shown:
where is the base scaling, the bonus velocity scaling is given at the rate of , is the translational velocity, and is the magnitude. Equation (2) is then used to set the slave position on the slave side. This allows for a low base scaling to get the benefits of the constant scaling solution when the operator is making small, precise motions. It also increases the scaling when the operator makes large motions, which requires less precision, in order to reduce the cost of time.
To find on a real system where there is noise, the following filter can be used:
This represents a running average with weights 0.25, 0.5, and 0.25 for three individual velocity measurements.
To measure the effectiveness of the solutions, a user study with 17 participants was conducted on the da Vinci® Surgical System. The delayed feedback to the user is a stereoscopic 1080p laparoscopic camera running at 30FPS. The camera feed is displayed to a console for stereo viewing by the operator. Both pairs of master and slave arms have seven degrees of freedom. A modified version of Open-Source da Vinci Research Kit was used on an a computer with an Intel® Core™ i9-7940X Processor and NVIDIA’s GeForce GTX 1060.
To ensure that the solutions are tested in a realistic, high delay condition, a round trip delay of 750 msec was simulated for the delayed environment in the user study. This follows a recent report showing satellite communication between London and Toronto measured a round trip delay of msec for a telesurgical task .
A peg transfer task is used as a test scenario. Fig. 3 shows a photo from the endoscope used by the operator in our study. The task involves:
Lifting a rubber o-ring from either the front left or right peg with the corresponding left or right arm
Passing the ring to the other arm
Placing the ring on the other front peg
Repeating steps 1 to 3 for the back pair of pegs
This task and environment was chosen because it inherits complex motions such as hand off, regions where larger movements are safe, and yet still requires precise motions during parts of the trajectory due to the tightness of fit between the rings and the pegs.
The metrics to evaluate task performance are time to complete task and an error metric. The error metric is an enumeration of different types of errors that we observed. These errors are weighted according to Table 1 and summed to get a weighted error. The weights were chosen such that severity of the error would be reflected properly in the error metric. Example errors are shown in Fig. 2. A sample video was created and shown to the participants before the study to show the task and the different types of errors.
|Stretch ring during hand-off for a second or less||2|
|Stretch ring on peg for a second or less||4|
|Stretch ring for an additional second||4|
|Knock down peg||20|
Weights associated with type of error
The scaling scenarios for the participants to complete the peg transfer task are as follows in both 0 msec and 750 msec round trip delay:
Constant scaling of 0.3
Constant scaling of 0.2
Constant scaling of 0.1
For constant scaling, the corresponding scaling value listed above was used for . For positional scaling, the following simplification was made: is the minimum 2D distance from center of each of the 4 pegs to the target tool-tip position projected on the plane constructed from the top of the 4 pegs (). The following equations are used to find the least squared solution of the plane:
where is the position of the center of the -th peg. The following is computed realtime to project the target tool-tip position and find :
Through experimentation, , , , and were chosen for positional scaling. Fig. 4 shows a generated map of the value of using these parameters. is calculated and applied to both arms individually. These positional scaling parameters mean that the total scaling from master to slave ranges from 0.1 to 0.2.
For velocity scaling, the values of and were found through experimentation. Similar to positional scaling, this is calculated and applied to both arms individually.
The procedure for each participant was as follows:
Practice: repeat the task twice with constant scaling of 0.2 under no delay
Record: perform the task with constant scaling of 0.2 under no delay
Practice: select a new scaling method at random under no delay and take 20 seconds to become accustomed to the new scenario
Record: perform the task with the previously selected scaling solution
Repeat step 3 and 4 with all scaling solutions
Repeat step 3 and 4 with constant scaling of 0.2
Repeat 1-6 with round trip delay of 750 msec.
Step 1 is so that participants get over the learning curve of the task and the system. By comparing the results of step 2 and 6, we can ensure that the user overcame the learning curve in step 1 and has not degraded in performance due to exhaustion. This is critical since it shows that the performance of a participant is only affected by the delay and different scaling methods. To further break any potential correlation, the randomization of the order of scaling methods in step 3 and 4 was used. Each participant has only 20 seconds to practice in the new scenario (step 3), so that the solutions with best results will be intuitive.
To show there is no significant change in performance between step 2 and 6 of the procedure, the results are compared using two-sided paired-sample t-tests. Under no delay,and for weighted error and time to complete task respectively is computed. For the roundtrip delay of 750 msec, and for weighted error and time to complete task respectively is computed. Therefore, no statistically significant () change in performance was measured. Combining this result with the randomization of order in step 3 of the procedure implies that the measured results will be uncorrelated with participant experience and exhaustion and only affected by the delay and scaling solution.
The results showing which solution performed best and second best for each individual person under delay are shown in Fig. 5. Only one out of seventeen participants performed best, with regards to weighted error, using the baseline scaling, constant scaling of 0.2 and 0.3. Furthermore, from Fig. 5 it is evident that even when including the second best performing solution, the vast majority of participants performed better with our proposed solutions with regards to weighted error.
The statistics of the results are shown in Table 2 and 3 for roundtrip delay of 750 msec and 0sec respectively. The -values are generated by comparing against the baseline scaling with a two-sided paired t-test to show if the solution had a statistically significant () effect on performance. The box plots of the results are shown at the end of the paper in Fig. 6. All the proposed solutions do achieve statistically significant lower weighted error under delay except for positional scaling compared against constant scaling of 0.2. The proposed solutions were also measured to make a statistically significant increase in time under delay when comparing against constant scaling of 0.2. Therefore, the solutions are working as intended, increasing the accuracy at the cost of time.
|Weighted Error||Time (sec)|
|vs. c = 0.3||vs. c = 0.2||vs. c = 0.3||vs. c = 0.2|
|c = 0.3||16.4 13.3||—||0.305||106 32.0||—||0.224|
|c = 0.2||13.2 8.22||0.305||—||92.5 31.8||0.224||—|
|c = 0.1||9.29 9.62||0.0149||0.0480||131 42.8||0.000316|
|pos.||12.3 14.6||0.0245||0.572||116 38.0||0.173||0.0474|
|vel.||9.31 8.46||0.00926||0.00114||113 35.1||0.284||0.00617|
Statistics from user study trials under a roundtrip delay of 750 msec. -value is generated from two-sided paired t-test.
|Weighted Error||Time (sec)|
|vs. c = 0.3||vs. c = 0.2||vs. c = 0.3||vs. c = 0.2|
|c = 0.3||6.00 5.60||—||0.266||40.3 13.7||—||0.320|
|c = 0.2||4.47 3.40||0.266||—||44.3 17.6||0.320||—|
|c = 0.1||5.35 5.73||0.686||0.486||72.7 27.9|
|pos.||6.12 5.01||0.833||0.243||60.0 20.5||0.000246||0.000100|
|vel.||4.76 4.49||0.259||0.784||53.2 24.5||0.0123||0.0965|
Statistics from user study trials under a roundtrip delay of 0sec. -value is generated from two-sided paired t-test.
The user study results of constant scaling of 0.1 under delay match with our initial hypothesis that decreasing constant scaling slows down the operators motions to achieve higher accuracy. As seen in Table 2, constant scaling of 0.1 makes a statistically significant decrease in errors at the cost of increasing the time to complete task. However, the hypothesis does not appear to hold true when looking at the performance of constant scaling of 0.3 to 0.2 under delay. We believe this is since participants had so many errors, such as ring drops, when using constant scaling of 0.3, that it severely affected their time to complete the task.
By comparing positional scaling and constant scaling of 0.1 relative time to complete task performances under delay, it is evident that positional scaling did decrease the cost of time as intended via dynamic scaling. However, positional scaling did not retain the weighted error performance of constant scaling of 0.1 when under delay. We believe this occurs because our implementation of positional scaling does not count the slave-arms as obstacles. Therefore during the ring pass, which typically occurs in the center of the plot in Fig. 4, there is little to no additional scaling to improve the accuracy. The data from the user study supports this since 2 ring stretches and 7 ring drops during hand off were recorded for positional scaling under delay, and 0 ring stretches and 2 ring drops during hand off were recorded for constant scaling of 0.1 under delay.
The final result is the performance of velocity scaling, which performed the best of all of the proposed solutions under delay. It has similar performance in weighted error as constant scaling of 0.1 at a lower cost of time. While under delay, velocity scaling on average reduced the weighted error rate by 43% and 29% and increased the time to complete task by no statistically significant margin and 22% when compared against constant scaling of 0.3 and 0.2 respectively.
The results from the user study confirms with statistical significance our original hypothesis which states that decreasing the constant scaling factor will improve accuracy at the cost of time. Velocity scaling was shown to improve on this by achieving the same accuracy with a lower cost of time. Positional scaling did not perform as well, and it has the added challenge of requiring environmental information. Participants were also given little time to adjust to the proposed scaling solutions, so the performance improvements implies that they are intuitive. Furthermore, the simplicity of the constant scaling and velocity scaling solutions allows for them to be easily deployed on any teleoperational system under delay. Future work involves dynamically learning the appropriate scales for individual operators and delay conditions.
We thank all the participants from the user study for their time, all the members of the Advanced Robotics and Controls Lab at University of California San Diego for the intellectual discussions and technical help, and the UCSD Galvanizing Engineering in Medicine (GEM) program and the US Army AMEDD Advanced Medical Technology Initiative (AAMTI) for the funding.
-  P. F. Hokayem and M. W. Spong, “Bilateral teleoperation: An historical survey,” Automatica, vol. 42, no. 12, pp. 2035 – 2057, 2006.
-  M. Yip and N. Das, “Robot autonomy for surgery,” arXiv preprint arXiv:1707.03080, 2017.
-  M. Anvari, C. McKinley, and H. Stein, “Establishment of the world’s first telerobotic remote surgical service,” Annals of Surgery, vol. 241(3), pp. 460–464, Mar 2005.
-  J. Marescaux, J. Leroy, M. Gagner, F. Rubino, D. Mutter, M. Vix, S. E. Butner, and M. K. Smith, “Transatlantic robot-assisted telesurgery,” Nature, pp. 379–380, Sep 2001.
-  S. Xu, M. Perez, K. Yang, C. Perrenot, J. Felblinger, and J. Hubert, “Determination of the latency effects on surgical performance and the acceptable latency levels in telesurgery using the dv-trainer ®simulator,” Surg Endosc., pp. 2569–76, 2014.
-  C. Y. Nguan, R. Morady, C. Wang, D. Harrison, D. Browning, R. Rayman, and P. P. Luke, “Robotic pyeloplasty using internet protocol and satellite network-based telesurgery,” The International Journal of Medical Robotics and Computer Assisted Surgery, vol. 4, pp. 10–14, Mar 2008.
-  A. Kapoor, R. Kumar, , and R. Taylor, “Simple biomanipulation tasks with ’steady hand’ cooperative manipulator,” Lect Notes Comput Sc., pp. 141–148, 2003.
-  A. Schweikard, R. Tombropoulos, L. Kavraki, J. Adler, and J. Latombe, “Treatment planning for a radiosurgical system with general kinematics,” IEEE Intl. Conf. on Robotics and Automation, pp. 1720–1727, 1994.
-  W. R. Ferrell, “Remote manipulation with transmission delay,” IEEE Transactions on Human Factors in Electronics, vol. HFE-6, pp. 24–32, Sept 1965.
-  W. R. Ferrell and T. B. Sheridan, “Supervisory control of remote manipulation,” IEEE Spectrum, vol. 4, pp. 81–88, Oct 1967.
-  W. R. Ferrell, “Delayed force feedback,” Human Factors, vol. 8, no. 5, pp. 449–455, 1966. PMID: 5966936.
-  R. J. Anderson and M. W. Spong, “Bilateral control of teleoperators with time delay,” IEEE Transactions on Automatic Control, vol. 34, pp. 494–501, May 1989.
-  G. Niemeyer and J. E. Slotine, “Using wave variables for system analysis and robot control,” in Proceedings of International Conference on Robotics and Automation, vol. 2, pp. 1619–1625 vol.2, April 1997.
-  M. C. Yip, M. Tavakoli, and R. D. Howe, “Performance analysis of a manipulation task in time-delayed teleoperation,” in IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5270–5275, Oct 2010.
-  M. C. Yip, M. Tavakoli, and R. D. Howe, “Performance analysis of a haptic telemanipulation task under time delay,” Advanced Robotics, vol. 25, no. 5, pp. 651–673, 2011.
-  W. Yoon, T. Goshozono, H. Kawabe, M. Kinami, Y. Tsumaki, M. Uchiyama, M. Oda, and T. Doi, “Model-based space robot teleoperation of ets-vii manipulator,” IEEE Transactions on Robotics and Automation, vol. 20, pp. 602–612, June 2004.
-  T. Imaida, Y. Yokokohji, T. Doi, M. Oda, and T. Yoshikawa, “Ground-space bilateral teleoperation of ets-vii robot arm by direct bilateral coupling under 7-s time delay condition,” IEEE Transactions on Robotics and Automation, vol. 20, pp. 499–511, June 2004.
-  R. C. Winck, S. M. Sketch, E. W. Hawkes, D. L. Christensen, H. Jiang, M. R. Cutkosky, and A. M. Okamura, “Time-delayed teleoperation for interaction with moving objects in space,” in IEEE International Conference on Robotics and Automation, pp. 5952–5958, May 2014.
-  S. M. Prasad, S. M. Prasad, H. S. Maniar, C. Chu, R. B. Schuessler, and R. J. Damiano, “Surgical robotics: impact of motion scaling on task performance,” J Am Coll Surg., vol. 199, pp. 863–868, Dec 2004.
-  R. Cassilly, M. D. Diodato, M. Bottros, and R. J. Damiano, “Optimizing motion scaling and magnification in robotic surgery,” Surgery, vol. 136, p. 291–294, Aug 2004.
-  J. Song, J. Wang, L. Zhao, S. Huang, and G. Dissanayake, “Dynamic reconstruction of deformable soft-tissue with stereo scope in minimal invasive surgery,” IEEE Robotics and Automation Letters, vol. 3, pp. 155–162, Jan 2018.
-  P. Kazanzides, Z. Chen, A. Deguet, G. S. Fischer, R. H. Taylor, and S. P. DiMaio, “An open-source research kit for the da vinci ®surgical system,” IEEE Intl. Conf. on Robotics and Automation, pp. 6434–6439, 2014.