Virtual Fixture Assistance for Suturing in Robot-Aided Pediatric Endoscopic Surgery

by   Murilo Marques Marinho, et al.

The limited workspace in pediatric endoscopic surgery makes surgical suturing one of the most difficult tasks. During suturing, surgeons have to prevent collisions between tools and also collisions with the surrounding tissues. Surgical robots have been shown to be effective in adult laparoscopy, but assistance for suturing in constrained workspaces has not been yet fully explored. In this letter, we propose guidance virtual fixtures to enhance the performance and the safety of suturing while generating the required task constraints using constrained optimization and Cartesian force feedback. We propose two guidance methods: looping virtual fixtures and a trajectory guidance cylinder, that are based on dynamic geometric elements. In simulations and experiments with a physical robot, we show that the proposed methods achieve a more precise and safer looping in robot-assisted pediatric endoscopy.



page 4

page 7


A Unified Framework for the Teleoperation of Surgical Robots in Constrained Workspaces

In adult laparoscopy, robot-aided surgery is a reality in thousands of o...

Dynamic Active Constraints for Surgical Robots using Vector Field Inequalities

Robotic assistance allows surgeons to perform dexterous and tremor-free ...

Robot-Assisted Surgical Training Over Several Days in a Virtual Surgical Environment with Divergent and Convergent Force Fields

Surgical procedures require a high level of technical skill to ensure ef...

Active Constraints using Vector Field Inequalities for Surgical Robots

Robotic assistance allows surgeons to perform dexterous and tremor-free ...

Real-time Virtual Intraoperative CT for Image Guided Surgery

Abstract. Purpose: This paper presents a scheme for generating virtual i...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Pediatric endoscopic surgery for infants (<1 year old) and neonates has additional difficulties when compared with adult endoscopic surgery. For example, the workspace is narrower, often being described by medical doctors as having a “golf-ball size”. The limited workspace increases the risks of collisions between tools, which can occur both inside and outside the patient. These difficulties have motivated the usage of surgical robots, such as the da Vinci Surgical System (Intuitive Surgical Inc., USA), which have been shown to improve dexterity, endurance, and vision. The da Vinci Surgical System has had great success in adult laparoscopy; however, it has been shown to be inapplicable to pediatric surgery owing to the large diameter of its instruments (8 mm) and the required in-patient length (5 cm) [1].

To provide appropriate robotic assistance to pediatric surgery and other applications in narrow workspaces, our group has been developing a novel master-slave robotic system, called SmartArm, in parallel with this work [2]. It consists of a pair of industrial robot arms, each of which is instrumented with an actuated flexible tool [3]. The proposed system has tools whose diameters are 3.5 mm, and the preliminary results indicate that our system can operate inside narrow workspaces, such as those in pediatric patients [4]. With the SmartArm system, we expect to bridge the gaps that prevent the wide adoption of robots in pediatric and neonate surgery [5].

As in other surgical robotics scenarios, robotic assistance in pediatric surgery may increase the task completion time owing to motion scaling as reported in the literature [6], especially when performing complex tasks. Suturing is among the most complex surgical procedures. It requires bimanual manipulation of a needle, a thread, and the target tissue. Suturing can be divided into four steps: (1) the tool grabs the needle, (2) the needle is inserted through both sides of the tissue, (3) one of the tools grabs the thread near the needle and loops the thread a few times around the other tool, and finally (4) the loose end of the thread is pulled to tighten the knot. To compensate for possible robot assistance drawbacks and further improve task performance, many groups have proposed assistance methods for suturing subtasks or a combination of subtasks [7, 8, 9, 10, 6, 11].

One of those methodologies is task automation [9, 6], which has been so far demonstrated in an unobstructed space, which is not the case in pediatric endoscopy. Moreover, although the future potential of such techniques is clear, currently they are still outperformed by human-operated robots and are unable to leverage surgical skill efficiently.

In contrast, virtual fixtures do not aim to fully automate the task. Instead, virtual fixtures are used to enhance the operator’s medical skill. A comprehensive survey on virtual fixtures was presented by Bowyer et al. [12]. The survey shows that virtual fixtures are often built using geometric elements such as points, lines, planes, and corresponding volumetric primitives. They are divided into regional virtual fixtures, to create a forbidden region or safe zone, and guidance virtual fixtures, to aid the operator in performing specific tasks.

In this letter, we focus on the generation of guidance virtual fixtures for the looping stage of a suture in an endoscopic pediatric surgery setting. Looping can be time-consuming and requires considerable skill to prevent collisions between tools as well as with the surrounding tissues and organs. This procedure can be particularly challenging when considering the reduced workplace, the reduced haptic perception, and the 2D endoscopic vision in pediatric surgery.

I-a Related works

Many studies have been conducted on the generation of guidance virtual fixtures for suturing. For instance, Kapoor et al. used guidance virtual fixtures to guide needle insertion [7] and in bimanual knot placement [13]. Chen et al. [10] introduced a knot-tying virtual fixture by constraining the tooltip to a plane. Fontanelli et al. [11] compared assistive methods for needle insertion and developed a guidance virtual fixture to constrain the position of the tool along a specific trajectory while the rotations are free. Selvaggio et al. [14] proposed a haptic-guided shared control for needle grasping that significantly improved needle re-grasping performance. The looping task differs from needle insertion in that both tools have to interact with each other and, especially inside the narrow workspace inside the infant, collisions have to be carefully taken into account.

To facilitate safer robot-assisted minimally-invasive partial nephrectomy, Banach et al. [15] proposed tool-shaft and anatomy collision avoidance using the elasto-plastic frictional force control model and validated it on the da Vinci Research Kit (dVRK) [16].

Looi et al. [17] showed a proof of concept of a robot for image-guided anastomosis in pediatric/neonate surgery. The authors reported that the robot was able to autonomously perform sutures in some scenarios, but had difficulties in more realistic situations owing to the workspace restrictions.

In prior works, our group has focused on the generation of dynamic regional virtual fixtures to prevent collisions between tools and to generate task constraints using vector-field inequalities

[18]. More recently, we applied vector-field inequalities to teleoperation tasks and developed a unified framework for teleoperation [4]. Those works included simulations and experiments, in which the relevant task constraints were appropriately maintained. The generation of guidance virtual fixtures, i.e. specific constraints to optimize task execution, has not yet been explored using our framework.

I-B Statement of contributions

In this letter, we

  1. briefly establish the benefits of the vector field inequalities (VFI) method over competing frameworks in the context of real-time virtual-fixtures generation (Section II-C2);

  2. propose an assistive method based on virtual fixtures to assist in the looping test in pediatric/neonatal surgery with the following components (Section III):

    1. (slave side) a set of dynamic forbidden-region virtual fixtures to both prevent collisions between instruments and reduce looping motion, to increase the safety of the looping task;

    2. (slave side) a trajectory guidance cylinder based on dynamic guidance virtual-fixtures to increase looping task performance;

    3. (master side) a Cartesian force-feedback that guides the user towards the trajectory guidance cylinder.

The proposed assistive method is evaluated in simulations and experiments with naive and expert users using an anatomically correct infant model and is shown to be beneficial to task performance in some cases.

Ii Mathematical Background

In our proposed method, virtual fixtures are modeled using dual quaternion algebra and the VFI method [18] based on quadratic programming for closed-loop inverse kinematics. In this section, we briefly present the required mathematical background and notation. The proposed technique for assistance is shown in Section III.

Ii-a Quaternions and dual quaternions

Quaternions can be regarded as an extension of complex numbers. The quaternion set is , in which the imaginary units , , and have the following properties: . The set has a bijective relation with . In other words, the quaternion represents the point . The set of quaternions with unit norm is expressed as , and any can always be written as , where is the rotation angle around the rotation axis [19].

The dual quaternion set can be defined as , where is the dual unit [20]. Elements of the set are called unit dual quaternions, and represent the combination of position and rotation of a rigid body. Given , it can be always written as , where and represents the orientation and position respectively [20].

Elements of the set are called pure dual quaternions and are convenient to represent points , lines, and planes in . Given , the inner product and cross product are respectively [19]

A Plücker line can be written by a pure unit dual quaternions such as [19]

where is a pure quaternion with unit norm that represents the line direction, and

is the line moment, in which

is a point in the line.

A plane can be written by a dual quaternion such as [21]

where is a pure quaternion with unit norm that represents the direction normal to the plane, and

Ii-B Constrained optimization algorithm

Without loss of generality, the following constrained optimization algorithm can be used to teleoperate two identical slave robots with [4]:

subject to


in which , , and are the unweighted cost functions related to the end-effector translation, end-effector rotation, and joint velocities of the -th robot, respectively; furthermore, each -th robot has a vector of joint velocities , a translation Jacobian , a translation error , a rotation Jacobian , and a switching rotational error

where and are the desired and current end-effector orientations, respectively. In addition, , and is a positive definite damping matrix, usually diagonal. Finally, are weights used to define the priorities between the translation and the rotation and the priorities between robots. The linear constraints can be used to avoid joint limits [22] and to generate task constraints [18]. Each parameter is explained in more detail in [4].

Ii-C VFI method

The VFI method [18] requires a function that represents the (signed) distance between two geometric primitives. It also requires a distance Jacobian and a residual relating the time derivative of the distance function and the joints’ velocities in the general form


where the residual contains the distance dynamics unrelated to the joints’ velocities. The relevant functions, distance Jacobians, and residuals for all relevant primitives used in this paper are shown in [18]. Finally, the VFI method requires the definition of a safe distance and an error to generate restricted zones or to generate safe zones.

With these definitions, and given , the signed distance dynamics for each pair of primitives is constrained by


Constraint 3 actively filters the robot motion only in the direction approaching the restricted zone boundary so that the primitives do not collide, and the other robot motions are unaffected.

To use VFIs to generate restricted zones, we use the constraint

for . Finally, safe zones are generated by using the constraint

Ii-C1 Generating an entry sphere using VFIs

As an example relevant to the application of this letter, in infant surgery, instead of an entry-point constraint, an entry-sphere constraint is used [4]. This constraint replicates the manual technique of medical doctors that utilizes the compliance of the infant’s skin to increase the reachable workspace. To generate this constraint, without loss of generality we define a coordinate frame whose axis is along the shaft of the tool of a given robot . The Plücker line associated with the tool shaft’s axis, can be expressed as

in which is the line direction and is the line moment. With the center of the entry sphere given by , the derivative of the squared distance between the entry point and the tool shaft is given by

where is the line-to-point squared distance Jacobian [18]. Using the VFI method, the tool shaft can be constrained by an entry sphere of squared radius by using the following linear constraint:


where is a gain that affects the allowed speed of the tool shaft toward the surface of the sphere.

Ii-C2 Why VFI?

There are a myriad competing techniques for the generation of virtual fixtures/active constraints [12]. For nonredundant robots, such as the da Vinci, virtual fixtures based on force feedback on the master side are effective [23, 12, 15]. For redundant robots, such as the SmartArm and similar systems, only force feedback on the master side is in general not enough, owing to possibly infinite mappings between master and slave postures. That is, with an unconstrained inverse kinematics, pushing the master in a given direction in general does not guarantee that a redundant slave’s links will move in a repeatable manner. This problem becomes more evident in surgeries in narrow spaces such as pediatric and neonate surgery. In this context, we have been developing a framework [4] based on active constraints generated through constrained optimization on the slave side, which guarantee the integrity of the robotic systems and the safety of the patients and operating room personnel. On the master side, we add Cartesian impedance to make the operator aware of the workspace constraints.

To validate the benefits of the VFI method over existing works, we show a brief simulation study.

Target path
Endoscopic view
Fig. 1: The V-REP simulation used in the simulation comparison. The circle’s -axis position changed in time following a sinusoidal wave .

In the existing literature, active constraints using constrained optimization in the context of robotic surgery were initially proposed by Kapoor et al. [8]. They proposed several primitives that can be used to assemble customizable virtual fixtures, and one of their primitives in common with the VFI method is the plane constraint. In this context, we compare the performance of the constraint proposed in [8] with what can be achieved with VFIs. Because there is no explicit objective function for teleoperation in [8], we used the same objective function (with the same gains) for both methods and changed only the constraints.

To compare both techniques, we used the V-REP simulation developed in [11]. A suitable scenario using the plane constraint common to both techniques is to require the robots’ tooltips to be restricted to a dynamic plane while the robot is tele-operated. The dynamic-plane distance changes in a sinusoidal manner along the -axis according to with a fixed normal.

The user was asked to trace, by using the master interface, a circle seen through the simulated endoscope. The experiment was performed once without any plane motion, and that trajectory was recorded and played back for each technique to ensure that the trajectory on the -plane was the same for all cases. To further increase the difficulty of the task, the tooltip starts  mm away from the dynamic plane.

The tooltip distance with respect to the plane for each method is shown in Fig. 2. A considerable deviation from the plane,  mm, was observed when using the constraints proposed in [8]. This occurred because the constraint proposed in [8] does not take into account the instantaneous velocity of the plane; therefore, there was a steady-state offset between the desired plane and the actual plane. When VFIs were used, the residual (as shown in (2)) acted as a feed-forward term that compensated for the plane’s instantaneous velocity, which allowed convergence to the moving plane.

This property is required for the proper generation of dynamic virtual fixtures such as the ones proposed in this letter.

0246810Time [s]-30-55Distance to moving planePrevious methodProposed methodReference-100-50050y [mm]Proposed technique-100-50050y [mm]-50050Prior techniqueOver planeOn planeUnder planePlane penetration during circle tracking
Fig. 2: Robot tooltip’s distance to the moving plane. When the plane constraint is defined by using the prior technique [8], the tooltip considerable deviates from the plane. Using our proposed technique, the tooltip converges to the moving plane and moves with it.

Iii Proposed assistive method for suturing

The proposed assistive method for suturing is divided into two parts: a constrained optimization algorithm on the slave side (Section III-B) and Cartesian impedance feedback on the master side (Section III-C).

Iii-a Problem statement

Infant Model Entry Spheres
Fig. 3: Robot setup for the endoscopic infant surgery.

Consider the setup shown in Fig. 3. Let there be two robots, and . with instruments as their end effector. Suppose that the instruments can be simplified as capsules (cylinders with spheres on their end points). For robot-aided endoscopic infant surgery, each instrument has to be inserted through the intercostal space (between the ribs) of the infant. Each incision on the skin cannot be pushed, to prevent further damage to the tissue. This is accomplished by adding an entry-sphere constraint for each robot, as discussed in Section II-C1..

One of the required steps of a surgical suture is to loop the thread about one of the instruments before grasping the loose end of the thread and tying the knot. In this step, the inexperienced user can loop too far (risking collisions with the anatomy) or too close (risking collisions with the other instrument). The proposed technique, described in the following sections, aims to assist the surgeon in performing the looping in a safer and more controlled manner.

Iii-B Slave side: Constrained optimization

Let be the slave robot operated by the non-dominant hand, and be the robot operated by the dominant hand. In this work, we propose a set of dynamic virtual fixtures attached to that constrain the motion of to aid the robotic thread looping task in suturing. The proposed virtual fixtures have been designed by careful inspection of videos of medical practice in pediatric surgery [24] and fruitful discussions with cooperating surgeons. We make two basic assumptions. First, restricting the motion of one tool with respect to the other using virtual fixtures during the looping task to a guidance region can be helpful in reducing extraneous motion. Second, adding a guidance surface can further improve task performance.

The dynamic virtual fixtures, which we call looping virtual fixtures (LVFs), are generated by employing a shaft—shaft collision avoidance primitive [18] plus three geometric primitives for the tooltip of . First, we attach a dynamic cylinder with a radius of around the axis of the end effector of

in which and are respectively the rotation and translation of . The cylinder is cut with a pair of planes and whose normals are collinear with and are respectively placed at and from the tooltip of

in which . These geometric constraints, shown in Fig. 4, limit the motion of so that its tooltip is constrained within a motion envelope to prevent large motions that can be detrimental to task performance, as well as preventing collisions between the shafts and the surrounding tissues. The radius of the cylinder and the placement of the planes can be tuned to balance loop size and task performance.

The LVF can be generated using linear constraints by employing VFIs. With the shaft-to-shaft collision avoidance given by [18]

the LVFs can be generated by the following linear constraints


Furthermore, to further increase assistance, we propose a guidance virtual fixture, called trajectory guidance cylinder (TGC). It comprises a cylinder with its centerline collinear to and radius that is placed inside the LVFs and used to guide the tooltip of .

Fig. 4: Visualization of LVFs (left) and TGC (right).

We propose the following constrained optimization problem to implement both the LVFs and the TGC


where is related to trajectory tracking as in (1) and is related to the TGC of the -th robot as follows

in which is the line-to-point distance Jacobian between the Plücker line collinear with the shaft of , , and the tooltip of , . Moreover, is the point-to-line distance Jacobian between and . Furthermore, the guidance error is defined as , in which is the squared distance between and . The aforementioned Jacobians and distances are calculated using [18]. Lastly, are, respectively, weights used to define the priorities between the translation and the rotation and between the master-slave tracking and the TGC. The linear constraints are given by

in which is related to joint limit avoidance [22], is related to the entry-sphere constraint as in (4), and finally as defined in (5).

Iii-C Master side: Cartesian impedance with guidance

In addition to the existing method of Cartesian force feedback introduced in [4], we propose an additional force feedback to guide the tooltip of to the cylinder in the form

for each th robot master–slave pair, where is the force on the master side, and are stiffness and viscosity parameters. is the translation error of the slave, seen from the point of view of the master, and is the linear velocity of that master interface. is the guidance error in the slave from the point of view of the master.

Iv Simulation and Experiment

To evaluate the technique proposed in this paper, we designed a simulation study and an experimental study. The simulation entailed naive users operating a simulator (V-REP, Coppelia Robotics, Switzerland) under three different conditions to evaluate the effects of the proposed technique. The experimental study investigated medical doctors’ performance using the proposed technique while operating a robotic system [25] based on the SmartArm architecture [2]

, with a two degrees-of-freedom (rotation, grasping) tool attached, in a setup equivalent to


The experimental setup is shown in Fig. 6. The 3D model of an infant [24] was placed between the two robotic arms, and the entry spheres (Section II-C1) were placed between the ribs of the infant model at their relevant locations. In these experiments, the two robotic arms (DENSO VS050, DENSO WAVE Inc., Japan) were equipped with rigid 3.5-mm-diameter tools. The simulator replicated the experimental setup.

Both the simulation and the experiment used the same software implementation on Ubuntu 16.04 x64 running ROS Kinetic Kame.111 Robot kinematics was implemented using the DQ Robotics.222 library, and constrained convex optimization was implemented using IBM ILOG CPLEX Optimization Studio333

All p values reported in this section were obtained through the (two-tailed) Wilcoxon signed-rank task.

3.5 20 10 2.5 -8 10
TABLE I: Virtual fixture parameters [mm].
0.999 0.6 0.01 150 30 1 0.02 0.0005 50

: translation error to orientation error weight (Section III-B).

: robot priority weight (Section III-B)

: weights between the master-slave tracking and TGC (Section III-B).

, ,: proportional gain of the kinematic controller, LVF, and TGC, respectively.

: Robot joint gains (Section III-B).

: Cartesian impedance proportional and viscosity gains, respectively (Section III-C).

MS: Motion scaling. A motion scaling of X means that a relative translation of the master was multiplied by X before being sent to the slave.

TABLE II: Control parameters for the simulation and the experiment.

Iv-a Simulation

For this study, five volunteers were recruited among the engineering students in the University of Tokyo, who had no medical experience. After being shown a video demonstrating the ideal double loop trajectory, the users were instructed to replicate that loop as closely as possible. Each user was asked to perform in the simulator a double loop in a total of five trials in each of the following three conditions: A1 with no virtual fixtures (control group), A2 with only the shaft–shaft collision avoidance (first mentioned in [18]), and A3 with the proposed LVFs and TGC. To reduce possible biases, the sequence of trials was randomized so that the users did not know which condition was activated at a given trial. The proposed virtual fixtures were implemented using (6). Their relevant parameters are shown in Table I and were determined by pilot studies that included a medical doctor. The control parameters used in the experiments are shown in Table II.

The simulation had two purposes. First, to find reasonable control parameters before implementing the proposed technique in the robotic system. Second, to investigate condition A1, which has no virtual fixtures; therefore, we can investigate collisions between instruments and the anatomical model without damaging the physical equipment. It is important to note that the surgical thread was not simulated; only the loop motion was evaluated.

Iv-A1 Results and discussion

Controller performance

To evaluate the performance of the looping task, we used the task completion time and the error between the tooltip position of and the surface of the guidance cylinder (Section III-B) that delineates the ideal looping surface. Table III shows the median time and error recorded for the five volunteers for each of the three experimental cases.

For ing A1, the users required an median of 27.2 s. That was the shortest required time. However, the mean trajectory error was the highest and, as expected, there were collisions between the instruments.

A2 provided a collision-free path for the instruments and a 14% reduction in the mean trajectory error with respect to A1. There was a 128% increase in the median required time with respect to A1.

Finally, after adding the proposed guidance virtual fixtures, in condition A3 the instrument path was also collision free. Moreover, there was a reduction of 30% in mean trajectory error with respect to A2 (40% with respect to A1). There was a 19% decrease in the median required time with respect to A2 (87% increase with respect to A1). The significance between the median completion times of A2 and A3 was p = 0.067 which does not reject the null hypothesis with the widely accepted 95% confidence level, but gives us reasonable confidence to say that the difference in median between the two groups is not due to sampling error. Nonetheless, more users will be recruited in follow-up studies to validate this hypothesis.

These results show that our proposed method, A3, guided the tools to the desired path while providing the required constraints for the anatomy (entry point) and the instruments (shaft–shaft constraint).

User evaluation

The users were asked to complete a modified version of the NASA TLX work load survey [26] to evaluate the workload in terms of four indicators: mental demand, physical demand, effort, and frustration. The original NASA TLX survey also has the temporal and performance dimensions, but those were already evaluated in the controller performance. The results of the survey are shown in Fig. 5. We normalized the scores of the NASA TLX to compare the relative scores between techniques.

A1 had the lowest score for all four indicators, because the robot motion is unconstrained (no entry sphere, no shaft–shaft collision avoidance, no guidance). However, with A1, no volunteer could successfully execute the looping task without collisions.

Furthermore, comparing A2 with A3, A3 significantly reduced the physical demand (p < 0.05) and mental demand (p < 0.05) during the looping motion.There was no significant difference in effort (p = 0.15) and frustration (p = 0.66).

The sum of the NASA TLX indicators for each condition were 32.8 (for A1), 42.0 (for A3), 48.6 (for A2) out of 84. The sum of A1 was significantly lower than A2 (p < 0.05) and A3 (p < 0.05). There was no significant difference in the sum for A2 and A3 (p = 0.077).

These results show that with the proposed guidance, A3, the mental and physical demands decreased with respect to A2.

Fig. 5: NASA TLX workload survey in four conditions for simulation. Conditions are as follows A1 with no virtual fixtures, A2 with only shaft-shaft collision avoidance, A3 with both LGFs and TGC. Values near the center indicate better results.
A1 A2 A3
Median time [s] 25 57 46
Mean error [mm] 1.95 1.69 1.19
TABLE III: User performance in the simulation.

Iv-B Experiment

Master Manipulatos Infant Model Slave robots Endoarm
Fig. 6: Master–slave robotic system. (Left) Master-side, (Right) Slave-side.
Endoarm Entry Sphere Robotic Tools Robotic Tools 3D printed infant model Needle Thread
Fig. 7: Experimental Setup. (Left) 3D printed infant model[1] from top view, (Right) Endoscopic View.

The second experiment was designed to study the effect of the proposed methods on the medical doctors during a robot-aided surgical looping. The experimental setup replicated the simulation, and the experimental parameters are shown in Table II. A surgical thread (5-0 PERMA-HAND SILK, ETHICON, USA) was held by the robot controlled by the operator’s dominant hand. In this experiment, the base of the robots and the tooltip position with respect to the robotic end effector were calibrated by using a visual-tracking system (Polaris Spectra, NDI, Canada) through a pivoting process [18].

Two pediatric surgeons, one expert level (EL) and one intermediate level (IL) were recruited. The surgeons were instructed to perform the double loop under two conditions: B1 with only shaft–shaft collision avoidance and B2 with both LVFs and TGC. The double loop was performed 10 times, 5 times for each condition The users operated the robots using the haptic interfaces and the images captured by the endoscope (Endoarm, Olympus, Japan) were displayed on a monitor. All the conditions were assigned in random sequence to reduce possible biases.

Iv-B1 Results and discussion

Controller performance

The medical doctors successfully conducted the surgical looping under both conditions. The task completion time and the error between the tooltip of and the surface of the guidance cylinder (Section III-B) were recorded during the experiment. The results are shown in Table. IV. In the task completion time, there was no significant difference (p > 0.30). The error between the tooltip and the guidance cylinder decreased by an average of 33% for the EL and by 11% for the IL.

User evaluation

The surgeons completed the same modified NASA TLX survey and there was no significant difference (p > 0.25) in the workload between B1 and B2 (Fig. 8).

These results indicate that the proposed method successfully guided the tools closer to the desired surface without a noticeable difference in the workload felt by the surgeon. For the median task completion time for conditions B1 and B2, there was no significant difference for the EL (p = 0.875) and for the IL (p = 0.32). The median task completion time for the IL appears skewed towards a reduction of task completion time when using the guidance virtual fixtures in condition B2, but more users will be required in follow-up studies to further investigate this hypothesis.

Fig. 8: NASA TLX workload survey in four conditions for experiment. Conditions are as follows B1 with only shaft-shaft collision avoidance, B2 with both LGFs and TGC. Values near the center indicate better results.
Expert Intermediate Total
B1 B2 B1 B2 B1 B2
Median time [s] 33 32 31.5 28 32 31
Mean error [mm] 0.89 0.59 2.16 1.92 1.82 1.40
TABLE IV: User performance in the experiment.

V Conclusions and future work

In this paper, two virtual fixtures for looping were proposed, the looping virtual fixtures (LVFs) and the trajectory guidance cylinder (TGC). These methods can improve the safety and precision of the looping task in surgical suturing under the constrained workspace of pediatric surgery. On the slave side, a constrained optimization algorithm generates the LVFs and the TGC. On the master side, a Cartesian impedance algorithm allows the user to “feel” safe directions (LVFs) and optimal directions (TGC) during the looping. The proposed algorithm is evaluated in a simulation and a experiment with users that show the safety and increased precision of the looping. Our results indicate that the proposed methods are more effective when used by operators with less experience.

In future works, we plan to test the performance of the LVFs and TGC in the nine degrees-of-freedom SmartArm. Furthermore, we are planning to use the LVFs as the stepping stone for full automation of surgical looping.


  • [1] S. Takazawa, T. Ishimaru, K. Harada, K. Deie, A. Hinoki, H. Uchida, N. Sugita, M. Mitsuishi, T. Iwanaka, and J. Fujishiro, “Evaluation of surgical devices using an artificial pediatric thoracic model: A comparison between robot-assisted thoracoscopic suturing versus conventional video-assisted thoracoscopic suturing,” Journal of Laparoendoscopic & Advanced Surgical Techniques, vol. 28, pp. 622–627, may 2018.
  • [2] M. Marinho, A. Nakazawa, J. Nakanishi, T. Ueyama, Y. Hasegawa, J. Arata, M. Mitsuishi, and K. Harada, “Conceptual design of a versatile robot for minimally invasive transnasal microsurgery,” in Micro-NanoMechatronics and Human Science (MHS), 2016 International Symposium on, pp. 1–3, IEEE, 2016.
  • [3] J. Arata, Y. Fujisawa, R. Nakadate, K. Kiguchi, K. Harada, M. Mitsuishi, and M. Hashizume, “Compliant four degree-of-freedom manipulator with locally deformable elastic elements for minimally invasive surgery,” in 2019 International Conference on Robotics and Automation (ICRA), IEEE, may 2019.
  • [4] M. M. Marinho, B. V. Adorno, K. Harada, K. Deie, A. Deguet, P. Kazanzides, R. H. Taylor, and M. Mitsuishi, “A unified framework for the teleoperation of surgical robots in constrained workspaces,” in 2019 International Conference on Robotics and Automation (ICRA), IEEE, may 2019.
  • [5] T. P. Cundy, H. J. Marcus, A. Hughes-Hallett, S. Khurana, and A. Darzi, “Robotic surgery in children: adopt now, await, or dismiss?,” Pediatric Surgery International, vol. 31, pp. 1119–1125, sep 2015.
  • [6] T. Osa, N. Sugita, and M. Mitsuishi, “Online trajectory planning and force control for automation of surgical tasks,” IEEE Transactions on Automation Science and Engineering, vol. 15, pp. 675–691, apr 2018.
  • [7] A. Kapoor, M. Li, and R. H. Taylor, “Spatial motion constraints for robot assisted suturing using virtual fixtures,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 89–96, Springer, 2005.
  • [8] A. Kapoor, M. Li, and R. H. Taylor, “Constrained control for surgical assistant robots.,” in Robotics and Automation (ICRA), 2006 IEEE International Conference on, pp. 231–236, 2006.
  • [9] S. Sen, A. Garg, D. V. Gealy, S. McKinley, Y. Jen, and K. Goldberg, “Automating multi-throw multilateral surgical suturing with a mechanical needle guide and sequential convex optimization,” in 2016 IEEE International Conference on Robotics and Automation (ICRA), IEEE, may 2016.
  • [10] Z. Chen, A. Malpani, P. Chalasani, A. Deguet, S. S. Vedula, P. Kazanzides, and R. H. Taylor, “Virtual fixture assistance for needle passing and knot tying,” in Intelligent Robots and Systems (IROS), 2016 IEEE/RSJ International Conference on, pp. 2343–2350, IEEE, 2016.
  • [11] G. A. Fontanelli, G.-Z. Yang, and B. Siciliano, “A comparison of assistive methods for suturing in MIRS,” in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, oct 2018.
  • [12] S. A. Bowyer, B. L. Davies, and F. R. y Baena, “Active constraints/virtual fixtures: A survey,” IEEE Transactions on Robotics, vol. 30, no. 1, pp. 138–157, 2014.
  • [13] A. Kapoor and R. H. Taylor, “A constrained optimization approach to virtual fixtures for multi-handed tasks,” in 2008 IEEE International Conference on Robotics and Automation, IEEE, may 2008.
  • [14] M. Selvaggio, A. Ghalamzan, R. Moccia, F. Ficuciello, and B. Siciliano, “Haptic-guided shared control for needle grasping optimization in minimally invasive robotic surgery,” p. (in press), 2019.
  • [15] A. Banach, K. Leibrandt, M. Grammatikopoulou, and G.-Z. Yang, “Active contraints for tool-shaft collision avoidance in minimally invasive surgery,” in 2019 International Conference on Robotics and Automation (ICRA), IEEE, may 2019.
  • [16] P. Kazanzides, Z. Chen, A. Deguet, G. S. Fischer, R. H. Taylor, and S. P. DiMaio, “An open-source research kit for the da vinci surgical system,” in IEEE Intl. Conf. on Robotics and Auto. (ICRA), (Hong Kong, China), pp. 6434–6439, 2014.
  • [17] T. Looi, B. Yeung, M. Umasthan, and J. Drake, “KidsArm an image-guided pediatric anastomosis robot,” in 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, IEEE, nov 2013.
  • [18] M. M. Marinho, B. V. Adorno, K. Harada, and M. Mitsuishi, “Dynamic active constraints for surgical robots using vector-field inequalities,” IEEE Transactions on Robotics, pp. 1–20, 2019.
  • [19] A. T. Yang, Application of quaternion algebra and dual numbers to the analysis of spatial mechanisms. PhD thesis, Columbia University., 1963.
  • [20] J. M. Selig, Geometric fundamentals of robotics. Springer-Verlag New York Inc., 2nd ed., 2005.
  • [21] B. V. Adorno, “Robot kinematic modeling and control based on dual quaternion algebra — Part I: Fundamentals,” 2017.
  • [22] F.-T. Cheng, T.-H. Chen, and Y.-Y. Sun, “Resolving manipulator redundancy under inequality constraints,” IEEE Transactions on Robotics and Automation, vol. 10, pp. 65–71, Feb 1994.
  • [23] S. A. Bowyer and F. R. y Baena, “Dynamic frictional constraints for robot assisted surgery,” in World Haptics Conference (WHC), 2013, pp. 319–324, IEEE, 2013.
  • [24] K. Harada, G. Ishikawa, S. Takazawa, T. Ishimaru, N. Sugita, T. Iwanaka, and M. Mitsuishi, “Development of a neonatal thoracic cavity model and preliminary study,” Journal of Japan Society of Computer Aided Surgery, vol. 18, no. 2, pp. 80–86, 2016.
  • [25] S. Kim, J. L. Buendia, S. A. Heredia-Perez, M. M. Marinho, K. Harada, N. Kaneko, T. Ushiku, and M. Mitsuishi, “Towards the automation of grossing in pathology examinations using industrial robotic arms,” in The 14th Asian Conference on Computer Aided Surgery (ACCAS), pp. 202–203, 2018.
  • [26] S. G. Hart, “Nasa-task load index (NASA-TLX) 20 years later,” Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 50, pp. 904–908, oct 2006.