Robust High-Transparency Haptic Exploration for Dexterous Telemanipulation

03/03/2020 ∙ by Keyhan Kouhkiloui Babarahmati, et al. ∙ 0

Robot teleoperation has been proposed as a solution to perform complex tasks in dangerous or remote environments. Teleoperation dexterity is not yet sufficient to enable effective interaction control, even in short-range operation, and low communication delays between the slave and the master. This manuscript explores the development of a haptic teleoperation setup that relies upon a recently proposed passive impedance controller, called Fractal Impedance Controller (FIC). The controller has a state-dependent impedance which enables to tune the slave task accuracy. The proposed controller has been compared with a similar setup using a traditional impedance controller. The results show that the proposed control architecture has higher transparency of interaction compared to the impedance controller. A preliminary study on the ability of an expert user to perform dexterous tasks has been conducted maintaining constant controller parameters. The tasks included opening an electrical box, driving a pile in the sand, pushing a ball on rough terrain, moving a sponge in an obstacle course, and pushing an E-stop button. The results show that the proposed method can complete all the task while the impedance controller on the slave could not. Furthermore, the FIC enables a shorter completion time on the tasks completed that have been also completed from the traditional impedance controller (pushing a ball, moving the sponge, and pushing E-Stop).



There are no comments yet.


page 1

page 2

page 4

page 6

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Exploration robots have already been deployed in multiple space missions both on Mars and the Moon. However, the current technology still offers limited manipulability. The development of more dexterous teleoperation platforms can help the deployment of robot-driven exploratory missions which can relieve the astronauts from the most dangerous tasks. This is the motivation of the METERON (Multi-Purpose End-To-End Robotic Operation Network) project [1, 2] from the European Space Agency (ESA) that aims to demonstrate the operational ability to teleoperated planetary robots from in-orbit crewed station. For example, on Mars, ancient potential life signs are expected to be found in sedimentary layers. Tasks such as collect and analyse on-site geological samples, assemble and build planetary structures are strongly demanded. Drilling, digging and collecting specific ground samples for analysis on complex surfaces of various mechanical properties will require dexterous manipulations with human supervision and low signal latency (e.g. from surface base) for dealing with complex situations. Similar capabilities are also strongly sought on earth where hazardous environments prevent direct human intervention such as nuclear decommissioning or underwater robotics. These delicate and dexterous tasks involving establishing and breaking multiple contacts with the environment are still challenging to model or fully automate using autonomous robots. Therefore, a lot of research effort is still put into studying human-in-the-loop guidance and supervision based on multimodal sensory interface, especially the haptic feedback for highly accurate manipulation tasks.

Figure 1: Experimental setup: (a) Slave (Panda 7-DoF Manipulator Arm) and Simulated Environment; (b) Master Setup: Sigma.7 and GUI.
Figure 2: Multimodal Teleoperation Set-up: the proposed system provides to the user two interfaces. The first interface is the GUI that allows to operate the robot remotely by changing the slave’s (Panda, Franka Emika AG) end-effector reference pose (). The GUI also provides visual feedback from a RGB camera (FLIR-AX8) and a virtual model of the environment. The second interface is the haptic device (Sigma.7, Force Dimension), which has two modalities. When the gripper button is activated, the current position error (with respect to a neutral position) in the master is added to (). In the second modality, the master controls the slave by applying a virtual force at the slave’s end-effector (). The interaction force/torque feedback (), measured with an Optoforce HEX-70-CH-4000N at the end-effector of the slave, is relayed back to the master’s admittance controller (A). Finally, there are two Fractal Impedance Controllers (FIC). The slave controller (FIC) in the slave, and the master controller (FIC)

Robots can provide a solution for reducing the intrinsic risk associated with with extreme and hazardous environments. However, they are not yet able to meet the robustness and dexterity of interaction required even by task that are mundane for human operators. Port-Hamiltonian controllers (i.e., impedance and admittance, [3, 4]) have been identified as a good candidate to enhance the interaction capabilities of robots deployed in such environments. These controllers are designed to drive the robot as an equivalent mass-spring-damper system. However, the coupling of the robot dynamics with the environment poses a threat to both their stability and robustness, if the robot is not able to track the energy exchanged during interaction [5, 6, 7]. Furthermore during bilateral telemanipulation, information delay is highly detrimental to the interactions tracking, which increases the risk of unstable behaviour and reduces the system robustness.

A known solution is guaranteeing the passivity of the system. Based on the passivity theory, any interaction with passive systems is guaranteed to be stable, and any combination of passive systems is guaranteed to be passive [8]. Passivity-based approaches have been implemented to derive necessary conditions for stability with respect to the energy flow of the system [9]. However, the traditional passive impedance control has been found only feasible under constrained conditions, which are determined by the aforementioned coupling of the robot and environmental dynamics [10]. Therefore, Virtual energy tanks have been proposed to achieve dexterous manipulation via the passivity framework [11, 12, 13, 14, 15, 16]. Energy tank controllers enforce passivity of an impedance control by tracking the energy exchange between the system and the environment. The non-conservative energy exchanges are tracked using virtual springs (i.e., energy tank). This allows the system to use this stored energy to guarantee that the extracted energy is less than or equal to the injected one [17, 18, 19]. Thus, they can retain a ”memory” of the interaction and return to the environment an energy equal or lower to the received amount. They can be used to implement passive variable impedance controllers [20, 11]. However, the tank-based controllers are sensible to the degradation of energy tracking, which occurs in dexterous teleoperation with the increase of the feedback delay and the uncertainties in environmental interactions.

A novel passive framework based on a state dependant impedance controller has been recently proposed in [21]. Differently from other variable impedance controllers, it does now rely upon an active viscous component for velocity tracking and stability. Instead, it achieves asymptotic stability through having an anisotropic behaviour between divergence and convergence (i.e., hysteresis cycle). The anisotropy is controlled using a switching controller that introduces an additional impedance when a velocity zero-crossing is detected. Furthermore, the controller has a maximum exertable force that can be calibrated based on the system characteristics once the desired stiffness profile during divergence has been chosen. All the other control parameters are automatically derived. Differently from other controllers, it does not rely on the tracking of non-conservative forces for passivity, making it extremely suitable for application dealing with interaction uncertainties and information delays.

This manuscript proposes the control architecture in Figure 2 and compares the interaction behaviour of the fractal impedance controller with a traditional impedance controller in short-range teleoperation (i.e., negligible communication delays). Therefore, the evaluation experiments have been constrained to evaluate uncertainties of interaction without time delay. The contribution of this framework is the development of a passive teleoperation framework based on the fractal impedance control proposed in [21].

The rest of the paper is constructed as follows: In section II the design of the proposed method for passive bilateral telemanipulation is presented. Then, section III discusses the experimental validation followed by section III-A, which consists of the experimental results of the carried out experiments. In the end, discussion, the conclusion and future works are provided in section IV.

Ii Control Architecture

The proposed method for bilateral haptic manipulation, Figure 2, has two physical interfaces. The first interface is between the user and the master. It is controlled using an admittance controller that mimics the interaction dynamics at the robot end-effector. The second interface is between the slave and the environment, where the system behaves as an impedance mimicking the user input acquired by the master device. Therefore, higher is the transparency of the intermediary (i.e., teleoperation system) higher is the dexterity of interaction [22]. An ideal teleoperation transparency is when the system set-up acts as an identity transformation with infinite bandwidth (i.e, there is no intermediary between the user and the environment).

The proposed architecture aims at improving the system transparency by using a Fractal Impedance controller (FIC) [21] instead of the classic impedance controller in the slave (IC). Appendix A gives a more detailed overview to the FIC. Within the context of this paper, the subscripts S and M stand for Slave and Master, respectively. We also proposed the introduction of an FIC for Safety and Haptic Enhancement (FIC), which will be used in this work as a safety mechanism on the master device. The master device is controlled using an admittance controller (A

), using an identity matrix as gain and includes the Sigma.7 gravity compensation. The master position error (

) is the input to P controller () generating a virtual force () on the slave’s end-effector whenever the button of the master’s gripper is not pressed. In the case where the user activates the gripper button, the slave’s desired position () is recorded. It will be continuously updated by adding the current master position to the initial value of until the gripper is released. In other words, .

Despite these control modality being useful in dexterous fine interaction, they are not easy to use for large movements. Therefore, we have implemented a keyboard based input through a GUI to modify the desired pose of the ( ), which will autonomously drive the robot to the desired pose. Finally, the GUI is also used to provide the visual feedback from the FLIR AX-8 camera.

Ii-a Master’s Controllers

The master joint torque command is determined by the summation of the admittance controller () and the FIC.


where is the (diagonal) damping gain and is the gravity compensation term and are the joint configurations.

Ii-B Slave’s Controllers

The Slave’s controller is the combination of the P controller, the FIC, Slave’s inverse dynamics compensation and null-space controller


where is the end-effector Jacobian, is the inertia matrix and is the null space torque. is the inverse dynamics compensation. is the task-space inertia matrix and is Coriolis’s matrix.

is the gravity compensation vector in joint-space. The following equation presents the wrench calculation in our proposed bilateral telemanipulation.

Ii-C Slave’s Impedance Controller

The impedance controller law is similar to the FICS control law. However, the fractal attractor and the non-linear impedance profile have been removed. While it has been kept the constant stiffness and the damping.


where is the constant component in .

Iii Experimental Validation

Figure 3: (a) Obj 1: It is the lid of the bigger plastic box in a. (b) Obj 2: Small rock. (c) Obj 3: Kitchen sponge (d) Obj 4: Tissue paper box (e) Obj 5: Metal box.

The comparison between the proposed method and the fractal impedance controller has been conducted from the perspective of the user, as shown in Figure 2. To evaluate transparency we analysed admittance characteristics of the set-up; thus, a spectral analysis has been conducted considering the force signals measured on the slave’s end-effector as input and the velocity produced on the master as output of the system.

The two following experiments were performed:

  1. System impulsive response: Force impulse were applied to the slave’s end-effector using an hammer and the master’s end-effector was not hold.

  2. Interaction effect on system response: The user drove the robot in contact with each of the 5 objects depicted in Figure 3 before releasing his hold on the master’s end-effector. The objects’ dynamic changes from stiff to compliant enabling to evaluate the controllers’ robustness to the coupling effects, which occurs in impedance controllers.

Figure 4: The impulse frequency response shows that the proposed controller and a traditional impedance controller have similar cut off frequencies at about 2Hz, which could have been expected since they shared the same gains. However, the proposed method behaves as a critically damped system, while the traditional impedance controller is under-damped.

The controller gains used in the experiment are reported in Table I. It shall be noted that the gains of the FIC have been used with both controllers as safety mechanism. However, the safety mechanism was never triggered during our experiments.

Master Slave
0 100 100
0 5 5
2.5 2.5 20
0 1.25 1.25
0.075 0.075 NA
1.0472 1.0472 NA
FIC: Fractal Impedance Control, IC: Impedance Control.    Where is the vector of the force saturation boundaries and , described in Appendix A
Table I: Controller Parameters

Iii-a Results

The impulsive response of the two controllers is portrayed in Figure 4. The system cut-off frequency is about 2Hz in both controllers. The FIC has a critically damped behaviour while the regular impedance controller has an under-damped behaviour. Figure Figure 5 also confirms these characteristics. However, we also observe a band-pass behaviour due to the user driving the robot using the master device until the slave’s end-effector makes contact with the environment, which filters the lower frequencies. It can also be observed how the band-pass is wider where the slave is driven using the FIC controller. Figure 6 compares the behaviour of the two controllers when interacting with different objects. The data show that the transient behaviour of the proposed method is more variable, and it has a lower attenuation of the input signal.

Figure 5: The spectral analysis of the teleoperation setup recorded for the interaction with the 5 objects in Figure 3 shows that the proposed method has a broader band-pass behaviour, centred on the 2Hz frequency.
Figure 6: Separating the data in Figure 5 based on the object interaction. (a) Shows the FIC’s higher band-pass is due to its ability to synchronise its behaviour with the environment. (b) Shows the less variable frequency spectrum of the IC.

Iv Discussion

The results show that the proposed method has a cut-off frequency similar to the traditional impedance controller, and an intrinsic adaptability to external environments even without gain adaptation (Figure 6). Nevertheless, the FIC also has online gain adaptation capabilities as reported in [21]. The spectral analysis in Figure 5 shows that the proposed method has a wider band-pass when driven by the user, confirming together with the consistently lower attenuation of the inputs a higher transparency compared to the traditional impedance controller.

To better visualise the benefit of such behaviour during haptic interaction, we tested the two controllers in a sequence of dexterous tasks. These tests are not presented as results because they have been conducted using a single skilled operator which had knowledge of the controller being used. The operator was only allowed a single attempt for each task and the choice of the starting controller was alternated to mitigate the operator learning bias. The chosen tasks are:

  1. Removing the lid from the metal box in e

  2. Pushing a golf ball to an end-goal on a rough terrain (sand, gravel, rocks)

  3. Driving a pile through the sand

  4. Diving the sponge in c through a course

  5. Press an E-stop button on a control panel

The completion times are reported in Table II. The outcome qualitatively shows that the proposed method is more reliable than a traditional impedance controller. The proposed controller succeeded in all the tasks as shown in Figure 7.

a Task i
b Task ii
c Task iii
d Task iv
e Task v
Figure 7: Snapshots of the 5 tasks from the FIC experiment.

Traditional impedance control achieved pushing the golf ball, driving the sponge, and pressing E-stop button. The driving task failed to a poorer movement accuracy, while the lid removal failed due to the Franka’s safety constraints being triggered.

i 34 NA
ii 26 32
iii 22 NA
iv 94 152
v 6 32
Table II: Task Completion Times

The experimental results show that the proposed method has intrinsic adaptability to different environmental dynamics, even without performing an online gain tuning. Such characteristics, based on our preliminary analysis, provide the user with better interaction dexterity. Future work will explore more in-depth the haptics performance of the proposed method. Mainly by taking advantage of the presence of the FIC and the online impedance tuning provided by the FIC framework, [21], to improve the framework dexterity capabilities.

Appendix A Fractal Impedance Control

The fractal impedance control is a passive controller which has been proposed in [21] . It has variable anisotropic state-dependent stiffness, driving the following dynamics at the end-effector as follows:


where is the task-space inertia, is a constant damping, is the anisotropic state-dependant impedance profile, and is the sign of the end-effector velocity which is used to discriminate if the controller is either in convergence or divergence phase.

The stiffness profile is determined as follows:

Divergence Phase ( or ):


Convergence Phase ():


Where and are the constant and the variable stiffness respectively, is the maximum exertable wrench by the robot with respect to its physical limitation, is the stiffness upper-bound. is the maximum displacement from reached during divergence, and is adjustable coefficient that controls the stiffness saturation-boundariess.

The upper-bound level for the controller stiffness and are calculated as follows:


where is the pose error set where the controller wrench saturates to . The FIC is summarised in algorithm 1.

1 if diverging from  then
2       Equation 5
5       Equation 6
7 end if
Algorithm 1 Fractal Impedance Control


This work has been supported by the following grants: EPSRC UK RAI Hub ORCA (EP/R026173/1), the Future AI and Robotics for Space (EP/R026092/1), and THING project in the EU Horizon 2020 (ICT-2017-1).


  • [1] P. Schmaus, D. Leidner, T. Krüger, A. Schiele, B. Pleintinger, R. Bayer, and N. Y. Lii, “Preliminary insights from the meteron supvis justin space-robotics experiment,” IEEE Robotics and Automation Letters, vol. 3, no. 4, pp. 3836–3843, 2018.
  • [2] P. Schmaus, D. Leidner, R. Bayer, B. Pleintinger, T. Krüger, and N. Y. Lii, “Continued advances in supervised autonomy user interface design for meteron supvis justin,” in 2019 IEEE Aerospace Conference.   IEEE, 2019, pp. 1–11.
  • [3] S. Part, “Impedance control: An approach to manipulation,” Journal of dynamic systems, measurement, and control, vol. 107, no. 17, 1985.
  • [4] N. Hogan, “A general actuator model based on nonlinear equivalent networks,” IEEE/ASME Transactions on Mechatronics, vol. 19, no. 6, pp. 1929–1939, 2013.
  • [5] Y. Li, G. Ganesh, N. Jarrassé, S. Haddadin, A. Albu-Schaeffer, and E. Burdet, “Force, impedance, and trajectory learning for contact tooling and haptic identification,” IEEE Transactions on Robotics, vol. 34, no. 5, pp. 1170–1182, 2018.
  • [6]

    J. Buchli, E. Theodorou, F. Stulp, and S. Schaal, “Variable impedance control a reinforcement learning approach,”

    Robotics: Science and Systems VI, pp. 153–160, 2011.
  • [7]

    T. Tsumugiwa, R. Yokogawa, and K. Hara, “Variable impedance control based on estimation of human arm stiffness for human-robot cooperative calligraphic task,” in

    Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No. 02CH37292), vol. 1.   IEEE, 2002, pp. 644–650.
  • [8] A. J. van der Schaft and A. Van Der Schaft, L2-gain and passivity techniques in nonlinear control.   Springer, 2000, vol. 2.
  • [9] J.-H. Ryu, D.-S. Kwon, and B. Hannaford, “Stable teleoperation with time-domain passivity control,” IEEE Transactions on robotics and automation, vol. 20, no. 2, pp. 365–373, 2004.
  • [10] N. Hogan and S. P. Buerger, “Impedance and interaction control,” in Robotics and automation handbook.   CRC press, 2018, pp. 375–398.
  • [11] F. Ferraguti, N. Preda, A. Manurung, M. Bonfe, O. Lambercy, R. Gassert, R. Muradore, P. Fiorini, and C. Secchi, “An energy tank-based interactive control architecture for autonomous and teleoperated robotic surgery,” IEEE Transactions on Robotics, vol. 31, no. 5, pp. 1073–1088, 2015.
  • [12] C. Secchi, S. Stramigioli, and C. Fantuzzi, “Position drift compensation in port-hamiltonian based telemanipulation,” in 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems.   IEEE, 2006, pp. 4211–4216.
  • [13] M. Franken, S. Stramigioli, S. Misra, C. Secchi, and A. Macchelli, “Bilateral telemanipulation with time delays: A two-layer approach combining passivity and transparency,” IEEE transactions on robotics, vol. 27, no. 4, pp. 741–756, 2011.
  • [14] V. Duindam and S. Stramigioli, “Port-based asymptotic curve tracking for mechanical systems,” European Journal of Control, vol. 10, no. 5, pp. 411–420, 2004.
  • [15] T. S. Tadele, T. J. de Vries, and S. Stramigioli, “Combining energy and power based safety metrics in controller design for domestic robots,” in 2014 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2014, pp. 1209–1214.
  • [16] G. Raiola, C. A. Cardenas, T. S. Tadele, T. De Vries, and S. Stramigioli, “Development of a safety-and energy-aware impedance controller for collaborative robots,” IEEE Robotics and automation letters, vol. 3, no. 2, pp. 1237–1244, 2018.
  • [17] K. Kronander and A. Billard, “Stability considerations for variable impedance control,” IEEE Transactions on Robotics, vol. 32, no. 5, pp. 1298–1305, 2016.
  • [18] C. Schindlbeck and S. Haddadin, “Unified passivity-based cartesian force/impedance control for rigid and flexible joint robots via task-energy tanks,” in 2015 IEEE international conference on robotics and automation (ICRA).   IEEE, 2015, pp. 440–447.
  • [19] A. Dietrich, C. Ott, and S. Stramigioli, “Passivation of projection-based null space compliance control via energy tanks,” IEEE Robotics and automation letters, vol. 1, no. 1, pp. 184–191, 2015.
  • [20] F. Ferraguti, C. Secchi, and C. Fantuzzi, “A tank-based approach to impedance control with variable stiffness,” in 2013 IEEE International Conference on Robotics and Automation.   IEEE, 2013, pp. 4948–4953.
  • [21] K. K. Babarahmati, C. Tiseo, J. Smith, H. C. Lin, M. S. Erden, and M. Mistry, “Fractal impedance for passive controllers,” arXiv preprint arXiv:1911.04788, 2019.
  • [22] D. Zhou and K. Tadano, “Relationship between system parameters and operator’s haptic sensation in bilateral controlled master–slave system,” International Journal of Human–Computer Interaction, vol. 36, no. 2, pp. 143–155, 2020.