WiredSwarm: High Resolution Haptic Feedback Provided by a Swarm of Drones to the User's Fingers for VR interaction

11/12/2019 ∙ by Evgeny Tsykunov, et al. ∙ Skoltech 0

We propose a concept of a novel interaction strategy for providing rich haptic feedback in Virtual Reality (VR), when each user's finger is connected to micro-quadrotor with a wire. Described technology represents the first flying wearable haptic interface. The solution potentially is able to deliver high resolution force feedback to each finger during fine motor interaction in VR. The tips of tethers are connected to the centers of quadcopters under their bottom. Therefore, flight stability is increasing and the interaction forces are becoming stronger which allows to use smaller drones.



There are no comments yet.


page 1

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1. Introduction

A noticeable part of human-robot interaction involves the usage of haptic and tactile feedback (Akahane et al., 2013; Tsykunov et al., 2019). Along with that, there are several research projects related to providing encountered-type kinesthetic and tactile feedback in VR via drones.

In (Hoppe et al., 2018), an object or surface is connected to the drone which is supposed to be touched by a human to deliver passive or active tactile feedback. Authors in (Knierim et al., 2017) also propose to hit the user with some object connected to a small drone to provide a haptic sensation. Abdullah et al. in (Abdullah et al., 2018) also proposed to push or pull the drone in direction to simulate force feedback for direct interaction. Abtahi et al. in their paper (Abtahi et al., 2019) developed more complicated scenario which incorporates rich interactions including passive force feedback and texture mapping.

The main limitations of the proposed solutions include low sensation resolution, instability during interaction, low impact force, and big size of a drone. In particular, authors in (Abdullah et al., 2018; Abtahi et al., 2019) selected bigger size drones. Usually spacial motion of human hands is fast. Although more powerful quadrotors could provide more noticeable force feedback, they could be slow for certain application. It is also hard to combine different types of feedback at the same time because there are not enough space for drones near the fingers.

Figure 1. Interaction of a user with a virtual scene using flying wearable haptic interface WiredSwarm.

2. Main contributions

In contrast with the discussed works, we propose to connect multiple micro-quadrotors and each finger with leashes. String is able to deliver a force vector (direction and magnitude) directly from the drone to the finger.

2.1. High sensation resolution

Due to the leash length, the drones are not in the close proximity near the fingers, therefore we can use multiple of them to deliver haptic feedback to the close regions - neighbour fingers (shown in Fig. 1). One option is to use one drone to deliver force to each finger. Another way is to combine five fingers into three groups and connect one drone to each group.

Two drones with different tether properties can be connected to the same finger. Combination of different types of leash (e.g., elastic or non-stretchable) could deliver a rich kinaesthetic feedback.

Using more that one drone allows to create high resolution encountered-type haptic feedback. In this case each finger has to be tracked by a localization system, such as Leap Motion.

2.2. Flight stability

During the interaction session the drones are not visible in VR, which rises additional safety challenges. Usual physical interaction strategy involves direct contact between the fingers and some part of the drone. The point of contact is far from the center of the drone (Hoppe et al., 2018; Abtahi et al., 2019)

, which generates high and sudden external moment. Such a high moment leads to the instabilities which are hard to overcome by a flight controller.

In the proposed solution tether is connected to the center of the drone under the bottom providing higher flight stability. The suggested design implies the selection of leash properties. One option is to make an elastic leash, which ensures smooth interaction and more safe flight.

2.3. Higher impact force

Each rotor, rotating at speed produces a force and a momentum which are defined as:


where is the force coefficient, is the momentum coefficient, for the quadrotor (Mahony et al., 2012). The resulting control inputs to the drone frame are expressed as:


The coordinate systems of the drone and world frame are defined as and , respectively. The majority of conducted research (Hoppe et al., 2018; Abtahi et al., 2019; Knierim et al., 2017) utilize force that is produced by the rotors in or direction. However, acts along axis. During the interaction with human, drones usually avoid highly aggressive maneuvers with high acceleration, pitch, and roll angles, which produces small inclination. Force calculated as has a small horizontal term (along and axis), and, therefore, the impact with human is negligible.

Authors in (Abtahi et al., 2019) also partially employ the moments and for the surface stiffness simulation. defined as:


It is possible to see from (3) that the potential impact from moment (combination of and ) is limited, comparing to . Only up to two rotors actually contribute to the force feedback. In addition, which requires a lot of power, suffers from highlighting and .

We propose the concept in which the tether is connected to the bottom of the quadrotor and the main force direction is along axis. Therefore, the most powerful control input is used to deliver the force feedback while the drone is in well-stabilized position.

3. System implementation

We used Crazyflie 2.0 quadrotors and Robot Operating System (ROS) environment for the drone control. The state of the human hand and drones (position and orientation) are estimated with a Vicon motion capture system (12 Vantage V5 cameras are covering

space). The position and attitude update rate was 100 Hz for all drones. For easy setup, we propose to connect the tethers to the fingers with magnets.

Drones follow the human hand while the user is moving in the virtual scene. We propose to predict the intersection between the human hand and the virtual object and the corresponding drones become activated and provide fine force feedback.

4. Conclusion and Future Work

The novel type of device, flying wearable glove, was developed by us. This type of the haptic interface considerably increases the working area comparing with the desktop type devices. Additionally, it does not produce fatigue to the user, as it is often the case with wearable devices, because of zero weight.

Specifically, the proposed technology provides strong advantages for rendering the interaction with horizontal virtual surfaces. For example, playing virtual musical instruments, e.g., virtual piano can be a scenario for its application. Moreover, application of WiredSwarm in VR surgical simulator will allow the user to feel the interaction with tissue, muscles, and bones.

WiredSwarm technology can be extended to deliver haptic sensation not only to the fingers but also to forearms and upper arms. To achieve rendering of textures of virtual objects it is reasonable to integrate vibromotors at the lower tip of the strings. In the future work we will add experimental data and theoretical analysis on the stability, resolution of force feedback, validation in a VR scenario.


  • M. Abdullah, M. Kim, W. Hassan, Y. Kuroda, and S. Jeon (2018) HapticDrone: an encountered-type kinesthetic haptic interface with controllable force feedback: example of stiffness and weight rendering. In 2018 IEEE Haptics Symposium (HAPTICS), External Links: Document Cited by: §1, §1.
  • P. Abtahi, B. Landry, J. (. Yang, M. Pavone, S. Follmer, and J. A. Landay (2019) Beyond the force: using quadcopters to appropriate objects and the environment for haptics in virtual reality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI '19, External Links: Document Cited by: §1, §1, §2.2, §2.3, §2.3.
  • K. Akahane, J. Hyun, I. Kumazawa, and M. Sato (2013) Two-handed multi-finger string-based haptic interface SPIDAR-8. In Springer Series on Touch and Haptic Systems, pp. 109–147. External Links: Document Cited by: §1.
  • M. Hoppe, P. Knierim, T. Kosch, M. Funk, L. Futami, S. Schneegass, N. Henze, A. Schmidt, and T. Machulla (2018) VRHapticDrones. In Proceedings of the 17th International Conference on Mobile and Ubiquitous Multimedia - MUM 2018, External Links: Document Cited by: §1, §2.2, §2.3.
  • P. Knierim, T. Kosch, V. Schwind, M. Funk, F. Kiss, S. Schneegass, and N. Henze (2017) Tactile drones - providing immersive tactile feedback in virtual reality through quadcopters. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems - CHI EA '17, External Links: Document Cited by: §1, §2.3.
  • R. Mahony, V. Kumar, and P. Corke (2012) Multirotor aerial vehicles: modeling, estimation, and control of quadrotor. IEEE Robotics Automation Magazine 19 (3), pp. 20–32. External Links: Document, ISSN 1070-9932 Cited by: §2.3.
  • E. Tsykunov, R. Agishev, R. Ibrahimov, L. Labazanova, A. Tleugazy, and D. Tsetserukou (2019) SwarmTouch: guiding a swarm of micro-quadrotors with impedance control using a wearable tactile interface. IEEE Transactions on Haptics 12 (3), pp. 363–374. External Links: Document Cited by: §1.