I Human-Swarm Interaction: the emergent field
With a strong initial influence from nature and bio-inspired models [31, 6], swarm systems are known for their adaptability to different environments  and tasks . As a result, swarm robotics research has recently been gaining popularity –- Fig. 1111Information retrieved from Scopus research database. –-. As the cost of robotic platforms continues to decrease, the number of applications involving multiple robots is increasing. These include targeted material transportation , where groups of small robots are used to carry tall, and potentially heavy, objects; precision farming [13, 1], where a fleet of autonomous agents shifts operator activities in agricultural tasks; and even entertainment systems , where multiple robots come together to form interactive displays.
The efficiency of performing tasks with robotic teams depends on two main factors: the level of robot autonomy, and the ability of human operators to command and control the team of robots. Regarding the latter, the transition from current application scenarios where several human operators control a single robot  to environments where a single human control multiple robots, has been identified as one of the main challenges in robotics research [15, 7, 17].
One of the clearest examples of this necessity is when the task conducted by the team of robots becomes extremely complex and begins to require high-level, cognitive-based decisions inline (e.g., exploration of dynamic, unstructured, and unpredictable environments for search and rescue applications). When a robot swarm needs to react to or quickly respond to an abrupt event (e.g., a fast stop), the absence of human intervention can even lead to complete mission failure. In these situations, full autonomy is still far from being reached by robot units alone, and human intervention is necessary for adequate performance.
However, the ability to command a swarm of robots requires a significant cognitive effort from human operators. Previous works [17, 18], have emphasized the complexity of these tasks and have compared them to computational complexity (). Likewise, swarm operators traditionally perform a repetitive sequence of steps to enable the system (i.e., the robot swarm) to fulfill an objective, or reach some desired goal state . Normally, these sequences of steps become more complex as the operator has to share his/her cognitive resources among a higher number of robots .
Under this framework, different command and control operations involving robot swarms can have different levels of complexity (Fig. 2). For instance, control modes, such as the leader-follower approach  where the number of possible actions () is independent of the number of robots, can represent a relatively low-level of complexity (ideally ) for human operators under their cognitive limit. In contrast, if several robots are performing independent tasks, the complexity level might increase linearly as new robots and tasks are included into the swarm (), eventually surpassing the cognitive abilities of the operator and making the operation of the swarm unsustainable. Moreover, task scenarios where robots need to tightly coordinate (e.g., transporting objects with deformable  shapes) are considered to have an exponential complexity level () due to the inter-dependencies between robots, making the operation of such group of robots even harder.
The primary purpose of this cognitive complexity framework was to emphasize the effort of human operators required to control a swarm robotics system, and the basic need of creating tools and techniques that allow operators to control higher number of robots without reaching their cognitive limits.
Human-Swarm Interaction (HSI) is a prominent research field –- Fig. 311footnotemark: 1 –- that aims to allow a human operator to be aware of certain swarm-level information that he/she can use to make decisions regarding the swarm behavior. However, this is a complicated process since some kind of mechanism is needed to bridge the information gap between the human operator and the robot swarm. Human supervision normally relies on global goals such as mission statements or route planning. In contrast, simple robot units are usually hardware-limited and can access only to local information.
The design of interfaces that allow operators to control a swarm of robots is receiving increasing research attention [25, 29, 14]. Several well-known technologies – including vision-based systems, haptic devices and electromyographical (EMG) receptors – have been proposed. However, seamless interaction between operators and robot swarms has not yet been achieved, not only due to the complexities of translating numerous local information streams (i.e., the robot swarm) to a unified global input scheme (i.e., the human operator), but also due to the complex infrastructure settings of existing interfaces such as vision-based sensors or global positioning systems, which only work in controlled environments, and a lack of appropriate feedback that can guide the operator and provide accurate information about the swarm’s state. These obstacles notwithstanding, a general-purpose human-swarm interface is required to tackle the next wave of challenges facing industry and advance the technology to a new state of the art.
In the following, I will discuss two promising technologies that, if combined, could support the development of a general purpose HSI interface to control robotic swarms in an efficient and natural manner.
I-a Gesture Recognition: a versatile high-level input mechanism
Gestures and body movements are a natural way to communicate intentions and strengthen messages. Gestures are part of our social communication skill set , which humans can use, understand and analyze. Hand gestures were very early adopted in research on human-robot interaction . However, it took more than 20 years to utilize them in HSI to convey an operator’s intentions to small swarms of robots .
Recently, gesture-based HSI has evolved with the development of rich gesture taxonomies –- e.g., Fig. 4 –-, which operators employ to control a group of robots. These taxonomies have mainly focused on remote interaction (i.e., tele-operation) applications conducted in controlled indoor environments [2, 29]. Despite the high correct classification rates (CCRs) and solid conceptual foundation for future research achieved by these works, their models are difficult to use in other experimental settings as they require complex infrastructure such as vision-based sensors and global positioning systems –- Fig. 5 –- or specialized hardware –- Fig. 6 –-.
A different approach was proposed in , in which robots had to distinguish in a distributed fashion the orders and commands provided by the operator. This method was designed to enable the operator to interact with the swarm in a proximity environment –- Fig. 7
–-, making the human operator a ‘special’ swarm member. However, the robots required a direct line of sight to the operator in order to detect and classify the operator’s gestures. A consensus mechanism was then used within the robot swarm to reach an agreement about the operator’s intentions. Its lack of complex infrastructure makes this method suitable for a wider range of scenarios, such as on-the-spot progress checks of a swarm’s operation, as well as in outdoor environments. However, the approach is nonetheless hamstrung by the limited sensing and computational power of individual robots, and so it is unclear if or how it could be applied to large swarms.
Even though the above-described remote and proximity interaction approaches are promising steps towards achieving suitable gesture-based control methods for specific applications, they have key limitations. Methods proposed for remote interaction rely on complex infrastructure, while proximity interaction methods suffer from scalability issues. Despite these problems, the aforementioned works prove that gestures can be a feasible way to control a swarm of robots in both remote and proximity interaction scenarios. Given enough flexibility, they may be used in a more general interface that could be suitable in both application scenarios in the future.
I-B Haptic Feedback: augmented assistance for the operator
Another popular approach to combine robot swarms with human input has been to explore the haptic channel. Haptic technology provides a way in which information related to swarm status can be transferred back to the operator via tactile or force feedback.
Haptic devices such as Phantom Omni – Fig. 8 – or Omega3 – Fig. 9 – have been extensively used to orchestrate the movements of whole swarms of robots , certain subgroups of the swarm  or robot teams using a leader-follower approach .
In previous research, haptic feedback has been used in combination with existing methods such as continuous visual input to assist a human operator . Haptic information has proven useful in guiding the operator in situations where a robot swarm is operating in obstacle-populated environments  or unstructured areas . In such scenarios, attraction/repulsion forces are calculated according to environmental obstacles or swarm members’ positions, and transferred back to the operator to assist his/her decisions.
Even though current haptic devices provide a way to receive feedback, they suffer from several limitations. Such devices rely only on Cartesian input information (trajectories, vectors, etc.) but cannot process high-level commands such as those provided using gesture-based interaction. Further, they unify input and feedback components, which could confuse the operator in situations where input and elaborate haptic feedback signals occur simultaneously. For instance, the operator might need feedback about the energy status of a robot at the same time as he/she is trying to accurately guide it through an obstacle-free path.
Another drawback of devices such as the Omega3 and Phantom Omni is that they represent a single point of failure in case of malfunction, which might be a problem in terms of building a general-purpose interface. Finally, one of their most significant constraints is that they are limited to indoor environments equipped with a computer terminal, thus excluding missions in outdoor or other environments.
Ii A wearable gesture-haptic interface for human-swarm interaction
The gaming and wearable technology industries have been a good source of breakthroughs and disruptive devices not only for commerce, but for academic research as well. Devices such as the Microsoft Kinect or Oculus Rift, initially designed as interactive controllers for common console platforms and game engines, have been extensively used within the robotics community to assist key research activities. In the last few years, we have observed how the second generation of wearable and gaming devices has reached the market with a special focus on haptic and monitoring capabilities. As the cost of these new devices decreases and their technology starts to provide enhanced features, novel application domains open up for their use in robotics research.
Gesture recognition devices such as the Myo armband – Fig. 10 – or Leap motion – Fig. 11 – demonstrate the feasibility of transferring gestures and simple commands to a computer system. Their low cost and suitability to both indoor and outdoor environments make them interesting candidates for the gesture recognition component of a general-purpose interface.
Myo is a wearable armband device that can recognize a rich set of gestures –- Fig. 12 -– by using 8 EMG muscle sensors installed in its frame. Its built-in accelerometers and gyroscopes also allow Myo to detect arm motion accurately. Several vibration motors give the additional possibility of providing haptic feedback in the form of short, medium and long vibrations. Myo is equipped with a rechargeable lithium-ion battery which is designed to last a day of continuous use. Finally, its Bluetooth connectivity allows Myo to transmit data to a computer or external device.
Leap Motion is a portable 3D motion sensor that is able to detect hand movements as well as finger positions in a large interaction space (eight cubic feet) – Fig. 13 –. The core of the device consists of two cameras that track the light reflected by three built-in infrared LEDs. Leap Motion has been used in academic research focused on fingers and their movements, such as on sign language recognition .
Leap Motion is a great complement to Myo since they each detect different types of information. Myo can recognize high-level intentions (gestures) whereas Leap Motion can recognize Cartesian information such as paths or routes, as well as deictic information like numbers through finger patterns. The possibility of combining these types of information allows for the creation of a wide range of control modes. For instance, high-level gestures could trigger ‘actions’ such as “follow path” or “change parameter value”, while deictic patterns could provide detailed information about the shape of the path to follow or the value of the parameter to change.
Advanced haptic devices have been recently introduced into the gaming industry to increase the immersion experience while playing video games. Gaming vests such as KOR-FX – Fig. 14 – or 3 Space – Fig. 15 – allow video game players to deeply engage with a game’s current events by producing vibrations (KOR-FX) or pneumatic pulses (3
Space) according to the in-game situation. Users of these devices obtain rich information such as direction, intensity, and rhythm. Transforming swarm-centric data such as force fields and obstacle/landmark positions into these modalities would give the human operator a better understanding of the state of a robot swarm in the field. Brooks changed the course of artificial intelligence (AI) by arguing that the world is its own best model. In a similar fashion, we now argue that the best-placed entity to obtain feedback about a swarm’s ‘body’ is the operator’s body.
Example of current use
|Wearable gesture recognition device (e.g., Myo)||
|Portable deictic-recognition device (e.g., Leap Motion)||
|Wearable advanced haptic device (e.g., 3 Space, KOR-FX)||
Table I outlines the main benefits, limitations, and mitigation strategies for all devices proposed before. First, Myo provides wireless gesture recognition without complex infrastructure settings. However, its gesture range is limited to the ones depicted in Fig. 12. Even though this does not represent a problem at the current, proof–of-concept stage, advanced users could extend the repertoire of recognizable gestures as their applications require. In that case, custom gestures could be played, recorded, and stored by using the device’s built-in EMG monitoring tool.
Second, Leap Motion offers high precision joint tracking for both hands. Even though Leap Motion has a small size and large interaction space, it requires a USB connection to operate. The inclusion of a Single Board Computer (SBC) such as Raspberry Pi222http://www.raspberrypi.org/products/raspberry-pi-3-model-b/ in the interface configuration could solve this problem, as well as provide additional computing and communication capabilities to the whole interface. In addition, the Leap Motion sensor needs to be located such that it can clearly sense the operator’s hands. To this end, a support frame could be attached to the haptic vest to place it in a suitable location. 3D-printing this support frame would ensure precise conformance with the vest’s dimensions.
Third, the 3 Space gaming vest provides a large haptic interaction body area (both frontal and back parts) and a solid software development kit (SDK) with which to build applications. However, device operation requires a USB connection as well as a portable air compressor. The former requirement could be solved by the inclusion of a mini PC (in the same way as described before) in the wearable interface; the latter could be solved by the inclusion of a small pocket in the vest to carry the air compressor, as well as a Li-ion rechargeable battery that could power the whole system.
In a nutshell, the combination of a wearable gesture recognition device that can detect high-level intentions, a portable device that can detect Cartesian information and finger movements, and a wearable advanced haptic device that can provide real-time feedback is a promising scheme for a general-purpose wearable interface for HSI applications. As far as this author knows, this work is the first to envision a wearable HSI interface that separates the input and feedback components of the classical control loop (input, output, feedback). Moreover, the proposed interface is suitable for both indoor and outdoor environments. In addition, such an enhanced interface might be able to provide other advanced interaction and interesting control capabilities: some are described below.
Ii-a Enhanced haptic feedback
Previous swarm robotics research involving haptic feedback has only explored the use of traditional static devices normally attached to a desktop computer terminal [28, 27]. This work is the first to suggest the use of wearable technology in a wider range of scenarios to allow the operator to obtain richer feedback. For instance, classical force feedback could be conveyed to the operator using this wearable technology and, therefore, increase the immersion experienced by the operator. Moreover, the operator might obtain additional information using this technology by utilizing different pulse or vibration patterns (e.g., heartbeat-like pulse patterns could serve to communicate the battery status of swarm robots). Finally, by decoupling the feedback and input components of the interface, a more robust and fault-tolerant interface can be achieved.
Ii-B Timing based input
Recent research [21, 20] has demonstrates that improper timing of control input by operators can lead to problems when commanding a swarm of robots, such as group fragmentation (i.e. unintentional division of the swarm). Group fragmentation causes delays in coordination, as well as motion and sensing errors that hurt the performance of swarm tasks. Operators who issue commands frequently showed higher levels of swarm fragmentation than those who allowed the swarm to adjust between new commands. Optimal timing studies are just beginning to emerge, and they are an interesting area for future research.
An interface that could determine optimal human timing as well as provide guidance and assistance could be crucial to achieving effective interaction between human operators and robot swarms. The proposed interface outlined in this work offers a suitable platform to conduct research on optimal human timing algorithms since it has all the necessary components to develop effective models. First, an embedded computing unit (e.g., Raspberry Pi) gathers input data from the robotic swarm and calculates proper timing threshold parameters. Second, an advanced haptic component (e.g., 3 Space Gaming Vest or KOR-FX) provides intuitive patterns based on previously calculated timing parameters to the operator to assist with his/her decision-making. Finally, high-level gesture (e.g., Myo) and deictic recognition (e.g., Leap Motion) components send commands to the robot swarm after the feedback is taken into account.
Ii-C Hierarchical control
Several recent surveys [17, 7] have pointed out that one of the main problems in the swarm robotics field is that robotic swarms cannot switch between different behaviors during the same mission at present.
Due to the loosely-coupled settings of an interface composed of different wearable parts, it may be possible to create a taxonomy of commands suitable for a wide range of robot behaviors. For instance, high-level commands such as gestures could serve as a switch mechanism between different robot behaviors, while deictic movements could command the swarm within that specific mode of operation
Ii-D Simple interface
Early studies [24, 32] in the field of swarm visualization and representation indicated that simplifying the large state of a swarm to a lower-dimensional representation can be beneficial when controlling a group of robots. Reducing the amount of noise as well as fusing information to simplify the problem of determining a swarm’s state are further promising topics in the HSI field.
The proposed interface outlined in this paper allows the possibility of mapping the state of the robot swarm to an operator’s body through haptic feedback. This capability could dramatically increase the amount of status information available without increasing the complexity of its representation.
Robotic swarms are expected to become an integral part of emerging technologies and open the door to future economic possibilities. However, the lack of a general purpose human-swarm interface that provides seamless interaction between a human operator and a group of robots confines the field to academic laboratories.
Recent advances in gaming technology have brought sophisticated devices that, if combined, could further advance the HSI discipline. Wearable gesture recognition devices recognize and interpret gestures in a wide range of scenarios. In addition, they provide the mobility required for an operator to command a group of robots in both remote and proximity interaction scenarios. Also, new haptic devices such as gaming vests provide a means for an operator to receive haptic feedback without interfering with his/her input signal. At the same time, they constitute a novel platform to obtain rich feedback without increasing the complexity of the swarm’s status information.
The aim of this work is to incorporate the underlying principles of these two novel technologies into a general-purpose HSI interface that is able to control adaptive robotic swarms, which can be controlled in a natural and seamless manner by human operators in order to tackle complex tasks. Potential applications, such as remote and proximity interaction with swarms of unmanned aerial vehicles (UAVs), could be achieved without complex calibration and infrastructure settings. Finally, outdoor applications (e.g. agricultural tasks) could greatly benefit from the proposed interface.
-  Swarm farm : Robotic agriculture. http://www.swarmfarm.com/. Accessed: 2015-11-3.
-  J. Alonso-Mora, S. Haegeli Lohaus, P Leemann, R Siegwart, and P Beardsley. Gesture based human - Multi-robot swarm interaction and its application to an interactive display. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 5948–5953. IEEE, may 2015.
-  J. Alonso-Mora, R. Knepper, R. Siegwart, and D. Rus. Local motion planning for collaborative multi-robot manipulation of deformable objects. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 5495–5502, May 2015.
-  Javier Alonso-Mora, Roland Siegwart, and Paul Beardsley. Human - robot swarm interaction for entertainment: From animation display to gesture based control. In Proceedings of the 2014 ACM/IEEE International Conference on Human-robot Interaction, HRI ’14, pages 98–98, New York, NY, USA, 2014. ACM.
-  Carlos Bentes and Osamu Saotome. Dynamic Swarm Formation with Potential Fields and A* Path Planning in 3D Environment. In 2012 Brazilian Robotics Symposium and Latin American Robotics Symposium, pages 74–78. IEEE, October 2012.
-  Eric Bonabeau, Marco Dorigo, and Guy Theraulaz. Swarm Intelligence: From Natural to Artificial Systems. 1999.
-  Manuele Brambilla, Eliseo Ferrante, Mauro Birattari, and Marco Dorigo. Swarm robotics: a review from the swarm engineering perspective. Swarm Intelligence, 7(1):1–41, jan 2013.
-  Rodney A. Brooks. Elephants don’t play chess. Robot. Auton. Syst., 6(1-2):3–15, June 1990.
-  Jianing Chen, M. Gauci, and R. Gross. A strategy for transporting tall objects with a swarm of miniature mobile robots. In Robotics and Automation (ICRA), 2013 IEEE International Conference on, pages 863–869, May 2013.
-  S. Y. Chien, M. Lewis, S. Mehrotra, N. Brooks, and K. Sycara. Scheduling operator attention for multi-robot control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 473–479, Oct 2012.
-  Ching-Hua Chuan, E. Regina, and C. Guardino. American sign language recognition using leap motion sensor. In Machine Learning and Applications (ICMLA), 2014 13th International Conference on, pages 541–544, Dec 2014.
-  J. W. Crandall, M. A. Goodrich, D. R. Olsen, and C. W. Nielsen. Validating human-robot interaction schemes in multitasking environments. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans, 35(4):438–449, July 2005.
-  Luis Emmi, Mariano Gonzalez-de Soto, Gonzalo Pajares, and Pablo Gonzalez-de Santos. New Trends in Robotics for Agriculture: Integration and Assessment of a Real Fleet of Robots. The Scientific World Journal, 2014:1–21, 2014.
-  E.C. Haas, K. Pillalamarri, C.C. Stachowiak, and M. Fields. Multimodal controls for soldier/swarm interaction. In RO-MAN, 2011 IEEE, pages 223–228, July 2011.
-  Sean T Hayes and Julie A Adams. Human-swarm Interaction: Sources of Uncertainty. In Proceedings of the 2014 ACM/IEEE International Conference on Human-robot Interaction, HRI ’14, pages 170–171, New York, NY, USA, 2014. ACM.
-  Ayoung Hong, Heinrich H. Bulthoff, and Hyoung Il Son. A visual and force feedback for multi-robot teleoperation in outdoor environments: A preliminary result. 2013 IEEE International Conference on Robotics and Automation, pages 1471–1478, 2013.
-  Andreas Kolling, Phillip Walker, Nilanjan Chakraborty, Katia Sycara, and Michael Lewis. Human Interaction With Robot Swarms: A Survey. IEEE Transactions on Human-Machine Systems, pages 1–18, 2015.
-  Michael Lewis. Human interaction with multiple remote robots. Reviews of Human Factors and Ergonomics, 9(1):131–174, 2013.
-  Peter Mundy, Christine Delgado, Jessica Block, Meg Venezia, Anne Hogan, and Jeffrey Seibert. Early social communication scales (escs). Coral Gables, FL: University of Miami, 2003.
-  Sasanka Nagavalli, Shih-Yi Chien, Michael Lewis, Nilanjan Chakraborty, and Katia Sycara. Bounds of neglect benevolence in input timing for human interaction with robotic swarms. In Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction, HRI ’15, pages 197–204, New York, NY, USA, 2015. ACM.
-  Sasanka Nagavalli, Lingzhi Luo, Nilanjan Chakraborty, and Katia Sycara. Neglect benevolence in human control of robotic swarms. In International Conference on Robotics and Automation (ICRA), Hong Kong, China, June 2014.
-  J Nagi, H Ngo, L M Gambardella, and Gianni A. Di Caro. Wisdom of the swarm for cooperative decision-making in human-swarm interaction. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 1802–1808. IEEE, may 2015.
-  Jawad Nagi, Hung Quoc Ngo, Alessandro Giusti, Luca Maria Gambardella, Jürgen Schmidhuber, and Gianni A. Di Caro. Incremental learning using partial feedback for gesture-based human-swarm interaction. In The 21st IEEE International Symposium on Robot and Human Interactive Communication, IEEE RO-MAN 2012, Paris, France, September 9-13, 2012, pages 898–905, 2012.
-  S. Nunnally, P. Walker, A. Kolling, N. Chakraborty, M. Lewis, K. Sycara, and M. Goodrich. Human influence of robotic swarms with bandwidth and localization issues. In Systems, Man, and Cybernetics (SMC), 2012 IEEE International Conference on, pages 333–338, Oct 2012.
-  Steven Nunnally, Phillip Walker, Nilanjan Chakraborty, Michael Lewis, and Katia Sycara. Using Coverage for Measuring the Effect of Haptic Feedback in Human Robotic Swarm Interaction. 2013 IEEE International Conference on Systems, Man, and Cybernetics, pages 516–521, 2013.
-  Steven Nunnally, Phillip Walker, Mike Lewis, Nilanjan Chakraborty, and Katia Sycara. Using haptic feedback in human robotic swarms interaction. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, volume 57, pages 1047–1051. SAGE Publications, 2013.
-  T. Setter, H. Kawashima, and M. Egerstedt. Team-level properties for haptic human-swarm interactions. In 2015 American Control Conference (ACC), pages 453–458, July 2015.
-  Tina Setter, Alex Fouraker, Hiroaki Kawashima, and Magnus Egerstedt. Haptic Interactions With Multi-Robot Swarms Using Manipulability. Journal of Human-Robot Interaction, 4(1):60, 2015.
-  Adrian Stoica, Theodoros Theodoridis, Huosheng Hu, Klaus McDonald-Maier, and David F Barrero. Towards human-friendly efficient control of multi-robot teams. In 2013 International Conference on Collaboration Technologies and Systems (CTS), pages 226–231. IEEE, may 2013.
-  A. Torige and T. Kono. Human-interface by recognition of human gesture with image processing-recognition of gesture to specify moving direction. In Robot and Human Communication, 1992. Proceedings., IEEE International Workshop on, pages 105–110, Sep 1992.
-  J H Walker and M S Wilson. Task allocation for robots using inspiration from hormones. Adaptive Behavior, 19(3):208–224, 2011.
-  P. Walker, S. Nunnally, M. Lewis, A. Kolling, N. Chakraborty, and K. Sycara. Neglect benevolence in human control of swarms in the presence of latency. In Systems, Man, and Cybernetics (SMC), 2012 IEEE International Conference on, pages 3009–3014, Oct 2012.
-  Holly A. Yanco, Adam Norton, Willard Ober, David Shane, Anna Skinner, and Jack Vice. Analysis of human-robot interaction at the darpa robotics challenge trials. Journal of Field Robotics, 32(3):420–444, 2015.