Adaptive Workload Allocation for Multi-human Multi-robot Teams for Independent and Homogeneous Tasks

07/27/2020 ∙ by Tamzidul Mina, et al. ∙ Purdue University 0

Multi-human multi-robot (MH-MR) systems have the ability to combine the potential advantages of robotic systems with those of having humans in the loop. Robotic systems contribute precision performance and long operation on repetitive tasks without tiring, while humans in the loop improve situational awareness and enhance decision-making abilities. A system's ability to adapt allocated workload to changing conditions and the performance of each individual (human and robot) during the mission is vital to maintaining overall system performance. Previous works from literature including market-based and optimization approaches have attempted to address the task/workload allocation problem with focus on maximizing the system output without regarding individual agent conditions, lacking in real-time processing and have mostly focused exclusively on multi-robot systems. Given the variety of possible combination of teams (autonomous robots and human-operated robots: any number of human operators operating any number of robots at a time) and the operational scale of MH-MR systems, development of a generalized framework of workload allocation has been a particularly challenging task. In this paper, we present such a framework for independent homogeneous missions, capable of adaptively allocating the system workload in relation to health conditions and work performances of human-operated and autonomous robots in real-time. The framework consists of removable modular function blocks ensuring its applicability to different MH-MR scenarios. A new workload transition function block ensures smooth transition without the workload change having adverse effects on individual agents. The effectiveness and scalability of the system's workload adaptability is validated by experiments applying the proposed framework in a MH-MR patrolling scenario with changing human and robot condition, and failing robots.



There are no comments yet.


page 1

page 3

page 6

page 8

page 10

page 14

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Multi-human multi-robot (MH-MR) systems have an immense potential for applicability in various independent and non-sequential tasks such as coverage problems of surveillance, patrolling, search and rescue, inspection or assembly of items in an industrial conveyor belt by robotic manipulators, and various other multi-agent scenarios. Robots allow long operation hours on repetitive tasks and provide consistent and precise performance beyond human capability, while human operators contribute improved situational awareness, experienced and intuitive decision making, and the ability to work around unexpected situations. While research on human-robot interaction has gained a lot of momentum in recent years [23, 61, 56], MH-MR systems are a relatively new area involving interaction and collaboration between multiple humans and robots.

Fig. 1: Conceptual illustration of the proposed multi-human multi-mobile-robot (MH-MR) system with adaptive workload allocation to human and robot conditions and performance with workload transitional considerations. Potential application includes autonomous and multi-human operated multi-mobile robot patrolling, surveillance, multi-robot manipulator tasks on a moving conveyor belt in an industrial setting etc. The dynamically allocated workspace in the different applications change with human and robot operator condition and performance in real-time.

Task/workload allocation is an important problem in MH-MR systems. Previous works have investigated team organization [31], a number of operator-mediated robot control methods [5], awareness studies in human-robot systems [9], and various classifications of human-robot systems [67] for task/workload allocation. Tsarouchi et al. introduced a system for designing and assigning tasks to operators and human workplaces [60]. Automation adaptation based on human perceived workload has also been studied in [21]. Physiological measurements of humans have been used as triggers in the control of unmanned aerial vehicles (UAVs) to initiate different workload states and adapt operator performance [50, 65]. Error rates and task difficulty as perceived by operators have also been used as triggers to re-allocate or automate workload [49]. However, task allocation in a multi-agent system increases in complexity if the triggers are less than perfect; sudden or unpredictable changes in workload or mission may have a negative impact on the operator’s performance pertaining to the understanding of the automation behaviors and the system functions they control [39]. Sudden and/or drastic changes may overwhelm or momentarily catch human operators off guard while trying to cope with their allocated work.

Musić and Hirche have proposed an architecture for planning human roles in robot team control [44] to optimize collaboration and teaming mechanisms across a wide range of human operators and robots. Task allocation in multi-robot scenarios have been widely studied in [30, 54, 26, 3, 46], considering resource constraints and robot performance. Task allocation with unknown robot capabilities have also been studied in [10]. Optimal task allocation with multi-humans in the loop has also been proposed in [36], where task allocation is performed over multiple-levels (group and individual) comprising of high-risk and low-risk information in order to maximize effectiveness of the entire system minimizing processing cost and time, considering human factors given limited resources. In market-based approaches for multi-agent task allocation, the team seeks to optimize an objective function based upon robots utilities for performing particular tasks [59, 70]; desirable features of these approaches include efficiency in satisfying the objective function, robustness and scalability of the system. However, in systems where fully centralized approaches are feasible, market-based approaches can be more complex to implement and can produce poorer solutions; when fully distributed approaches suffice, market-approaches can be unnecessarily complex in design and can require excessive communication and computation [8]

. Mixed integer linear programming optimization approaches have also been used for task allocation

[1, 7, 40]

. Population based approaches such as the genetic algorithm was also proposed for task allocation in disaster scenarios

[19]. Ant colony optimization has also been proposed for task allocation of multi-agent systems in [63].

Most of the task/workload allocation methods proposed in literature have focused on maximizing the system’s work output without considering individual agent conditions. Moreover, most of the research work on task allocation has remained confined to multi-robot systems only. In contrast, we present a task/workload allocation method considering quantified both human and robot condition and performance equally, i.e. prioritizing the ability of all agents to work in an MH-MR system maintaining agent level work efficiency, while ensuring full coverage of the application workspace.

In this paper, we present a generalized MH-MR framework capable of workload allocation for independent, non-sequential homogeneous tasks, consisting of independent modular function blocks assessing human and robot conditions and the performances of human-operated and autonomous robots. The system is designed to be compatible with previously established normalized quantitative human and/or robot health and performance assessment tests. The framework also incorporates a re-allocated workload transition model to minimize the effects of sudden changes in workload or mission that may have negative impacts on operator or robot performance. We demonstrate the applicability, effectiveness, and scalability of the framework through various scenarios of a MH-MR patrolling application as validation of the proposed concept. An overview of our MH-MR work allocation concept applied in example scenarios of mobile robot coverage problems and robotic manipulators in an assembly line conveyor belt is shown in Fig. 1.

Ii Preliminaries and Assumptions

We consider a homogeneous group of robots capable of carrying out autonomous missions, each denoted as , for with state definition of , where represents the dimension of the system workspace. may be teleoperated by any number of human operators at any time, each denoted as for , modeled as an edge in an undirected graph without any self-connectivity, where represents the set of nodes , such that . We denote the set of indices of human-operated robots as and the set of indices for autonomous robots in the system as . For the convenience of the reader, we summarize the terminology usage in this section as: for robots, for human Operators, and for condition and performance (human and robot).

Each human operator may control multiple robots and assume/relinquish control of any robot in the system at any time, triggering a change in the robot operation mode. We do not limit robots operated by humans to be only teleoperated; some level of autonomy might exist while the human operator acts as a supervisor. Regardless, the performance of such a robot is dependent on the state of the human operator as well.

The condition or health status of each robot , for in the MH-MR system can be monitored at all times as a set of robot health states denoted as , where equals the number of robots in the system . Physiological measurements and the emotional state of each human operator , for in the MH-MR system can be monitored at all times as a set of human operator health states denoted as , where is the number of human operators in the system .

Definition II.1.

The performance of each robot (autonomous and human-operated)

on their respective allotted mission/task can be evaluated based on a predefined evaluation metric relevant to the mission/task using observation set

, .

Fig. 2: Adaptive MH-MR workload allocation system modular framework based on individual human and robot condition and performance. The MH-MR system may consists of autonomous robots and combinations of single human-operated single robot, multi-human operated single robots and/or single human operated multi-robots. The workload allocation module takes equal weighted metric inputs from each modular condition and performance evaluation module to allocate new workload. The allocation transition module ensures a smooth transition to the new workload.

We define the constraints of the system workspace (workload) as finite, known apriori, and covered by robots without any overlap. We assume each robot is equipped with appropriate low level velocity/position controllers with collision avoidance relevant to the MH-MR application and is capable of fully autonomous behavior when required [11, 48, 37]. Once a mission is assigned to a specific robot, an autonomous robot uses its own individual mission planning/coordinating algorithms to conduct the mission. Individual mission/workload assigned to human-operated robots is coordinated by their human counterparts.

Definition II.2.

We define workload on (either autonomous or human-operated) at time as and the corresponding workspace as regardless of its task depending on the application. Upon mission assignment to the MH-MR system, the initial workload for each robot may or may not be equally distributed.

The objective is to provide a systematic approach to an adaptive workload allocation in MH-MR systems based on: (a) robot and human operator state monitoring ( and ), and (b) autonomous robot and human-operated robot work performance states (). The transition process between workload changes for a robot or operator must consider the effect of the prescribed change. The framework must maintain generality for applicability in any MH-MR system.

At the core of the proposed adaptive MH-MR system framework, the adaptive workload allocation system and a workload transition system, designated as module A.1 and module A.2 respectively, provide a workload distribution solution on the assigned mission based on the condition and performance of each human and robot unit operating in the system. The state of each human and robot in the system, and the performance of each autonomous and human-operated robot are assessed to adaptively re-allocate the total mission workload for continuous performance.

We realize that the relevance of such evaluations or assessments are application specific and may be irrelevant in certain systems. Therefore, to maintain generality of our adaptive MH-MR system framework, we propose a modular design consisting of robot and human state and performance assessment function blocks. Each module provides a real-time metric of its unit system adhering to its own procedure based upper and lower bounds. The modules and their metrics are listed as follows:

  • Module R: Robot state monitoring and evaluation metric from , normalized as

  • Module H: Human operator state monitoring and evaluation metric from , normalized as

  • Module P: Robot (human-operated or autonomous) performance assessment metric from , normalized as ,

where and represent the lower and upper bounds of the corresponding metric respectively.

The aforementioned human and robot states and performance metrics from the modular function blocks are fed into the adaptive workload allocation module A.1 for workload re-allocation. The workload is re-allocated to maximize overall system performance at all times. The modular design ensures that any module may be added or removed from the system depending on the application requirement, pertaining to the generalization of the framework. Fig. 2 illustrates the proposed modules of the MH-MR system framework.

Iii Adaptive MH-MR System Framework

Iii-a Module A.1 Adaptive Workload Allocation

We design the workload allocation of the system based on the maximum outcome of combining the incoming human and robot states and performance metrics. A variant of the softmax function, also known as the normalized exponential function is proposed to determine the workload allocation for each of the

robots. We define a vector

S of normalized inputs such that,


where is a vector of , , , and . By this design, the terms ensure that the system allocates zero workload to a robot (autonomous or human-operated), if the corresponding robot and/or human operator is detected to have voluntarily/involuntarily stopped working ( would equal to zero), completely failed, incapacitated and/or may have suffered from any discontinuity or disconnectedness in the teleoperation and communication graph structure ( and/or would equal to zero). It also ensures that the workload allocated is proportional to the worst human/robot condition/performance in situations where one increases and another decreases equally.

We calculate the share of the total workload for robot , at current time as,


The normalization ensures that the sum of all is , pertaining to the total workload of the system.

Iii-B Module A.2 Workload Allocation Transition

Sudden changes in workload allocation may have overwhelming effects on a host. The transitions must be smooth and manageable without any drastic changes. We model such a transition process for the workload change from the current actual workload allocation to the proposed workload allocation considering the effect of the change to the highest affected agent in the system as follows.

We model the workload transition process as,


where , and is a transition model coefficient dependent on the highest effect of the proposed change on the system.

Denoting the proposed 2-D workspace for corresponding to proposed workload as , we determine the highest effect on the system as , where denotes the shortest Euclidean distance between the boundary of workspace and , . In situations where a complete robot failure occurs or a human operator is incapacitated, the failed robot is ignored in determination.

The transition coefficient can therefore be modeled as an exponential function of ,


where is a positive scaling constant. The exponential nature of the transition allows for a smooth change in workload where may be tuned to control the rate of transition depending on particular application scenarios.

The workload allocation cycle must be synced with the contributing modules in the framework. The workload allocation update cycle time constant can therefore be set as where , and denote the required operation cycle time constants for function modules P, H, and R respectively.

Iii-C Human and Robot, Condition and Performance

Iii-C1 Module H: Human Condition Evaluation Metric

We define human operator condition as their ability to perform their task of teleoperating robots as a function of stress, emotion, and/or direct physiological measurements depending on the MH-MR application. For human operator condition evaluation, we refer to previous studies in literature for quantitative and qualitative techniques. Primary approaches include predicting stress or emotion from audio signals [34], gestures [45, 12], facial expressions [28], body gestures [13] or physiological signals such as heart rate, skin conductance, and respiration [69, 4, 62, 52, 58, 41]. The measurements and predictions are used to evaluate stress and psychological dynamics in the interest of creating effective working conditions [16]. Individual or a combination of a number of emotional responses may be measured and used as human operator condition for Module H, but at this stage we focus on human operator stress levels that have been shown to have a direct negative impact on work performance [64].

Galvanic skin response (GSR) or skin conductance is a reliable indicator of stress [17]. Under stress, skin conductance of an individual is increased [33] due to increase in moisture on the surface of the skin, which increases the flow of electricity. Healey et al. proposed a continuous stress measurement metric in [17] that can be normalized and used as a measure of the human operator condition directly for our proposed framework. Also, a number of other such human operator condition measurement metrics based on facial expressions, body gestures, heart rate and respiration have been summarized in the stress recognition literature survey [55] that may be used as Module H in our proposed framework.

Stress detection using a combination of multiple noninvasive physiological variables such as galvanic skin responses, blood volume pulse, pupil diameter and skin temperature have been proposed in [68]

. A support vector machine is used to perform the supervised classification of effective states between ”stress” and ”relaxed”. Stress levels may also be further discretized as ”low”, ”medium” and ”high”; such discrete states may be quantified as discrete human operator condition values simply as

, and respectively, or a moving average may also be applied depending on the application. We stress here that our proposed method is designed for continuous human operator condition values, but may still be adapted with a discrete human operator condition evaluation system as well appropriate of the application.

(a) Raw data of R-R intervals from [18]. R-R interval was measured from participants for a walking task between  mins followed by a small recovery period, and then a task to watch a horror film between   mins.
(b) Stress detection using the TPOT algorithm [29]. A possible moving average filter implementation for continuous stress quantification (blue) is shown followed by its human condition metric assessment (red).
Fig. 3:

An example implementation of a machine learning-based stress detection algorithm using HRV signals and R-R intervals; (a) raw data of R-R intervals from the existing dataset

[18], and (b) the stress detection and subsequent human condition assessment.

Heart rate variability (HRV) in terms of the length between heartbeat intervals, also called an R-R interval or inter-beat interval (IBI) plays an important role in predicting human condition in neurosciences and medical fields [25]. HRV has been utilized to predict stress in various works of literature [42, 38, 15, 20], due to its responses to physiological and environmental stimulus. Ottesen proposed a stress detection algorithm using wearable devices and machine learning technology [47] using both heart rate (HR) and HRV as a training dataset from [18] as shown in Fig. 2(a)

; a machine learning model was proposed for automated machine learning and the evolutionary algorithm called TPOT

[29]. The model had a stress detection accuracy of % using an existing dataset from a user study, where participants watched a horror movie after a 15-minute walking task to differentiate between physical and mental stress, such as lowering the R-R intervals. An implementation of the TPOT algorithm for stress detection is shown in Fig. 2(b).

An example implementation of the moving average filter as a possible continuous stress quantification method is shown on this discrete assessment of stress from the machine learning-based stress detection algorithm as the blue line denoted as . This moving average filter smooths the rapidly changing binary output of the stress detection algorithm in the time domain [53]. The human condition metric is defined to be in the closed range for worst to perfect; therefore, the continuous stress plot

is mapped to the estimated human condition as

to obtain a continuous human condition metric required in the MH-MR workload allocation framework. We chose this specific dataset in our study to show stress and subsequent human condition assessment because of the following properties: drastically changing human condition between  mins, slow change between  mins and sudden changes between  mins. We validate the effectiveness of our proposed framework on simplified cases of these rates of change of human condition in Section IV-A3.

We deem noise reduction and disturbance rejection in measuring human operator condition as beyond the scope of our current work and included within the above presented human condition measurement metrics; a few specific works on signal processing and noise filtering of physiological measurements have been proposed in [27, 32]. Therefore, we assume that human operator condition can be measured and quantified with enough certainty and noise rejection for application in our proposed MH-MR workload allocation framework.

To establish the generality of the modular human condition assessment function block in the proposed work allocation framework, we stress the following notes on possible human-operated robot scenarios. In cases where one human operates one robot or one human operates multiple robots, the human operator’s health condition would be independently used in Module H for each of the operated robot’s work allocation in Module A. However, if multiple humans control a single robot for an MH-MR application, the condition assessments of all the human operators of this particular robot would have to be considered for its workload allocation. In such a scenario where one robot is operated by more than one human operator, we assume that each human operator of the robot has exclusive on-board tasks: one operates navigation, one operates surveillance etc.; if one operator’s condition deteriorates, one on-board task is affected. i.e. as a whole, this specific multi-human robot team’s working ability is also affected. The definition of in Eq. (1) ensures that the system allocates zero workload to a robot for any of its operators becoming incapacitated. The proposed MH-MR workload allocation system is therefore general to any number of operators controlling any number of robots in the system. With these generality notes, Eq. (3) assigns workload to individual robots reflecting its individual ability to work considering the conditions of all its human operators.

Iii-C2 Module R: Robot Condition Evaluation Metric

On-board quantitative measurements of robot health may include battery level, communication signal strength, internal temperature and a variety of other factors [51]. Detecting sub-nominal characteristics and isolating problems through self-checking have also been considered in different autonomous robot platforms currently available. Qualitative evaluations may also be included for robot condition evaluation based on the robot’s physical state. We refer to the Neglect Tolerance metric [14] for autonomous robots as a measure of how a robot’s effectiveness declines in autonomous mode without any human supervision or control. It includes task complexity and robot capability among various other factors to provide an overall measure of a robot’s condition of autonomy.

Iii-C3 Module P: Robot Performance Evaluation Metric

Robot (either autonomous or human-operated) performance metrics such as percentage area coverage or distance travelled proposed in [66] may be used to asses robot performance depending on the MH-MR application. Performance of robots may be determined in terms of task completion time, path following cross-track error [6] etc. depending on the application of the proposed framework. Human Robot Interaction (HRI) metrics recommended by Steinfeld et al. [57] and reviewed by Murphy [43] in terms of navigation (e.g. localization, effective path determination around objects), perception (e.g. surveillance, target identification, sensor area coverage), manipulation, and management at the human, robot, and system level perspectives may be used to evaluate human-operated robot performance using an arbitrary evaluation function plugged in as Module P. We leave performance assessments for Module P at the discretion of their relevance to particular applications.

Iv Validation & Results

As validation of the effectiveness of the adaptive task allocation mechanism, we present our experimental findings of applying the proposed framework to a MH-MR patrolling application, where human operator and robot conditions affect their patrolling ability.

Four experiment scenarios were independently investigated. In the first scenario (S1), we simulate temporary and permanent deteriorated conditions for a human operator and an autonomous robot in sequence, while in the second scenario (S2) we simulate complete failure of a robot, and analyze the system’s workload adaption in each scenario. In two further scenarios (S3 and S4) we present workload transitioning and scalability analysis with similar conditions as S1 and S2 respectively.

Before moving on to including real human operators in the experiments, it is vital that controllable evaluation scenarios are used to validate our proposed work. Therefore, in this paper we present the investigation results of our proposed method using simulated human operator conditions of different characteristics.

Iv-a S1 and S2: Adaptive Workload Allocation in Patrolling

Iv-A1 Patrolling Application

Machado et al. broadly defined patrolling as “the act of walking or traveling around an area, at regular intervals, in order to protect or supervise it” [35]. Therefore, we set up our representative patrolling application with a given number of robots traveling around allocated rectangular regions on a plane, where the sum of the area of all rectangular regions represents the total workload. The allocated workload from our proposed framework may be directly used in more complex region allocations for the patrolling scenario following capacity-constrained Voronoi tessellation works proposed in literature [2]. Applications in multi-robot coverage problems include [22]. However, for simplicity and ease of analysis we validate our system using rectangular patrolling regions, and we define patrolling for each robot as boundary following its allocated area within a specified time within its ability.

A patrolling performance metric is defined for comparison study with and without the proposed workload allocation method. Patrolling performance of the complete MH-MR system is measured as the maximum time taken to patrol the entire area once by the MH-MR system expressed as,


where denotes the patrolling lap time of for during one cycle of full area patrolling, given that the entire area is covered by all robots.

Fig. 4: Experiment setup with Jackal mobile robots; and human-operated, and autonomous. Robots patrol rectangular regions on the plane, defined as boundary following its allocated area. Patrolling velocities are modeled dependent on human operator and robot conditions.

Iv-A2 Experiment Setup

We consider a MH-MR system of human operators (simulated) and mobile robots (Jackals from Clearpath Robotics) with position defined as , for on a level plane as shown in Fig. 4. Robot position data was recorded using a VICON system. True velocity estimation of the Jackals were made from the collected position data with time. We simulate robots and as being controlled by human operators while remains autonomous in patrolling. To simulate the human operators, human operator condition assessment inputs are provided for and ; all robots utilize the same low level line-of-sight path following controller for consistency. The effect of workload change in the system at time depends on the minimum distance from to the changing rectangular region boundaries using the allocation transition coefficient model in Eq. (4).

At initial time, the patrolling area was distributed equally among all robots as rectangular regions with a specified safety distance between rectangular boundaries to prevent inter-robot collisions while patrolling, and the human operator and robot conditions were considered optimal. In course of the experiments, the rectangular region areas were re-allocated based on the proposed workload allocation framework. We acknowledge that increasing workload on an agent due to re-allocation, may reduce performance or in turn cause condition deterioration. Nevertheless, such effects on agents were ignored for validation purposes of the proposed framework.

We model robot patrolling ability dependent on current human operator and robot condition,




where is a vector of , assuming and are bounded within , and denotes the maximum allowed velocity of .

The required patrolling velocity of is modeled as,


where the patrolling time threshold is set as  s and . Velocity of is therefore modeled as,


The initial value of is arbitrarily set large enough for experimental analysis purposes with  m/s.

Validation setup parameters include the simulated human and robot condition update frequency time set as  ms, workload transition scaling constant .

Performance of each robot on the patrolling task is measured as cross-track error with an error margin of . In reality, the performance measure would also include corresponding to deteriorated performance of the robot. However, we intentionally do not consider velocity differences in our robot performance assessment in this validation setup, since we focus on independent analysis and assessment of the proposed workload allocation based on human operator and robot condition only. Performance measure of all robots is assumed to be unity at all times.

Iv-A3 S1: Adaptation to Deteriorated Conditions

(a) Simulated human conditions over time. deteriorates drastically at , and then subsequently recovers back to after a further small deterioration at ; remains at at all times.
(b) Simulated robot conditions over time. deteriorates permanently at ; , remains at at all times.
(c) Workload allocation of patrolling robots change according to the simulated human and robot conditions: allocated workload of robots with deteriorated human and/or robot condition is reduced and compensated by robots with better human and/or robot condition. The workload transition function ensures that drastic changes are smoothened for a manageable effect on the host; yet remains sensitive enough to capture sudden changes in agent condition.
(d) Translation velocity profiles of patrolling robots with and without workload allocation ignoring region corner rotations. Translation velocity of robots with deteriorated human and/or robot conditions is observed to have lowered velocity (, after and respectively); robots with increased workload after re-allocation having better conditions observed to increase their translation velocity (, after and after ).
Fig. 5: S1: Adaptive workload allocation for temporary and permanent human and robot condition deterioration.

We model the conditions for the human operator of denoted as to deteriorate drastically at time and then subsequently recover slowly back to after a sudden further small deterioration at time as shown in Fig. 4(a); this is based on simplified, observed and analyzed condition patterns of quantified human stress from Fig. 3 (drastically changing human condition between  mins, slow change between  mins and sudden changes between  mins). Here we stress the design of the simulated events having drastic and different rates of changes on the two separate time instances (abrupt and slow), to show their effects on the workload allocation. The minor further deterioration before recovery after is simulated to investigate the sensitivity of the proposed workload allocation framework to sudden small changes in operator condition. Deteriorated condition of the autonomous robot denoted as is simulated as shown in 4(b). is simulated to deteriorate permanently at . The experiment S1 is repeated with and without the proposed adaptive workload allocation framework to compare their effects on the patrolling application using the defined patrolling performance metric. The results are presented in Fig. 5, 6 and 7.

(a) Patrolling trajectory of robots with workload allocation
(b) Patrolling trajectory of robots without workload allocation
Fig. 6: S1: Patrolling trajectory following comparison with and without workload allocation for temporary and permanent human and robot condition deterioration.
Fig. 7: S1: Total and individual patrolling time required comparison with and without workload allocation.

With initially set equal workload, all robots patrol their equally allocated rectangular boundaries until time event . At where shows drastically falling conditions, the workload is re-allocated to reduce load on the human operated robot and increased equally among and having better conditions as seen in Fig. 4(c); the re-allocation is reflected as a smaller patrolling region for and equal larger regions for and in Fig. 5(a). and were both positioned roughly equally close to the changing boundary of their rectangular regions during the first event at , and thus both robots transition to their allocated workload at the same time of around  s shown in Fig. 4(c). The corresponding changes in the velocity profiles for each robot with workload allocation is shown in Fig. 4(d). The velocity of is seen drastically reduced with ; and with increased allocations of patrolling regions, the other two robots at this point still in good condition are observed to slightly increase their velocities ( and ). The patrolling time for is observed as increasing above the tolerance at lap and eventually levelling at lap due to the slow workload transition process as shown in Fig. 7; and remained high over laps and due to more frequent slower turning at corners. Patrolling times for and with optimal conditions remained steady within defined tolerance after time event .

Similar observations are made after time event , where robot suffers a sudden condition deterioration. The workload of is reduced and re-distributed among the other two as seen in Fig. 4(c) and 5(a). Velocity of decreases permanently due to its deteriorated condition. is left with patrolling a larger region in comparison to the others; its velocity increases to maintain the patrolling time requirement. However, the velocity of remains the same due to its previously deteriorated condition. Thus, the patrolling time for remains considerably higher than and for laps and after time event as shown in Fig. 7.

Compared to the drastic change of at and at , gradually returns to after a sudden drop at . The workload allocation is seen to change relatively slowly as well for this time event over time period  s to  s as observed in Fig. 4(c). Right before the recovery, the workload allocation plot shows a slight dip in allocated workload for and small increases for and validating the effective sensitivity of the proposed workload allocation method. Upon condition improvements of at , the workload is redistributed again to equal portions among and with corresponding rectangular block patrolling trajectories shown in Fig. 5(a). The velocity of and equalize to a larger value than to compensate for the re-allocated patrolling regions with lower workload for . Patrolling lap times for all robots return within the defined tolerance after event at patrol lap with the highest time taken by .

(a) Simulated human conditions over time. and remains at at all times.
(b) Simulated robot conditions over time. and remains at at all times; deteriorates permanently at .
(c) Workload allocation of patrolling robots change according to the simulated human and robot conditions: allocated workload of incapacitated robots is zero and compensated by robots with better human and/or robot condition. The workload transition function module ensures that drastic changes are smoothened for a manageable effect on the host.
(d) Translation velocity profiles of patrolling robots with and without workload allocation ignoring region corner rotations. Translation velocity of incapacitated robots (deteriorated ability) observed to be zero ( after ), while robots with increased workload after re-allocation having better conditions observed to increase their translation velocity to compensate (, after ).
Fig. 8: S2: Adaptive workload allocation for complete robot failure condition.

To validate the effectiveness of the proposed method, the experiment scenario is repeated without using the adaptive workload allocation framework. The robot patrolling trajectories followed the initial equal rectangular region allocation throughout the experiment as presented in Fig. 5(b). With equal rectangular region allocation over the entire experiment duration, the robot velocities only reflected the temporary and permanent deteriorating conditions of () and () showing slower patrolling speed as observed in Fig. 4(d) and higher patrolling times after and respectively. Referring to the previously defined patrolling performance metric, the total area patrolling time was recorded to be  s higher on lap (right after ) without workload allocation. With the initial dip in after , the area patrolling time was initially recorded to be  s higher on lap without workload allocation in comparison, that reduced within the set tolerance on lap , when the simulated gradually returned to .

The events , and triggered changes in workload on immediate neighbors of , allowing it to quickly adjust its velocity to meet the required patrolling lap time. The workload change after event , was slower for in comparison as it adjusted to the change following transitioning of , resulting in increased patrolling lap times in laps and . The event triggered in between patrolling laps and of permanently kept its patrolling velocity at  s with workload allocation. In comparison, its patrolling lap time without workload allocation is observed to increase on lap and permanently stay  s higher for the rest of the experiment. Although insignificant compared to , the total area patrol time remained  s less due to with the proposed workload allocation after event on lap .

Iv-A4 S2: Adaptation to Robot Failure

Experimental cases of complete robot failures have also been investigated, where is completely incapacitated by setting at event in a separate experiment. Fig. 7(a) and 7(b) shows the simulated human and robot conditions for S2.

Fig. 9: S2: Patrolling trajectory of robots with workload allocation for complete robot failure condition.

We refer to Fig. 7(c) to present the resulting allocated workloads after event . At event , the initial area of is equally allocated amongst and for continuous full patrolling area coverage; i.e. at any time instant, the total allocated workload was always unity with the proposed adaptive workload allocation framework. This verifies that the workload was always re-allocated to ensure total area coverage by the MH-MR patrolling system. The resulting robot trajectory plots are shown in Fig. 9. The modeled velocity plots shown in Fig. 7(d) confirm the increased patrolling velocities for and to compensate for their allocated larger equal areas. For S2, we omit comparison of patrolling performance with and without using adaptive workload allocation, since total area patrolling could only be achieved with the proposed adaptive workload allocation framework.

(a) Initial and final workload allocated regions for stationary robot MH-MR system.
Fig. 10: S3: Transition analysis for stationary robots equidistant from their region boundaries along the horizontal axis, with equal initial workload allocation. Human and robot conditions are simulated as , , , with the rest as from , and the system adaptively converges to the new workload depending on . Green dots represent robots. Zoomed sections of plots shown in insets.

Iv-B S3 and S4: Workload Allocation Transition Analysis

The workload transitioning module of the proposed MH-MR workload allocation framework is a function of . We present the effects of different on workload transition with simulation results of

robots stationary at all times. Scenario S3 simulates deteriorating human and robot conditions with all robots initially placed at the center of their regions along the horizontal axis; scenario S4 simulates failed robot cases with all robots initially placed closer to the left boundary of their rectangular regions. Odd-indexed robots are assumed to be human-operated while even-indexed robots are assumed autonomous. At

, the agent conditions are set to , and in both scenarios; is set to as deteriorated condition in S3 and as failed condition in S4 with the rest of the agent conditions remaining at at all times. Fig. 9(a) and 10(a) illustrate the initial setup for S3 and S4 respectively.

(a) Initial and final workload allocated regions for stationary robot MH-MR system.
Fig. 11: S4: Transition analysis for stationary robots closer to their left region boundary along the horizontal axis, with equal initial workload allocation. is simulated to completely fail with , along with human and robot conditions , , from , and the system adaptively converges to the new workload depending on . Green dots represent working robots and red dots represent failed robots. Zoomed sections of plots shown in insets.

The workload convergence and convergence error plots for S3 shown in Fig. 9(b) and 9(c), present a uniform workload transition for all robots to their re-allocated workloads consistent with the setup having all robots initially placed at the center of their regions along the horizontal axis. converged to the lowest allocated workload followed by and , while the other robots compensated with increased allocated workload. As such, the workload convergence rate was highest for followed by and with increasingly slower rates respectively following smaller . The rest of the robots showed the smallest rate of convergence to increased allocated workload with small and equal change in .

In contrast, S4 where is simulated to fail completely, converges to the zero allocated workload much faster given the larger as shown in Fig. 10(b). The actual workload convergence rate for was followed by and with increasingly slower rates respectively following smaller in comparison. and are both observed to gain a higher workload initially at around due to their close proximity to the largest changing workload allocation in the system for , before reaching an equilibrium workload with the other robots. Between with shrinking to zero faster than the other robots, and compensate with a larger share of actual workload temporarily experiencing a faster convergence rate compared to the other agents before the adjustments are propagated to the rest of the robots reaching an equilibrium in the group. The convergence rate of workload allocation for is initially observed slightly higher than for consistent with the proposed workload transition model with initially placed further away from the changing boundary with . then shows a higher workload convergence rate between as the changing boundary moves away from and closer to . The rest of the robots in the group reach an equilibrium workload fairly slowly in comparison, as and adjust over time.

Fig. 12: Effect of on the total workload transition time for robots following scenario S3: human and robot conditions set to , , and . The system adaptively converges to the new workload fastest for and increasingly slower with lower as expected.

In Fig. 10(c), the workload transition error of shows an initial error of due to the condition at . Over the next few time steps, the error is observed to rise sharply a little over as the effect of the deteriorated conditions of and are propagated to the transitioning workloads of the rest of the robots including ; i.e. the error for was compounded with the compensating errors of and before reaching a transitional error of zero. With a complete robot failure in S4, the amount of workload to be re-allocated was larger while considering transitional effects on all agents; hence the workload convergence time for S4 was recorded higher than S3. The workload convergence and the convergence error plot for S4 are shown in Fig. 10(b) and 10(c) respectively.

As scalability analysis of the proposed MH-MR workload allocation and transition framework, scenario S3 was repeated for with varying . Fig. 12 plots the total workload transition error along a logarithmic time scale for , , and . The total workload transition error for all cases of reach zero in finite time. yielded the fastest convergence of the error to zero with increasingly slower rates for lower values of as expected. Similar observations are made for , and robot cases each with and as shown in Fig. 13; a larger yielded a faster convergence of the total workload convergence error to zero. The effect of larger gets smaller with larger ; a minor difference is observed for the two cases for . The total initial error was higher for smaller due to the initially equal distribution of workload assumption of the scenario.

V Discussion

The workload-allocation and transition problem is addressed from a high-level abstraction to maintain generality of its application. The proposed MH-MR framework is suitable for homogeneous and heterogeneous robots (ground, aerial etc.) on homogeneous tasks, given that all robots in the group are capable of completing the homogeneous task of the system independently, and each robot is equipped with all appropriate and relevant low-level controllers. The system is robust to addition and removal (varying ) of autonomous and teleoperated robots alike at any time during the mission since each update cycle of the workload allocation process is independent of the previous; the system will simply reallocate the workload accordingly in the next update cycle following Eq. (1). Workload transition considerations of added robots in the next update cycle are also made with determined with following Eq. (3).

The system allows autonomous robots and any number of human operators to teleoperate any number of robots: each human operator may teleoperate multiple robots and multiple human operators may teleoperate a single robot. The graph structure represents the variable human-robot connectivity of the system. The graph connectivity must therefore be updated within the system update cycle . Any discontinuity or disconnectedness in the teleoperation or communication graph structure defined in Section II is treated as a failed robot with zero health.

Our current work limits the update cycle to the slowest frequency of the individual modules. However, we acknowledge that it may be improved by considering the highest frequency of all the individual modules and relying on current estimates for the slower modules; an implementation of the Kalman filter for the slower modules may also be used for better current estimates. We identify this potential improvement in the system update cycle as future work on our proposed framework.

Fig. 13: Scalability analysis with , , and robots following scenario S3: human and robot conditions set to , , and . Effect of is consistent with larger yielding faster total workload transition time. Increasingly larger resulted in increasingly smaller total workload allocation error and longer transition times as expected.

We acknowledge that if a large number of agents suffer from deteriorated conditions at once and the rest of the agents are asked to compensate, since the system is designed to ensure that the entire workspace is allocated at all times, it may overwhelm them as well and in turn affect their health/ability and performance as well. The current framework is unable to consider how much of the total workload can actually be allocated to the given number of agents such that certain agents are not overwhelmed even if their health/ability are optimal. We identify this as a limitation to the proposed framework and hope to address this issue in our future work. The current framework is therefore only applicable assuming enough agents are in optimal conditions in terms of health/ability, such that the total workspace could be covered at all times without affecting compensating agents.

The current MH-MR framework assigns workload relying only on agent condition and performance. However, human operators in the system may have different levels of skill, experience and responsiveness despite the measured human condition metric. As such, we acknowledge that with the current design for workload allocation the full potential of the human robot team may not be utilized. Different human operators may also have different working capabilities even under stress or different emotional states which have not been considered in the current system. A number of other complexities also exist on measuring human operator health and condition in the real world in terms of applicability of sensors, the variable calibration requirements and environmental effects that contribute to the huge variety in recorded human behavior [24]. Therefore, as future work of our MH-MR workload allocation framework, we intend to investigate independent human condition assessment and incorporate further human operator attributes in the workload allocation process.

The proposed framework allows multi-human and multi-robots to work together in a given application; robots are free to work autonomously and may also be teleoperated by human operators all the while ensuring that the total work always sums to unity. Therefore, autonomy of the system on the application is shared amongst all individual agents. With the lap time comparison for scenario S1, and failed robot case in scenario S2 presented in the validation section of the manuscript, we established the effectiveness of our proposed workload allocation framework. Therefore, we believe that the proposed MH-MR workload allocation framework is an effective shared-autonomy tool. S3 and S4 presented the effects of on workload allocation followed by the scalability of the system established for various .

The proposed workload allocation and transition function modules are designed to reflect the condition and performance of the autonomous robots, and the humans and robots in teleoperated robots focusing on the working ability of each individual rather than overall optimal system performance. This approach is important for operators (any agents in the system) to believe that the system will consider any deterioration in their health/ability to work and adjust their workload accordingly, such that they are never overwhelmed. We believe that this function could potentially instill trust in operators (any agents in the system) on the shared-autonomy of the system while working to ensure that they are never overwhelmed with their currently allocated workload.

Vi Conclusion

An adaptive multi-human multi-robot system framework has been proposed that performs real-time workload allocation based on both human operator and robot conditions and on performance, with workload transitional considerations. The design allows compatibility with previously established quantitative human and robot health assessments tests; assuming human and/or robot conditions and/or performance are measured with enough accuracy, the modular design of the framework can be used for a wide variety of multi-agent applications, including search and rescue, exploration, surveillance and monitoring based on specific requirements. The system functions independent of the number of humans or robots and is therefore scalable to hosting any number of agents.

The applicability, effectiveness and scalability of the proposed framework was validated experimentally with a MH-MR patrolling application, demonstrating system adaptation to maintain performance despite simulated temporary and permanent, deteriorating human and robot conditions, including complete robot failures. Further work on incorporating modular function blocks on human experience, skill, responsiveness and safety protocols within the work allocation module in the presence of sub-nominal human and/or robot conditions is currently underway, along with field deployment studies of the proposed framework.


  • [1] N. Atay and B. Bayazit (2006) Mixed-integer linear programming solution to multi-robot task allocation problem. Cited by: §I.
  • [2] M. Balzer (2009) Capacity-constrained voronoi diagrams in continuous spaces. In 2009 Sixth International Symposium on Voronoi Diagrams, pp. 79–88. Cited by: §IV-A1.
  • [3] K. E. Booth, G. Nejat, and J. C. Beck (2016) A constraint programming approach to multi-robot task allocation and scheduling in retirement homes. In International conference on principles and practice of constraint programming, pp. 539–555. Cited by: §I.
  • [4] D. Carneiro, J. C. Castillo, P. Novais, A. Fernández-Caballero, and J. Neves (2012) Multimodal behavioral analysis for non-invasive stress detection. Expert Systems with Applications 39 (18), pp. 13376–13389. Cited by: §III-C1.
  • [5] S. Chien, M. Lewis, S. Mehrotra, N. Brooks, and K. Sycara (2012) Scheduling operator attention for multi-robot control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 473–479. Cited by: §I.
  • [6] N. A. Cruz and J. C. Alves (2014) Navigation performance of an autonomous sailing robot. In 2014 Oceans-St. John’s, pp. 1–7. Cited by: §III-C3.
  • [7] M. Darrah, W. Niland, and B. Stolarik (2005) Multiple uav dynamic task allocation using mixed integer linear programming in a sead mission. In Infotech@ Aerospace, pp. 7164. Cited by: §I.
  • [8] M. B. Dias, R. Zlot, N. Kalra, and A. Stentz (2006) Market-based multirobot coordination: a survey and analysis. Proceedings of the IEEE 94 (7), pp. 1257–1270. Cited by: §I.
  • [9] J. L. Drury, J. Scholtz, and H. A. Yanco (2003) Awareness in human-robot interactions. In SMC’03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme-System Security and Assurance (Cat. No. 03CH37483), Vol. 1, pp. 912–918. Cited by: §I.
  • [10] Y. Emam, S. Mayya, G. Notomista, A. Bohannon, and M. Egerstedt (2020) Adaptive task allocation for heterogeneous multi-robot teams with evolving and unknown robot capabilities. arXiv preprint arXiv:2003.03344. Cited by: §I.
  • [11] D. Fox, W. Burgard, S. Thrun, and A. B. Cremers (1998) A hybrid collision avoidance method for mobile robots. In Proceedings. 1998 IEEE International Conference on Robotics and Automation (Cat. No. 98CH36146), Vol. 2, pp. 1238–1243. Cited by: §II.
  • [12] E. Garcia-Ceja, V. Osmani, and O. Mayora (2015) Automatic stress detection in working environments from smartphones’ accelerometer data: a first step. IEEE journal of biomedical and health informatics 20 (4), pp. 1053–1060. Cited by: §III-C1.
  • [13] D. Giakoumis, A. Drosou, P. Cipresso, D. Tzovaras, G. Hassapis, A. Gaggioli, and G. Riva (2012) Using activity-related behavioural features towards more effective automatic stress detection. PloS one 7 (9), pp. e43571. Cited by: §III-C1.
  • [14] M. A. Goodrich and D. R. Olsen (2003) Seven principles of efficient human robot interaction. In SMC’03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme-System Security and Assurance (Cat. No. 03CH37483), Vol. 4, pp. 3942–3948. Cited by: §III-C2.
  • [15] J. Gouin, K. Wenzel, S. Boucetta, J. O’Byrne, A. Salimi, and T. T. Dang-Vu (2015) High-frequency heart rate variability during worry predicts stress-related increases in sleep disturbances. Sleep medicine 16 (5), pp. 659–664. Cited by: §III-C1.
  • [16] S. Greene, H. Thapliyal, and A. Caban-Holt (2016) A survey of affective computing for stress detection: evaluating technologies in stress detection for better health. IEEE Consumer Electronics Magazine 5 (4), pp. 44–56. Cited by: §III-C1.
  • [17] J. A. Healey and R. W. Picard (2005) Detecting stress during real-world driving tasks using physiological sensors. IEEE Transactions on intelligent transportation systems 6 (2), pp. 156–166. Cited by: §III-C1.
  • [18] J. A. Healey (2000) Wearable and automotive systems for affect recognition from physiology. Ph.D. Thesis, Massachusetts Institute of Technology. Cited by: 2(a), Fig. 3, §III-C1.
  • [19] E. G. Jones, M. B. Dias, and A. Stentz (2011) Time-extended multi-robot coordination for domains with intra-path constraints. Autonomous robots 30 (1), pp. 41–56. Cited by: §I.
  • [20] E. Jovanov, A. Lords, D. Raskovic, P. G. Cox, R. Adhami, and F. Andrasik (2003) Stress monitoring using a distributed wireless intelligent sensor system. IEEE Engineering in Medicine and Biology Magazine 22 (3), pp. 49–55. Cited by: §III-C1.
  • [21] D. B. Kaber, C. M. Perry, N. Segall, C. K. McClernon, and L. J. Prinzel III (2006) Situation awareness implications of adaptive automation for information processing in an air traffic control-related task. International Journal of Industrial Ergonomics 36 (5), pp. 447–462. Cited by: §I.
  • [22] N. Karapetyan, K. Benson, C. McKinney, P. Taslakian, and I. Rekleitis (2017) Efficient multi-robot coverage of a known environment. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1846–1852. Cited by: §IV-A1.
  • [23] M. Khoramshahi and A. Billard (2019) A dynamical system approach to task-adaptation in physical human–robot interaction. Autonomous Robots 43 (4), pp. 927–946. Cited by: §I.
  • [24] R. Khusainov, D. Azzi, I. E. Achumba, and S. D. Bersch (2013) Real-time human ambulation, activity, and physiological monitoring: taxonomy of issues, techniques, applications, challenges and limitations. Sensors 13 (10), pp. 12852–12902. Cited by: §V.
  • [25] H. Kim, E. Cheon, D. Bai, Y. H. Lee, and B. Koo (2018) Stress and heart rate variability: a meta-analysis and review of the literature. Psychiatry investigation 15 (3), pp. 235. Cited by: §III-C1.
  • [26] A. Koubaa, H. Bennaceur, I. Chaari, S. Trigui, A. Ammar, M. Sriti, M. Alajlan, O. Cheikhrouhou, and Y. Javed (2018) General background on multi-robot task allocation. In Robot Path Planning and Cooperation, pp. 129–144. Cited by: §I.
  • [27] S. Lahmiri and M. Boukadoum (2015) Physiological signal denoising with variational mode decomposition and weighted reconstruction after dwt thresholding. In 2015 IEEE international symposium on circuits and systems (ISCAS), pp. 806–809. Cited by: §III-C1.
  • [28] S. M. Lajevardi and H. R. Wu (2012) Facial expression recognition in perceptual color space. IEEE transactions on image processing 21 (8), pp. 3721–3733. Cited by: §III-C1.
  • [29] T. T. Le, W. Fu, and J. H. Moore (2020) Scaling tree-based automated machine learning to biomedical big data with a feature set selector. Bioinformatics 36 (1), pp. 250–256. Cited by: 2(b), §III-C1.
  • [30] D. Lee (2018) Resource-based task allocation for multi-robot systems. Robotics and Autonomous Systems 103, pp. 151–161. Cited by: §I.
  • [31] M. Lewis, H. Wang, S. Chien, P. Scerri, P. Velagapudi, K. Sycara, and B. Kane (2010) Teams organization and performance in multi-human/multi-robot teams. In 2010 IEEE International Conference on Systems, Man and Cybernetics, pp. 1617–1623. Cited by: §I.
  • [32] Q. Li, R. G. Mark, and G. D. Clifford (2007) Robust heart rate estimation from multiple asynchronous noisy sources using signal quality indices and a kalman filter. Physiological measurement 29 (1), pp. 15. Cited by: §III-C1.
  • [33] W. Liao, W. Zhang, Z. Zhu, and Q. Ji (2005)

    A real-time human stress monitoring system using dynamic bayesian network


    2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05)-Workshops

    pp. 70–70. Cited by: §III-C1.
  • [34] H. Lu, D. Frauendorfer, M. Rabbi, M. S. Mast, G. T. Chittaranjan, A. T. Campbell, D. Gatica-Perez, and T. Choudhury (2012) Stresssense: detecting stress in unconstrained acoustic environments using smartphones. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing, pp. 351–360. Cited by: §III-C1.
  • [35] A. Machado, G. Ramalho, J. Zucker, and A. Drogoul (2002) Multi-agent patrolling: an empirical analysis of alternative architectures. In International workshop on multi-agent systems and agent-based simulation, pp. 155–170. Cited by: §IV-A1.
  • [36] M. S. Malvankar-Mehta and S. S. Mehta (2015) Optimal task allocation in multi-human multi-robot interaction. Optimization Letters 9 (8), pp. 1787–1803. Cited by: §I.
  • [37] A. Martinez, E. Tunstel, and M. Jamshidi (1994) Fuzzy logic based collision avoidance for a mobile robot. Robotica 12 (6), pp. 521–527. Cited by: §II.
  • [38] D. McDuff, S. Gontarek, and R. Picard (2014) Remote measurement of cognitive stress via heart rate variability. In 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 2957–2960. Cited by: §III-C1.
  • [39] C. A. Miller and R. Parasuraman (2007) Designing for flexible interaction between humans and automation: delegation interfaces for supervisory control. Human factors 49 (1), pp. 57–75. Cited by: §I.
  • [40] A. R. Mosteo and L. Montano (2006) Simulated annealing for multi-robot hierarchical task allocation with flexible constraints and objective functions. In Workshop on Network Robot Systems: Toward Intelligent Robotic Systems Integrated with Environments”. IROS, Cited by: §I.
  • [41] A. Muaremi, B. Arnrich, and G. Tröster (2013) Towards measuring stress with smartphones and wearable devices during workday and sleep. BioNanoScience 3 (2), pp. 172–183. Cited by: §III-C1.
  • [42] N. Munla, M. Khalil, A. Shahin, and A. Mourad (2015) Driver stress level detection using hrv analysis. In 2015 International Conference on Advances in Biomedical Engineering (ICABME), pp. 61–64. Cited by: §III-C1.
  • [43] R. Murphy and D. Schreckenghost (2013) Survey of metrics for human-robot interaction. In Proceedings of the 8th ACM/IEEE international conference on Human-robot interaction, pp. 197–198. Cited by: §III-C3.
  • [44] S. Musić and S. Hirche (2017) Control sharing in human-robot team interaction. Annual Reviews in Control 44, pp. 342–354. Cited by: §I.
  • [45] F. Noroozi, D. Kaminska, C. Corneanu, T. Sapinski, S. Escalera, and G. Anbarjafari (2018) Survey on emotional body gesture recognition. IEEE transactions on affective computing. Cited by: §III-C1.
  • [46] M. Otte, M. J. Kuhlman, and D. Sofge (2020) Auctions for multi-robot task allocation in communication limited environments. Autonomous Robots 44 (3), pp. 547–584. Cited by: §I.
  • [47] C. A. Ottesen (2017-08) Investigating heart rate variability: a machine learning approach. Master’s Thesis, Queen Mary University of London. Cited by: §III-C1.
  • [48] A. Pandey, S. Pandey, and D. Parhi (2017) Mobile robot navigation and obstacle avoidance techniques: a review. Int Rob Auto J 2 (3), pp. 00022. Cited by: §II.
  • [49] R. Parasuraman, M. Barnes, K. Cosenzo, and S. Mulgund (2007) Adaptive automation for human-robot teaming in future command and control systems. International Journal of Command and Control 1 (2), pp. 43–68. Cited by: §I.
  • [50] L. J. Prinzel, F. G. Freeman, M. W. Scerbo, P. J. Mikulka, and A. T. Pope (2000) A closed-loop system for examining psychophysiological measures for adaptive task allocation. The International Journal of Aviation Psychology 10 (4), pp. 393–410. Cited by: §I.
  • [51] K. M. Reichard (2004) Integrating self-health awareness in autonomous systems. Robotics and Autonomous Systems 49 (1-2), pp. 105–112. Cited by: §III-C2.
  • [52] A. Sano and R. W. Picard (2013) Stress recognition using wearable sensors and mobile phones. In 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction, pp. 671–676. Cited by: §III-C1.
  • [53] H. Sarker, M. Tyburski, M. M. Rahman, K. Hovsepian, M. Sharmin, D. H. Epstein, K. L. Preston, C. D. Furr-Holden, A. Milam, I. Nahum-Shani, et al. (2016) Finding significant stress episodes in a discontinuous time series of rapidly varying mobile sensor data. In Proceedings of the 2016 CHI conference on human factors in computing systems, pp. 4489–4501. Cited by: §III-C1.
  • [54] T. Setter and M. Egerstedt (2016) Energy-constrained coordination of multi-robot teams. IEEE Transactions on Control Systems Technology 25 (4), pp. 1257–1263. Cited by: §I.
  • [55] N. Sharma and T. Gedeon (2012) Objective measures, sensors and computational techniques for stress recognition and classification: a survey. Computer methods and programs in biomedicine 108 (3), pp. 1287–1301. Cited by: §III-C1.
  • [56] M. Shiomi, K. Shatani, T. Minato, and H. Ishiguro (2018) How should a robot react before people’s touch?: modeling a pre-touch reaction distance for a robot’s face. IEEE Robotics and Automation Letters 3 (4), pp. 3773–3780. Cited by: §I.
  • [57] A. Steinfeld, T. Fong, D. Kaber, M. Lewis, J. Scholtz, A. Schultz, and M. Goodrich (2006) Common metrics for human-robot interaction. In Proceedings of the 1st ACM SIGCHI/SIGART conference on Human-robot interaction, pp. 33–40. Cited by: §III-C3.
  • [58] F. Sun, C. Kuo, H. Cheng, S. Buthpitiya, P. Collins, and M. Griss (2010) Activity-aware mental stress detection using physiological sensors. In International conference on Mobile computing, applications, and services, pp. 282–301. Cited by: §III-C1.
  • [59] F. Tang and L. E. Parker (2007) A complete methodology for generating multi-robot task solutions using asymtre-d and market-based task allocation. In Proceedings 2007 IEEE international conference on robotics and automation, pp. 3351–3358. Cited by: §I.
  • [60] P. Tsarouchi, G. Michalos, S. Makris, T. Athanasatos, K. Dimoulas, and G. Chryssolouris (2017) On a human–robot workplace design and task allocation system. International Journal of Computer Integrated Manufacturing 30 (12), pp. 1272–1279. Cited by: §I.
  • [61] V. Villani, F. Pini, F. Leali, C. Secchi, and C. Fantuzzi (2018) Survey on human-robot interaction for robot programming in industrial applications. IFAC-PapersOnLine 51 (11), pp. 66–71. Cited by: §I.
  • [62] V. Villani, L. Sabattini, C. Secchi, and C. Fantuzzi (2017) Natural interaction based on affective robotics for multi-robot systems. In 2017 International Symposium on Multi-Robot and Multi-Agent Systems (MRS), pp. 56–62. Cited by: §III-C1.
  • [63] J. Wang, Y. Gu, and X. Li (2012) Multi-robot task allocation based on ant colony algorithm. Journal of Computers 7 (9), pp. 2160–2167. Cited by: §I.
  • [64] M. Westman and D. Eden (1996) The inverted-u relationship between stress and performance: a field study. Work & Stress 10 (2), pp. 165–173. Cited by: §III-C1.
  • [65] G. F. Wilson and C. A. Russell (2003) Operator functional state classification using multiple psychophysiological features in an air traffic control task. Human Factors 45 (3), pp. 381–389. Cited by: §I.
  • [66] S. C. Wong, L. Middleton, B. A. MacDonald, and N. Auckland (2002) Performance metrics for robot coverage tasks. In Proceedings of Australasian Conference on Robotics and Automation, Vol. 27, pp. 29. Cited by: §III-C3.
  • [67] H. A. Yanco and J. Drury (2004) Classifying human-robot interaction: an updated taxonomy. In 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No. 04CH37583), Vol. 3, pp. 2841–2846. Cited by: §I.
  • [68] J. Zhai and A. Barreto (2006) Stress detection in computer users based on digital signal processing of noninvasive physiological variables. In 2006 international conference of the IEEE engineering in medicine and biology society, pp. 1355–1358. Cited by: §III-C1.
  • [69] M. Zhao, F. Adib, and D. Katabi (2016) Emotion recognition using wireless signals. In Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking, pp. 95–108. Cited by: §III-C1.
  • [70] R. Zlot and A. Stentz (2006) Market-based multirobot coordination for complex tasks. The International Journal of Robotics Research 25 (1), pp. 73–101. Cited by: §I.