Human-Robot Teams (HRT) are becoming increasingly prominent in applications such as autonomous cars , disaster response , and remote inspection . In such HRT, humans employ their knowledge and experience about the system, along with information from the user interfaces, to gain awareness of the robot’s status. In contrast, the robot’s AI agent does not possess the ability to estimate the human’s status by default and it should be explicitly provided. Of critical importance w.r.t the human’s status, is the level of workload that they experience during task execution. Research shows that using operator workload estimates, has a positive impact on the overall HRT performance [24, 2]. Also, in cases where cognitive workload rises, catastrophic consequences could occur to the human involved, to the robot, and even to the entire mission . Hereupon, agents understanding the state of each other, translates into a more efficient HRT interaction.
This work focuses on cognitive workload (referred to as workload for the rest of the paper) in the context of remotely operated robots, commonly used in safety-critical scenarios such as disaster response. For efficient Human-Robot Interaction (HRI), the robot’s AI should not only have awareness of the mission parameters but also be cognizant of the operator’s performance and status. Evidence shows a strong correlation between performance and workload [7, 21]. Thus, methods enabling the estimation of cognitive workload could prove to be particularly important in HRT and especially in variable autonomy systems. For instance, the robot’s AI agent could issue warning indications to Human-Initiative systems ) or adjust its policies to actively alleviate the burden of control in Mixed-Initiative [3, 18] and shared control systems ). In such cases, behavioral entropy is used as an index of behavioral predictability to characterise operator workload induced by mentally taxing tasks. [24, 8].
We propose a method for the estimation of the operator’s workload in real-time, via the use of behavioural entropy called Fessonia333Fessonia is the Roman Goddess who aids the weary. Our method is non-intrusive as it uses the control input of the operator to estimate the workload, an advantage over more intrusive workload estimation methods such as using physiological measures (e.g. EEG  and heart rate ). Additionally, compared to post-hoc subjective measures of workload such as the NASA-TLX , our method offers real-time estimation.
We consider the work of Nakayama et al. , who developed an offline workload estimation system, that utilises steering data to estimate the operator workload. In this work we propose the following novelties. First, Fessonia is multidimensional as multiple parameters of the operator’s driving profile (e.g. angular and linear velocity commands) can be used. Second, our method is handling the measurements in real-time resulting in an online estimation of workload. Third, Fessonia is capable of updating online the operator’s driving behavior profile model. We consider the latter to be the main contribution of this paper as it allows the system to adapt its entropy calculation according to the level of the operator’s adaptiveness vis-à-vis system and mission familiarity which significantly affect performance. Dynamically adjusting the robot’s understanding of the operator driving profile allows for a more sensitive workload estimation for high performing operators. Lastly, a Warning And Indication System (WAIS) was developed. This system continuously monitors the workload of the operator and when it exceeds a defined threshold, the system provides an audio-visual indication that suggests a more attentive driving approach.
Ii Related Work
Early work utilised the entropy of steering data (angle of steering) and is mostly focused on vehicle driver workload estimation [1, 16]. Nakayama et al.  presented a methodology based on the assumption that the driver’s steering behaviour tends to become more unpredictable, under heavy workload. They proposed a metric based on the steering angle and steering angle estimations, to formulate a methodology that detects unpredictable behaviour using behavioural entropy and thus quantify the driver’s workload. In , they used a similar methodology with , but they substituted the Taylor expansion approximation with a second-order auto-regressive model. This method is more sensitive to roads with high curvatures or roads with sudden turns.
Other related work is focused on virtual reality systems. In  they presented a methodology that combines sample entropy and template matching, applied in a virtual reality system. Their proposed methodology allows for the creation of different input templates, and the entropy is calculated based on the number of similar templates. Even though this method is computationally expensive, it provides a relative noise-free multi-dimensional metric for workload estimation.
One metric that can be used in real-time is presented in 
and is called Discrete N-Dimensional Entropy of behaviour (DNDEB). The discretized inputs are used in combination with a long short-term memory network to produce confusion matrices for a predefined data window, and calculate entropy based on those confusion matrices. The advantages of this work are that the metric accepts discrete inputs of any size, and is tunable to the specific application. In the main focus was to use entropy as a means to estimate the performance of human-machine rather than operator workload.
Our research shares similarities with approaches that employ behavioural entropy to estimate the workload of vehicle drivers [16, 1], while further being motivated by works that appraise human workload in virtual reality  and human-robot team performance in robotic wheelchairs . To the best of the authors’ knowledge, most related work focuses on using entropy as a post-hoc measure of human workload, except for  where they used entropy to develop a collision prediction mechanism. Furthermore, previous research assumes that after the initial training of the operator, the levels of familiarity, experience, and driving ability to stay static [1, 24]. In this work, we aim to address these issues by proposing a framework for real-time workload estimation that takes into consideration the operator’s current condition and evolving driving profile. Furthermore, it uses said estimates on a warning indication system that issues warnings during elevated operator workload.
Iii Problem Formulation and Development
We consider the problem of estimating the workload of an operator, controlling a remotely situated robot and conducting a mentally demanding task. For example, a search and rescue scenario where a human-robot team is assigned to navigate and explore an environment with the objective to detect victims. These situations are very taxing and stressful to human operators as they often require multitasking on top of the cognitively demanding robot control .
Similar to , we make an informed assumption that high operator workload correlates with unpredictable driving behaviour profile. While the cognitive load imposed increases, the operator is likely to make errors during the primary task (i.e. robot navigation), and thus is more prone to resort to abrupt error corrective commands and more specifically jerky velocity commands of the joypad control, i.e. linear command velocity and angular command velocity .
To detect these jerking motions we use a second-order Taylor expansion model (Eq. 1) that estimates a typical expected operator workload profile (referred to as typical operator workload), using operator command parameters such as linear and angular velocity commands. We used behavioural entropy in the estimation errors since it is a commonly used measure of unpredictability, i.e. high entropy reflects high workload.
Iii-a Behavioural Entropy With Multiple Dimensions
Initially, we defined the parameters that are required to calculate behavioural entropy, which are the angular and linear velocity commands of the operator (Eq. (2)).
These commands were sampled at a rate of 20 Hz and a three sample moving average filter was used, in order to reduce measurement noise. The frequency of filtered samples were roughly 7 Hz, which is equal to the humans manual tracking delay [11, 20]. We used a second-order Taylor expansion to model the typical driving profile at time-step n (Eq. 1). Let and be the estimated linear command velocity and the estimated angular command velocity of the robot at time-step n respectively. Using the model from Eq. 1, the estimation errors were calculated:
For calculating the entropy, the 90th percentile (value ) of the estimation error frequency distribution was calculated. One example of the distribution of estimation errors from our trials can be seen in Fig. 1. The value is indicative of the different operator driving profiles and ultimately operator workload. When an operator deviates from typical behaviour, the prediction errors recede from zero, making the estimation error distribution wider (Fig. 2) and bigger.
Using the value of the distribution, we divided the distribution into 9 bins according to , resulting in 8 bin boundaries .
The operators need to first undergo a trial-run session (see IV-B), in which they operate the robot without any specific mission task apart from driving, to calculate the initial baseline driving behaviour profile. We use the entropy , defined as:
The values denote the frequency of the errors that fall in the respective bins while denotes the number of bins. In our implementation the entropy calculation is two dimensional , as the input has one linear () and one angular () velocity command dimension. The proposed method has the additional advantage of being designed for modularity. This can be attributed to the fact that it can be trivially augmented with the inclusion of more inputs and thus more dimensions. Here, the two dimensions of the entropy are considered equally significant to the estimated Total Entropy (Eq. 4).
The estimation errors are calculated every 0.15 seconds however, the entropy calculation occurs periodically with a defined frequency. We empirically observed the average corrective manoeuvre span to be less than .25 seconds and chose entropy computation period (0.4 Hz). Entropy calculation less than 0.2 Hz or more than 0.4 Hz results in a slow, or unreliable workload estimation respectively. This frequency also affects the batch size of estimation errors included in the entropy computation. To elaborate, when the entropy is calculated every 3 seconds, the estimation errors handled are .
Iii-B Driver Profile Update
In this work, we explicitly consider the: adaptation of humans to different situations; individual differences in task-related skills between operators; evolvement of task-related skills over time. It is natural for an operator to be constantly improving their performance by experience and having the ability to adapt to dynamic environments. A static operator profile will produce estimation errors that falsely indicate low workload when high workload is imposed, as these errors lie in closer to zero bins, rather than further from zero bins.
In our work, we developed an algorithm called Driver Profile Update (DPU) that monitors the entropy history and updates the operator profile characteristics ( and bins) accordingly. If no prior knowledge is available, the initial thresholds (i.e. average and std variables) can be defined as extremely high values. We used entropy history acquired from a familiarization session (Section IV-B
). If the average and standard deviation of the 100 previous entropy history time-steps become lower than the initial thresholds, DPU saves the last 100 estimation errors to the error history and utilises said error history to calculate an updated value forand new bins (see Algorithm 1). The defined thresholds are also updated to the newly updated average and standard deviation of the last 100 entropy history.
Iii-C Warning And Indication System (WAIS)
To assist the operator in their mission, a Warning And Indication System (WAIS) is proposed. WAIS continuously monitors the entropy calculations and issues warning indications, according to the entropy produced by the operator’s manoeuvres. When the operator manoeuvres through a challenging turn, they might momentarily experience high workload. To alleviate the effect of those sudden entropy spikes, we use a moving average filter of five computations.
WAIS informs the operator about their workload with two indications, one for normal and one for erratic operation (see Fig. 4). The operator sees a NORMAL OPERATION indication until the moving average is greater than a defined threshold, which was empirically assigned at 0.6 to capture severe workload escalations. We concluded to this threshold empirically by monitoring the entropy levels during no-workload test runs, where the entropy spikes did not exceed 0.6. An audio-visual warning message is issued on the GUI, to alert of a possible detrimental operation. The audio warning is a soft ”ping” sound to attract the attention of the operator. The visual warning is a HIGH WORKLOAD indication that substitutes the ”NORMAL OPERATION” indication in the GUI.
Iii-D Method overview
The above-presented methods are handled by three subsystems, namely the asynchronous entropy calculation (including sampling, estimation model, estimation error calculation and initial baseline driving behaviour profile calculation), the DPU, and the WAIS (see Fig. 3). What follows is the testing and evaluation of the above subsystems.
Iv Experimental study
We conducted a preliminary experimental evaluation of the proposed methods on a mission inspired by disaster response and remote inspection scenarios, where an operator controlling a remote robot, is tasked with exploring an environment (e.g. for points of interest such as people trapped in debris). The experiments aimed to evaluate the: a) accuracy of the workload estimation and how it quantifies different levels of workload; b) ability of the method to detect the adapting driving profile of human operators; c) effect that the warning indication system has on behavioural entropy and as a consequence on the workload of an operator.
Iv-a Apparatus and Software
All software was developed in the Robot Operating System (ROS) and is available in our code repository444The code for the experiments is provided under MIT license in the Extreme Robotics Lab GitHub repository: https://github.com/uob-erl/hrt_entropy. The environment and the robot were simulated in Gazebo. The simulated robot was a Husky model equipped with a camera to provide a video feed and a 2D LiDAR laser sensor for localization. The specific experimental setup based on simulation was selected because repeatability was considered of high importance. Moreover, during the operation of remote robots, the interface/cockpit and the operator’s experience is almost identical in simulation of this work as it is in reality. As Gazebo is a high-fidelity robotic simulator the realism or meaningfulness of the results were not compromised.
Participants operated the robot via a joypad while acquiring situation awareness (SA) via a GUI (see in Fig. 4). The GUI provides information about the position of the robot relative to the map, an operation status indication (shows if the robot is stopped or teleoperated), a workload warning indication, and video streaming from the robot’s onboard camera. Moreover, the GUI has a secondary window for the high workload secondary task (described below) used to evaluate the impact of workload on entropy. Finally, the operator used a computer mouse to execute the secondary tasks that are designed to assess the workload estimation (see Section IV-B).
Iv-B Experimental Protocol
Three participants took part in this study. Two of the participants had previous experience with similar control configurations and interfaces while the third one was a complete novice. Before the experiment, participants were trained by driving around the robot arena for five minutes to get familiar with the system and accustomed to the controls (see Fig. 5). After said familiarization, the participants navigated the arena, without any secondary tasks, for 10 minutes to develop their initial baseline driving behaviour profile (i.e. the value of ). Each participant performed 3 experimental trials. In each trial, the participants were required to navigate the robot from point A to point B of the map (Fig. 5) four times, except from the first experiment trial which included 12 round trips from A to B. Between each experiment, the participants took a 15-minute break to minimize their fatigue levels, which could affect the level of workload estimates. The instruction given to the participants was to navigate the robot at a comfortable pace while avoiding collisions.
In the first trial, participants were tasked to operate the robot focusing only on the navigation task. This experiment was conducted to evaluate the DPU system. In the second experiment, the workload estimation algorithm was exclusively evaluated. To validate the impact that workload has on entropy, three secondary tasks were introduced in sequence, from higher to lower workload impact. The cognitive impact the secondary tasks inflict might vary for each participant. To to be consistent and normalise the impact across all participants, we gave specific instruction to perform the secondary tasks at a comfortable pace with no time restrictions. The following tasks were inspired by previously validated secondary tasks from the literature ( ). The High workload task includes pressing one out of four buttons, upon request on a secondary GUI window. The secondary GUI will play a sound and request the press of a specific button. The Medium workload task is counting down from 500 by subtracting 3. Finally, during the Low workload task the maximum speed of the robot was doubled and as a result, the robot became more sensitive to small control commands. The high workload secondary task proposed here simulates operating a touch panel tested in  or the press of a button from , and the medium workload secondary task is similar to the mental arithmetics from [16, 1].
In the last experiment, the participants were required to navigate the arena, while the high workload imposing secondary task was in effect. The instructions were the same as before but when an indication was issued, participants were instructed to ignore completely the secondary task and continue conducting the primary task (operating the robot), until the warning indication ceased. This experiment was designed to evaluate the effect WAIS has on the operator workload. Across all experiments, the focus was to investigate the effect of the secondary tasks on the operators’ workload rather than performance on tasks.
|Baseline trial portion||Low workload trial portion||Medium workload trial portion||High workload trial portion|
Entropy descriptive statistics for three participants on three sequentially increasing workload imposing tasks
In the first trial, results show that the value (operator driving profile) is decreasing over the duration of the experiment, suggesting a predictable behaviour (Fig. 6). The second trial showed that the multi-dimensional entropy was proportional to the different levels of imposed workload (see Table I). Finally, from the third trial, three examples of issued warnings from the third 45-minute trial are presented for each operator (Fig. 7). These examples show a duration of an average 10 seconds, to decrease the entropy after a warning was issued.
Vi Discussion, Limitations and Future Work
Evidence from the preliminary evaluation suggests that the proposed method estimates successfully the workload using multiple control command parameters given by the operator. The descriptive statistics presented in Table I show a considerable distinction between the secondary tasks with different workload impacts. This is in accordance with results presented in the literature [1, 16, 24, 19] which show that entropy can capture different levels of workload. Due to the real-time functionality of the method, the entropy calculation, in some situations, could be zero. For instance, where the operator follows a typical driving profile and thus is predictable, all estimation errors will fall in the bin around zero. In other words, the rate of errors will be 1 for one specific bin, resulting in an entropy estimation equal to , and in consequence in greater standard deviation. Nonetheless, the standard deviation does not have a substantial effect on the workload estimation. Results suggest that the zero entropy might slightly increase the standard deviation, but the standard deviation is a stronger indication of individual driving differences among the participants.
The DPU system results support our initial assumption, which is that the operators can adapt to certain situations and control schemes, in addition to becoming more competent in the task over time. The characterises the driving profile of the operator. Smaller would suggest a more predictable driving profile (i.e. typical operation), and over time, decreasing suggests that the operator becomes more predictable. One can argue that operators who are becoming more predictable are essentially adapting their driving profile, becoming more familiar with the task, and increasing the overall performance of the HRT. Every participant showed a different adapting process, but all of them adapted on some level. In addition, each experiment was conducted for 30 minutes and did not fully converge for any of the participants. This means that they could get even more acclimated to the situation. The results also suggest that the initial trial-run session to calculate the value is redundant (which was required in previous literature [1, 16]). The system can start with a relatively high value and the system can adjust this value due to the DPU if the operator consistently produces lower entropy computations.
The WAIS trials showed promising results. In the experiments, the participants were instructed to completely ignore the secondary task when a warning was issued. The results present a 21.3% average decrease in entropy across all participants. Specifically, Participant one had a 21% decrease in , Participant 2 had a 14% in and Participant 3 had an 18% decrease in value. The entropy reduction was more than expected since the main instruction was to only focus on the navigation task. In a real-world scenarios the operator could be multitasking during a mission and thus a warning could make them aware of critical performance and safety impairment factors. Drawing the operator’s attention aims to remind them to focus on the safe driving of the robot to avoid potential accidents. Depending on the mission, the existence of a specific protocol to be followed in the case of a warning will be beneficial. Additionally, WAIS displays the sensitivity of the entropy computation to driving behaviours. The participants become aware of the deteriorated performance upon indication and change their driving behaviour which is encapsulated by the reduced entropy.
Assuming that all control command dimensions equally contribute to the total entropy, might not hold true for different applications and environments. In future work, a dynamic thresholding system could assign thresholds proportionally to the correlation coefficient of each control command dimension to entropy. Despite the promising results of WAIS, in a real high-stress scenario, a warning system could potentially increase the workload and stress levels of an operator if not integrated carefully with the mission protocols. More research is needed to appropriately design WAIS, to minimize the stress and workload imposed by the indication itself, while preserving the value of the warning itself. Lastly, high entropy estimations could be the result of increased operator fatigue over time, rather than increased workload. Our experimental design aimed to minimize this possibility. However, in future work, to differentiate between fatigue and workload, measurements from other non-intrusive methods (e.g. eye-tracking etc) can be used. In literature there have been examples of using eye tracker systems to detect fatigue levels . Our system could benefit from measurements of such detectors.
In this study, we propose Fessonia, a method for estimating the cognitive workload of an operator controlling a robot, via the use of behavioural entropy. This method’s contribution is the capability of estimating operator workload given multiple parameters of the operator’s driving profile. An additional contribution is the Driver Profile Update (DPU) algorithm which continuously observes the performance history of the operator and adapts the workload estimation based on the adapted operator profile. Lastly, a warning indication system (WAIS) is presented, that advises the operator based on workload estimates. Fessonia was evaluated in a navigation scenario. The results indicate that our proposed method captures and successfully estimates different levels of workload. Furthermore, our proposed DPU algorithm can detect individual driving profiles, improvements in operator performance, and adapts accordingly. Hence, the workload estimation becomes more responsive to the level of competence of the operator. Lastly, the use of WAIS resulted in an average 25% entropy reduction across all participants. The future work’s focal point is to better improve our method by redesigning WAIS to minimize any potential stress and workload imposed by the suggestion itself. Furthermore, it is under consideration to include additional dimensions, that would provide Fessonia with the ability to distinguish between operator fatigue and imposed workload. Lastly, the promising experimental results, enable the future integration of our method in variable autonomy human-robot teams. Said system could utilise these entropy calculations to actively support the human operator, e.g. by completely relinquishing control from the operator over to the robot’s AI agent.
-  (2000) Behavioral entropy as an index of workload. In Proceedings of the XIVth Triennial Congress of the International Ergonomics Association and 44th Annual Meeting of the Human Factors and Ergonomics Association, ’Ergonomics for the New Millennium’, pp. 125–128. External Links: Cited by: §II, §II, §IV-B, §VI, §VI.
-  (2012) Collaborative control for a robotic wheelchair: evaluation of performance, attention, and workload. 42 (3), pp. 876–888. External Links: Cited by: §I.
-  (2021) Mixed-Initiative variable autonomy for remotely operated mobile robots. ACM Transactions on Human-Robot Interaction 10 (4). Cited by: §I.
-  (2016) Experimental analysis of a variable autonomy framework for controlling a remotely operating mobile robot. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3581–3588. External Links: Cited by: §I.
-  (2009) Risk behaviour and mental workload: multimodal assessment techniques applied to motorbike riding simulation. Transportation Research Part F: Traffic Psychology and Behaviour 12 (5), pp. 361–370. External Links: Cited by: §I.
-  (2010) The search for survivors: cooperative human-robot interaction in search and rescue environments using semi-autonomous robots. In 2010 IEEE International Conference on Robotics and Automation, Vol. , pp. 2858–2863. External Links: Cited by: §III.
-  (2017) The impact of workload and fatigue on performance. In Human Mental Workload: Models and Applications, L. Longo and M. C. Leva (Eds.), Cham, pp. 90–105. External Links: Cited by: §I.
-  (2004) Behavioral Entropy in Human-Robot Interaction. pp. 24–26. Cited by: §I.
-  (1988) Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. 1 (3), pp. 139–183. Cited by: §I.
-  (2018) A survey of workload assessment algorithms. 48 (5), pp. 434–451. External Links: Cited by: §I.
-  (1966) A “critical” tracking task for manual control research. IEEE Transactions on Human Factors in Electronics HFE-7 (4), pp. 138–145. External Links: Cited by: §III-A.
-  (2013) Mathematical foundations of information theory. Dover books on advanced mathematics, Dover, New York, NY. External Links: Cited by: §III-A, §III-A.
-  (2014) Designing, developing, and deploying systems to support human–robot teams in disaster response. Advanced Robotics 28 (23), pp. 1547–1570. External Links: Cited by: §I.
-  (2018) Human-robot interaction and self-driving cars safety integration of dispositif networks *. In 2018 IEEE International Conference on Intelligence and Safety for Robotics (ISR), Vol. , pp. 494–499. External Links: Cited by: §I.
-  (2011-10) Robot-assisted bridge inspection. Journal of Intelligent and Robotic Systems 64, pp. 77–95. External Links: Cited by: §I.
-  (1999) Development of a steering entropy method for evaluating driver workload. SAE Technical Papers (724). External Links: Cited by: §I, §II, §II, §III, §IV-B, §VI, §VI.
-  (2020) VFH+ based shared control for remotely operated mobile robots. External Links: Cited by: §I.
-  (2020) Human operator cognitive availability aware mixed-initiative control. In 2020 IEEE International Conference on Human-Machine Systems (ICHMS), Vol. , pp. 1–4. External Links: Cited by: §I.
-  (2019) Entropy of controller movements reflects mental workload in virtual reality. 26th IEEE Conference on Virtual Reality and 3D User Interfaces, VR 2019 - Proceedings, pp. 802–808. External Links: Cited by: §II, §II, §VI.
-  (1974) Man-machine systems: information. Control, and Decision Models of Human Performance. MIT press. Cited by: §III-A.
-  (2009-06) Robotic assistance improves intracorporeal suturing performance and safety in the operating room while decreasing operator workload. 24, pp. 377–82. External Links: Cited by: §I.
Classification of a driver’s cognitive workload levels using artificial neural network on ecg signals. 59, pp. 326–332. External Links: Cited by: §I.
-  (2018) Detecting mental fatigue from eye-tracking data gathered while watching video: evaluation in younger and older adults. Artificial Intelligence in MedicineSurgical endoscopyFrontiers in Human NeuroscienceApplied ErgonomicsIEEE Transactions on Human-Machine SystemsIEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics)IEEE International Conference on Intelligent Robots and SystemsProceedings of Performance Metrics for Intelligent SystemsarXiv preprint 1911.04848, To appear in journalHuman mental workload 91, pp. 39–48. External Links: Cited by: §VI.
-  (2019) Discrete N-Dimensional Entropy of Behavior: DNDEB. pp. 2227–2233. Note: Yellow is notes and questions. Red is search about this. Purple is methodology. Green is results. Grey is Discussion. External Links: Cited by: §I, §I, §II, §II, §VI.