Multi-Modal Measurements of Mental Load

06/25/2019 ∙ by Ingo Keller, et al. ∙ Heriot-Watt University 0

This position paper describes an experiment conducted to understand the relationships between different physiological measures including pupil Diameter, Blinking Rate, Heart Rate, and Heart Rate Variability in order to develop an estimation of users' mental load in real-time (see Sidebar 1). Our experiment involved performing a task to spot a correct or an incorrect word or sentence with different difficulties in order to induce mental load. We briefly present the analysis of task performance and response time for the items of the experiment task.



There are no comments yet.


page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1. Introduction

There is an ongoing interest in creating robots that can adapt their behaviour according to users traits in the field of Human-Robot/Computer Interaction (HCI/HRI) (Ahmad et al., 2017). User’s Cognitive Load or Operator’s Mental Load are among important attributes during the HCI. It has been found that a higher level of cognitive or mental load of the users may have an adverse effect on their task performance during HCI scenario (Reeves et al., 2004).

The amount of effort placed on a user’s working memory during a task is referred to as the cognitive load. According to the Cognitive Load Theory (Sweller, 1994), there are three different types of load produced during problem-solving or learning: 1) intrinsic load, 2) the extraneous load and 3) the germane load.

  1. Intrinsic load depends on the complexity of the structure of the material and its association with the learner.

  2. The extraneous load is caused by the way this material is presented to the learner.

  3. The germane load is a result of the learner’s ability to assimilate the material.

On the other hand, cognitive load is also defined as a construct that can be measured in three dimensions: 1) mental load, 2) mental effort and 3) performance (Sweller et al., 1998). In other words, past research indicates that cognitive load and mental load are related to each other in terms of the working memory. Therefore, we will be using them interchangeably. It is also interesting to note that the amount of workload varies between individuals (Baddeley, 1992). This suggests that the amount of working memory is different for each individual. Thus, it highlights the significance of measuring the amount of the mental load in real-time in order to adapt the Interface during the HRI or HCI. Consequently, we are also involved in creating the technology that can adapt to the user’s mental load in real-time. More specifically, we want to personalise the behaviour of the robot or adjust the information on the user interface to adjust humans mental overload.

It is currently an open challenge to measure the individual’s mental load in a non-intrusive and robust manner in real-time (Paas et al., 2003; Chen et al., 2012). Existing literature shows that cognitive load can be measured in three different ways: through subjective rating questionnaires (NASA Task Load Index111 NASA Task Load Index - downloads/TLXScale.pdf), through physiological sensors, or through performance-based objective measures (Mathematical Equations). However, subjective rating and performance-based objective measures are limited and cannot be utilised in real-time during HCI or HRI to adjust the interface or the behaviour of the robot. On the contrary, existing literature has highlighted different physiological measures that are continuous and can be used to estimate mental load in real-time (Paas et al., 2003). These measures include Pupil Diameter (PD), Blinking Rate (BR), Heart Rate (HR), Heart Rate Variability (HRV), Electroencephalography (EEG) and Galvanic Skin Response (GSR). Each of these methods has been shown to measure mental load during various tasks; however, each of them has its limitations. For instance: Eye activity (PD, BR) may not be suitable for tasks requiring continuous reading. Similarly, the sensitivity of pupil changes in cognitive load diminishes with age (Van Gerven et al., 2004). Additionally, the HRV may depend on the physical fitness of the individuals (Luque-Casado et al., 2013). It is, therefore, significant to understand the relation between these physiological behaviours under different tasks or circumstances with each other and also onto each other. As our long-term goal is to estimate the user’s mental load in real-time to personalise interfaces or robot’s behaviour, therefore, we believe that understanding the relationship between different physiological behaviour is vital and will help us in achieving our long-term goal of creating a model that takes various physiological behaviours as input to estimate mental overload or cognitive load in an efficient manner.

To achieve our long-term goal, we are currently collecting data from various physiological sensors to understand the relationship between aforementioned physiological behaviours under different task during HCI and HRI respectively (Yaseen and Lohan, 2018; Sabyruly et al., 2015a; Minadakis and Lohan, 2018). Building on the prior work, this position paper describes another series of one of our experiments conducted to understand the relationship between different physiological behaviours during a game-based task. We intend to collect data based on various physiological behaviours (HR, HRV, BR, and PD) during different tasks and later want to create datasets based on these behaviours. Our long-term goal is to use these datasets for a linear mixed-effects regression model to estimate mental or cognitive load in real-time.

2. Related Work

In the past, empirical findings have highlighted a relationship between mental load or cognitive load and various physiological behaviour (PD, HRV, HR, BR, GSR and EEG). It has been shown that the HRV reduces with an increase in the amount of cognitive load during a range of computerised tasks (Mukherjee et al., 2011). Similarly, another study observed an increase in HR with an increase in the difficulty level of the task or in other words in a situation demanding higher cognitive load (Cranford et al., 2014). In addition, we find a body of literature that suggests that pupil diameter increases with an increase in the number of mental load (Reilly et al., 2018; Sabyruly et al., 2015b). Moreover, research indicates that the average number of blinks per minute in normal healthy adults varies according to the task. A study conducted with 150 adults showed that individuals on average blink four to five times per minute during a reading activity. Similarly, the rate of blinking per minute is higher during resting (17) and during conversation (26) (Bentivoglio et al., 1997). Later, it has been shown that the blinking rate is minimised in the case of higher mental overload (Holland and Tarlow, 1972).
In summary, we find enough empirical evidence from past research implying that the change in the physiological behaviour can be attributed to cognitive load or mental load and it also refers to various levels of mental processing. It is also important to note that the data collected on physiological behaviours through the various state of the art sensors is not only continuous but is also a robust and accurate representation of the particular behaviour (HRV, 2018; Tommaso, 2018). It has been highlighted that users are required to wear cumbersome equipment; however, with the advancement of design and technology, existing solutions have been devised to collect such data in non-invasive ways.

In relation to our long-term goal, we currently find empirical evidence that few of the physiological behaviours are correlated with each other. For instance, both pupil dilation and blink-eye rate were seen to be correlating with each other in a digit-sorting task (Siegle et al., 2008). However, to the best of our knowledge, the relationship between all these behaviours have not been explored and is necessary to efficiently create an estimation for cognitive load or mental load. As highlighted we need to understand the relationships between different behaviours because this can help in creating a linear mixed-effect regression model that may estimate mental load in real-time during HCI or HRI. Consequently, We, in this paper, presents intial finding of a study conducted in order to collect data on these behaviours in a synchronous way to help us analyze the relationships between different physiological behaviours and later create a dataset that can be used to estimate cognitive load in real-time.

3. Study

The purpose of the study was to understand the relationship between different physiological measures (HR, HRV, PD and BR) to use these variables to help estimate mental load in real-time.

3.1. Experiment Protocol

The experiment was conducted with 41 participants (demographics as shown in table 1) in seven different steps:

  • Participant fills a Consent Form.

  • Participant fills a General Questionnaire to report information on age, numbers of languages they speak and in case they have reading difficulties and a Physical Activity Questionnaire (Fogelholm et al., 2006) to control for any bias in the HR and HRV measurements.

  • Participant wears Tobii Eye Tracker and performs a simple Calibration to get the values for their Pupil Diameter.

  • Participant wears corSense Heart-Rate Device and performs the task.

  • Participant plays spot the real or made-up word and sentence game.

  • Mental load is measured from Tobii Glasses 2 (BR, PD), Heart Rate Monitor (HR, HRV), and Webcam facing the participant during the game task.

  • Participant fills the NASA TLX Questionnaire.

idth=.5]setup.jpg Setup: Participant is performing the game based task

3.2. Task

The task given was a correct or an incorrect English word or sentence recognition task with different difficulties. The participants were presented with items from the following categories. All word items were 10 letters long. All sentence items consisted of 10 words each.

Item Type Content
1 a correct English word
2 as 1 but with the middle letters switched
3 as 1 but with scrambled letters
4 an arbitrary mnemonic word
5 a correct English sentence from a movie review dataset (Pang and Lee, 2004)
6 as 5 but with rearranged words rendering them incorrect

Table 1: Task Item Overview

3.3. Initial Results

The following table shows the first analysis of task performance and response time for the items of the experiment task.

Participants 41
Gender 20 Female / 21 Male
Age 18-37 (Mean: 23.3, 2 unreported)
Average Task Performance (120 Items) 111.7 (SD=5.72)
Average Task Performance - Items 1-4 75.51 (SD=3.91)
Average Task Performance - Items 5-6 36.19 (SD=2.54)
Mean Task Response Time - Items 1-4 1.03s (SD=0.34)
Mean Task Response Time - Items 5-6 4.03s (SD=1.60)

Table 2: Initial results with participant’s demographics

We are currently creating the dataset that will be comprised of eye and front camera video and the pupil dilation data from the eye tracker, the data from the heart rate monitor, and the video from a webcam facing the participants. Due to in-completion of the dataset, we are not reporting the results in this paper but give an overview of some of its properties.

3.4. Conclusion and Future Work

In the paper, we presented the initial results of our study conducted to understand the relationship between various physiological behaviours (HR, HRV, BR, PD) during an HCI task. We are currently analysing the data on all these behaviours, and our long-term goal lies in the development of a system that could potentially provide a robust non-intrusive measurement of mental workload in real-time during HCI. Our work is focused on creating a dataset of these physiological behaviours and later using them for a regression-based model to estimate mental load during various HCI and HRI scenarios.

We acknowledge funding and support from the EPSRC ORCA Hub (EP/R026173/1, 2017-2021) and consortium partners.


  • (1)
  • Ahmad et al. (2017) Muneeb Ahmad, Omar Mubin, and Joanne Orlando. 2017. A systematic review of adaptivity in human-robot interaction. Multimodal Technologies and Interaction 1, 3 (2017), 14.
  • Baddeley (1992) Alan Baddeley. 1992. Working memory. Science 255, 5044 (1992), 556–559.
  • Bentivoglio et al. (1997) Anna Rita Bentivoglio, Susan B Bressman, Emanuele Cassetta, Donatella Carretta, Pietro Tonali, and Alberto Albanese. 1997. Analysis of blink rate patterns in normal subjects. Movement Disorders 12, 6 (1997), 1028–1034.
  • Chen et al. (2012) Fang Chen, Natalie Ruiz, Eric Choi, Julien Epps, M Asif Khawaja, Ronnie Taib, Bo Yin, and Yang Wang. 2012. Multimodal behavior and interaction as indicators of cognitive load. ACM Transactions on Interactive Intelligent Systems (TiiS) 2, 4 (2012), 22.
  • Cranford et al. (2014) Kristen N Cranford, Jessica M Tiettmeyer, Bryan C Chuprinko, Sophia Jordan, and Nathaniel P Grove. 2014. Measuring load on working memory: the use of heart rate as a means of measuring chemistry students’ cognitive load. Journal of Chemical Education 91, 5 (2014), 641–647.
  • Fogelholm et al. (2006) MIKAEL Fogelholm, JARMO Malmberg, Jaana Suni, MATTI Santtila, Heikki Kyröläinen, Matti Mäntysaari, and Pekka Oja. 2006. International physical activity questionnaire: validity against fitness. Medicine and science in sports and exercise 38, 4 (2006), 753–760.
  • Holland and Tarlow (1972) Morris K Holland and Gerald Tarlow. 1972. Blinking and mental load. Psychological Reports 31, 1 (1972), 119–127.
  • HRV (2018) Elite HRV. 2018. How do you calculate the HRV score? Webpage. {}
  • Luque-Casado et al. (2013) Antonio Luque-Casado, Mikel Zabala, Esther Morales, Manuel Mateo-March, and Daniel Sanabria. 2013. Cognitive performance and heart rate variability: the influence of fitness level. PloS one 8, 2 (2013), e56935.
  • Minadakis and Lohan (2018) Georgios Minadakis and Katrin Lohan. 2018. Using Pupil Diameter to Measure Cognitive Load. arXiv preprint arXiv:1810.07593 (2018).
  • Mukherjee et al. (2011) Shalini Mukherjee, Rajeev Yadav, Iris Yung, Daniel P Zajdel, and Barry S Oken. 2011. Sensitivity to mental effort and test–retest reliability of heart rate variability measures in healthy seniors. Clinical Neurophysiology 122, 10 (2011), 2059–2066.
  • Paas et al. (2003) Fred Paas, Juhani E Tuovinen, Huib Tabbers, and Pascal WM Van Gerven. 2003. Cognitive load measurement as a means to advance cognitive load theory. Educational psychologist 38, 1 (2003), 63–71.
  • Pang and Lee (2004) Bo Pang and Lillian Lee. 2004.

    A Sentimental Education: Sentiment Analysis Using Subjectivity Summarization Based on Minimum Cuts. In

    Proceedings of the ACL.
  • Reeves et al. (2004) Leah M Reeves, Jennifer Lai, James A Larson, Sharon Oviatt, TS Balaji, Stéphanie Buisine, Penny Collings, Phil Cohen, Ben Kraal, Jean-Claude Martin, et al. 2004. Guidelines for multimodal user interface design. Commun. ACM 47, 1 (2004), 57–59.
  • Reilly et al. (2018) Jamie Reilly, Alexandra Kelly, Seung Hwan Kim, Savannah Jett, and Bonnie Zuckerman. 2018. The human task-evoked pupillary response function is linear: Implications for baseline response scaling in pupillometry. Behavior research methods (2018), 1–14.
  • Sabyruly et al. (2015a) Yerlikzhan Sabyruly, Frank Broz, Ingo Keller, and Katrin Solveig Lohan. 2015a. Gaze and Attention During an HRI Storytelling Task. Artificial Intelligence for Human-Robot Interaction : AAAI 2015 Fall Symposium Series, AI-HRI 2015 ; Conference date: 12-11-2015 Through 14-11-2015.
  • Sabyruly et al. (2015b) Yerlikzhan Sabyruly, Frank Broz, Ingo Keller, and Katrin Solveig Lohan. 2015b. Gaze and Attention During an HRI Storytelling Task. In 2015 AAAI Fall Symposium Series.
  • Siegle et al. (2008) Greg J Siegle, Naho Ichikawa, and Stuart Steinhauer. 2008. Blink before and after you think: blinks occur prior to and following cognitive load indexed by pupillary responses. Psychophysiology 45, 5 (2008), 679–687.
  • Sweller (1994) John Sweller. 1994. Cognitive load theory, learning difficulty, and instructional design. Learning and instruction 4, 4 (1994), 295–312.
  • Sweller et al. (1998) John Sweller, Jeroen JG Van Merrienboer, and Fred GWC Paas. 1998. Cognitive architecture and instructional design. Educational psychology review 10, 3 (1998), 251–296.
  • Tommaso (2018) Davide De Tommaso. 2018. Tobii Pro Glasses 2 Python controller. Webpage. {}
  • Van Gerven et al. (2004) Pascal WM Van Gerven, Fred Paas, Jeroen JG Van Merriënboer, and Henk G Schmidt. 2004. Memory load and the cognitive pupillary response in aging. Psychophysiology 41, 2 (2004), 167–174.
  • Yaseen and Lohan (2018) Abdelrahman Yaseen and Katrin Lohan. 2018. Playing Pairs with Pepper. arXiv preprint arXiv:1810.07593 (2018).