An Open Platform for Research about Cognitive Load in Virtual Reality

The cognitive load can be used to assess if someone is struggling while performing a task. It can be used in many different situations such as in driving, piloting, studying, playing, working, etc. This information can help to design better systems and even to create interactive systems that can be aware of the user's cognitive load and adapt itself to the user. We propose an open source platform that can be used for doing research about cognitive load in virtual reality (VR). Our platform can be used for stimulating cognitive load through several VR scenes and for analyzing cognitive load through objective and subjective measurements.

READ FULL TEXT VIEW PDF

Authors

page 1

page 2

05/31/2020

A Virtual Obstacle Course within Diverse Sensory Environments

We developed a novel assessment platform with untethered virtual reality...
08/04/2019

Epistemological approach in immersive virtual environments and the neurophysiology learning process

Currently virtual reality (VR) usage in training processes is increasing...
04/27/2020

Wireless VR/Haptic Open Platform for Multimodal Teleoperation

With emerging trends in the fifth generation and robotics, the Internet ...
05/13/2022

AVCAffe: A Large Scale Audio-Visual Dataset of Cognitive Load and Affect for Remote Work

We introduce AVCAffe, the first Audio-Visual dataset consisting of Cogni...
11/07/2021

A Virtual Reality Simulation Pipeline for Online Mental Workload Modeling

Seamless human robot interaction (HRI) and cooperative human-robot (HR) ...
04/27/2020

Facial Electromyography-based Adaptive Virtual Reality Gaming for Cognitive Training

As life expectancy rises, age-related diseases causing dementia become m...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Virtual Reality (VR) is a promising technology that the industry is starting to use more and more for professional training. Systems that are expensive, dangerous or not easily available can be modeled and inexperienced users can practice with them safely. VR is also a useful tool for psychologists as they can reproduce specific experiments to analyze the user’s mind and behavior. Detecting automatically the mental workload in real time can help to design interactive systems that can adapt themselves to the user’s needs. But it is still a research challenge, and open source platforms implementing cognitive load measurements and giving access to prefabricated scenes will hopefully help researchers.

In this paper we present a platform that can be used for stimulating and measuring cognitive load in virtual reality (VR). Our platform is open source and available on two GitLab repositories: one with a detailed documentation for developers111https://git.enib.fr/g7broche/vr-ppe and another one ready to use for standard users222https://git.enib.fr/g7broche/vr-ppe-build.

The following sections will present what kind of measurements can be done with the proposed platform and then the VR scenes that have been implemented in the platform to stimulate the cognitive load.

2 Cognitive load measurement

The cognitive load theory is based on human cognition and is characterized by the central assumption that the amount of knowledge acquisition depends on the efficiency of the use of available cognitive resources in working memory. A person’s cognitive load is the mental effort associated while performing a task or engaging in a learning process. The cognitive load measurements are usually divided into two main categories: subjective measures and objective measures.

2.1 Subjective measures

Most research on cognitive load is using the subjective measurement to obtain the participant’s mental effort. It is subjective because is related to the user’s own perception of the amount of mental effort used while performing a task. The user is asked to answer some questions in order to estimate their level of cognitive load.

One of the most famous subjective rating scales is the NASA Task Load Index (NASA-TLX) [3]. The NASA-TLX is a subjective rating scale, originally designed to analyze the mental and physical demands of aircraft pilots, with six different items : mental demand, physical demand, temporal demand, performance, effort, and frustration level. The Figure 1 shows the NASA-TLX implemented in our platform.

Figure 1: A visualization of the NASA-TLX questionnaire implemented in our VR platform.

Another commonly used scale is the one proposed by Paas [5]. It consists of one item for task difficulty and one item for mental effort on a 9-point Likert scale. These scales are placed for the participant to answer after each completed task.

The advantage of subjective measures is that the person who performed the task is the only one who can estimate the difficulty of the task faced, and the effort to complete it. However, there are technical measurement problems with regard to the objectivity, validity and reliability of the participants’ own responses.

2.2 Objective measures

The objective measurement is a way to obtain the user’s amount of mental effort by direct methods. These direct methods can be obtained through participant’s physiological information [2, 4] or a dual-task performance [6, 1].

The analysis of physiological data obtained from the participant, such as brain activity, pupil movement and dilation, heart rate, electrodermal activity, etc. can be used to indicate the amount of mental effort made by the user. Physiological data can be an accurate indicator of the cognitive load, but the sensor’s raw data usually need to be processed, features need to be extracted and sometimes machine learning models need to be trained before being able to predict the amount of mental effort based on these signals. Our platform exports all the physiological signals available from the VR headset for this purpose.

The dual-task performance method is another approach where the participants are engaged in two different tasks. When the performance of the secondary task is reduced, it indicates an increase of the participant’s amount of mental effort, imposed by the primary task. Lower performances are typically reflected by a larger number of errors or a larger response time.

The values obtained by the dual-task paradigm are objectives, so analysis, and interpretation of data are measured independently of a self-report rating. The problem of the dual-task methods is a possible interference between the primary and secondary tasks, when one of the tasks somehow prevents the execution of the other. Our platform logs the number of errors and the response time.

3 Cognitive load stimulation

Our VR platform was developed using the ”Unity” game engine. Our platform is packaged as a single application composed of different scenes. The scenes are themselves composed of several phases (in our case two) each followed by a break where the NASA-TLX is displayed to the user. Each phase has its own set of tasks, all aimed at stimulating the user’s cognitive load.

The tasks require users to use logical reasoning, concentration and to perform some simple mathematical calculations. They will be detailed in the following subsections. During the tasks the user will hear audio beeps and is asked to press a button as soon as it is heard. This reaction time will be used as an objective measure of the cognitive load.

The game can be started by a unique executable file and the preferences of the game must be settled before launching the game, using a configuration file. The configuration file allow to set some parameters such as: the user information, the scene that will be played, the execution order of the phases, the duration time of each phase and the pause, the use (or not) of NASA-TLX scale, the use (or not) of a tutorial, the repetition rate of audio beeps, etc.

All the data emitted by the VR headset sensors are recorded in a comma-separated values (CSV) file. In the same file, events that occur during the execution of scenes (phase start and end, task start and end, task completion or failure) are also recorded. A text (TXT) file is also generated during the game. It will be filled with the user’s NASA-TLX answers, the average secondary task response time, and the average value of cognitive load. Both the CSV file and TXT file can be accessed at the end of the game. This information can then be used for future analysis.

We will now present the two main scenes available in the platform: the progressive task scene and the dual-task scene.

3.1 Progressive task

The aim of the progressive task scene is to stimulate the user’s cognitive load progressively through a sequence of increasing hard phases. We implemented a mathematical calculation task where the user needs to point at a numeric keyboard to enter the result of the displayed calculation. In the first phase, only arithmetic sums between numbers of maximum two digits will be displayed while during the second phase the sums can be between numbers of three digits.

3.2 Dual-Task

In this scene the user starts with a single task during the first phase, and then has to perform two different tasks at the same time in the second phase. During the first phase, the user needs to keep a line that is moving inside a defined interval by pressing a controller’s button. During the second phase, the user keeps doing the same task plus another one. In our case he will have to perform a simple calculation (as described in the previous subsection).

Figure 2: A visualization of the dual-task scenario. The user needs to do a mental calculation and to maintain a line in a specific interval at the same time. The application logs the user’ errors, physiological signals, NASA-TLX answers, and the response time to audio beeps.

4 Conclusion and Future Work

We presented a VR platform that can be used for stimulating and measuring the cognitive load in various ways.

For now, our platform is only compatible with the VR headset ”HP reverb G2 Omnicept”333https://www.hp.com/us-en/vr/reverb-g2-vr-headset-omnicept-edition.html, but further work will be done to integrate other headsets such as the ”HTC Vive Pro Eye”.

The HP reverb G2 Omnicept is build in with a Tobii eye tracker, a photoplethysmogram sensor (it detects the blood volume changes such as the heart beat), a face camera located on the bottom of the headset pointing at user’s mouth, and an inertial measurement unit including 3-axis accelerometers and 3-axis gyroscopes. The headset also has a machine learning model that can predict the user’s cognitive load as a continuous value ranging from 0.0 to 1.0.

One of the purposes of building this platform is to check if the cognitive load value given by the headset is usable and to which extent. In order to do this we will start an experiment and collect data of some users to see if the value is correlated to other cognitive load measurements available in our platform such as the reaction time, the NASA-TLX of the dual-task performances.

Acknowledgements.
This work has been done in the European Center of Virtual Reality (CERV)444cerv.enib.fr, a research platform of ENIB. We would like to thank the support of other CERV members and especially Nathalie Le Bigot for her expertise in psychology.

References

  • [1] R. Brünken, S. Steinbacher, J. L. Plass, and D. Leutner. Assessment of cognitive load in multimedia learning using dual-task methodology. Experimental psychology, 49(2):109, 2002.
  • [2] E. Haapalainen, S. Kim, J. F. Forlizzi, and A. K. Dey. Psycho-physiological measures for assessing cognitive load. In Proceedings of the 12th ACM international conference on Ubiquitous computing, pp. 301–310, 2010.
  • [3] S. G. Hart and L. E. Staveland. Development of nasa-tlx (task load index): Results of empirical and theoretical research. In Advances in psychology, vol. 52, pp. 139–183. Elsevier, 1988.
  • [4] C. S. Ikehara and M. E. Crosby. Assessing cognitive load with physiological sensors. In Proceedings of the 38th annual hawaii international conference on system sciences, pp. 295a–295a. IEEE, 2005.
  • [5] F. G. Paas. Training strategies for attaining transfer of problem-solving skill in statistics: a cognitive-load approach. Journal of educational psychology, 84(4):429, 1992.
  • [6] B. Park and R. Bruenken. The rhythm method: A new method for measuring cognitive load—an experimental dual-task study. Applied Cognitive Psychology, 29(2):232–243, 2015.