Play with One's Feelings: A Study on Emotion Awareness for Player Experience

06/16/2020 ∙ by Yoones A. Sekhavat, et al. ∙ 0

Affective interaction between players of video games can elicit rich and varying patterns of emotions. In multiplayer activities that take place in a common space (such as sports and board games), players are generally aware of the emotions of their teammates or opponents as they can directly observe their behavioral patterns, facial expressions, head pose, body stance and so on. Players of online video games, however, are not generally aware of the other players' emotions given the limited channels of direct interaction among them (e.g. via emojis or chat boxes). It also turns out that the impact of real-time emotionawareness on play is still unexplored in the space of online digital games. Motivated by this lack of empirical knowledge on the role of the affect of others to one's gameplay performance in this paper we investigate the degrees to which the expression of manifested emotions of an opponent can affect the emotions of the player and consequently his gameplay behavior. In this initial study, we test our hypothesis on a two-player adversarial car racing game. We perform a comprehensive user study to evaluate the emotions, behaviors, and attitudes of players in emotion aware versus emotion agnostic game versions. Our findings suggest that expressing the emotional state of the opponent through an emoji in real-time affects the emotional state and behavior of players that can consequently affect their playing experience.



There are no comments yet.


page 1

page 3

page 4

page 5

page 6

page 7

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Digital games are designed to offer rich affective experiences for players through their interactions with the various forms of game content and types of non-player or player agents. Multi-player digital games extend the affective capacities of a game as they make it possible to play against a number of different opponents (adversarial play) or teammates (collaborative play) including friends, strangers or artificial intelligence (AI)-controlled agents 

[yannakakis2018artificial, gaina20172016, schiller2018inside]. Such social interactions between players in multi-player games may result in augmented involvement and immersion during gameplay [ravaja2006spatial, yannakakis2011experience, sekhavat2017comparison, sekhavat2018sense]. One can only assume that the understanding of such experiences and the study of player emotion is both crucial and beneficial to modern game design [yannakakis2014guest]. The relationship between emotion awareness, however, and in-game experience and behavior is currently understudied.

Humans are social beings with motivations for social interaction. Given that social relationships are affective by nature, we expect that playing against opponents that manifest varying emotions can elicit different emotional responses to the player. As emotions in social settings can affect human decision making, we also expect that the awareness of the opponent’s emotion may have an influence on the behaviors of a player. Motivated by the lack of a comprehensive empirical study examining the impact of emotion awareness in multi-player settings, this paper investigates the influence of an opponent’s emotions on the player’s state. As a first step, this initial study examines the degrees to which a player’s emotional and behavioral state is affected by manifested emotional expressions of an opponent—via displayed emojis—in a two-player car racing game. Although earlier research has studied the emotional impact of playing against different player types [cagiltay2015effect, ravaja2006spatial], to the best of our knowledge, no other study has examined the effect of emotional awareness of opponents to the player’s behavior and experience. In particular, we test the following hypotheses within the genre of two-player adversarial car racing games:


The emotional state of the opponent (as expressed through an emoji) attracts the attention of the player.


The behavior of a player is affected by the emotional state of the opponent.


The emotional state of a player is affected by the emotional state of the opponent.


The awareness of the opponent’s emotional state enhances the perceived competitiveness.


The awareness of the opponent’s emotional state increases the sense of social presence.


The awareness of the opponent’s emotional state increases the player’s reported enjoyment.

To test the aforementioned hypotheses, we developed a special-purpose two-player car racing game equipped with a eye-tracking device which captures gaze data of players, and a facial expression analysis component that detects the emotional state of the player during play. We designed two game modes and conducted a game user study with 20 participants. Players were asked to play the game in the emotion aware mode—in which the emotional state of the opponent is presented to the player during play—and in the emotion agnostic mode—in which the player is not aware of the opponent’s emotions. Experiments in this paper test collectively how the awareness of the emotional state of an opponent affects the emotions and consequently the gameplay behaviors of players. It is important to note that these are facial expressions that are naturally manifested as a result of playing a game and not deliberately self-reported. While the self-report of emotions via emojis is a popular game design practice—as e.g. in Clash Royale (Supercell, 2016)—the real-time detection of natural expressions proposed here offers more flexibility and granularity for the design of games and their experience. Self-reported emojis in games could form a valid experimental control against manifested emojis during gameplay; such an experiment, however, is outside the scope of this paper.

Our key findings suggest that all but one (H5) hypotheses are validated indicating that the expression of the opponent’s emotional state through an emoji affects the emotional and behavioral state of players. Finally, it seems that being aware of an opponent’s affective state yields higher perceived competition and enjoyment.

I-a Why Facial Expression?

A number of methods can be used to detect and classify the emotional state of a player from signal streams that are manifested during the game. Among the various modalities of player input that are available, we focus on the analysis of a player’s facial expression as it offers a number of key advantages. First, player experience can be inferred automatically based on facial expression detection and affect modeling analysis. Second, various degrees of in-game challenges can also be determined from facial expressions 

[blom2014towards]. Finally, emotion detection via facial cues is a non-intrusive task that can be performed using simple off-the-shelf webcameras [burns2017detecting]. In this paper we use facial expression analysis techniques to detect and analyse the emotional states of racing game players and in particular, we study the happy, sad, angry, or neutral player states as manifested through facial expression.

I-B Why Car Racing Games?

All of our hypotheses regarding the expression of manifested emotions of opponents are tested in a two-player adversarial car racing game. There are three reasons for selecting this game genre. First, driving in common racing games is not conducive to rich social interactions between players because players can only see the cars; normally players are not able to see the drivers and their emotional expressions. This common design feature of the racing game genre suppresses the social presence of another human player behind the steering wheel; hence, racing games can directly benefit from the expression of manifested emotion as the approach followed in this study can enhance the emotional relations between car racing players. Second, the task of playing car racing games well is challenging to study as it requires substantial spatial coordination and kinaesthetic skills even though the possible actions of a player are somehow simple and limited (i.e. accelerating, braking and steering). Third, unlike planning-heavy games such as strategy games that require the player to wait for a long period before seeing the consequences of their actions, fast-paced racing games offer instant feedback of a player’s actions that, in turn, affect the player’s performance.

Ii Background: Emotion Expression in Games

The study of emotions in games is an area with an increasingly central role within game research [yannakakis2018artificial]. The understanding of players’ emotions and their linking to playing experience is one of the main targets for game design [yannakakis2014emotion]. In addition, affect-based game interaction can result in emotional patterns which may enhance the playing experience. Emotions in games are realized within an affective loop, in which the game elicits, detects and responds to the emotions of players. Various artificial intelligence algorithms and human-computer interaction methods can be used for such purpose [yannakakis2018artificial, yannakakis2014emotion]. When it comes to racing games and the investigation of player experience in that genre Tognetti et al. [tognetti2010enjoyment] have developed a framework that detects the levels of player enjoyment directly from physiological signals via the use of preference learning. Georgiou and Demiris. [georgiou2017adaptive] took a further step forward for the detection of a car racing player state by integrating additional modalities of user input to the player model.

Earlier studies have shown that the core characteristics of an opponent may have an impact on the affective or cognitive state of a player of a competitive computer game [weibel2008playing, mandryk2006using]. For example, different physiological responses can be observed in players when playing against a computer or a friend. In particular, playing against another person elicits greater arousal compared to playing against a computer opponent [weibel2008playing]. It has also been shown that playing with friends can result in higher degrees of spatial presence, engagement, and physiological arousal than playing with strangers [cagiltay2015effect]. Mandryk et al. has also reported different physiological responses of players when playing against a computer or a friend [mandryk2006using]. As discussed in [jervcic2018effect], human collaborators are perceived as more credible and socially present than non-human collaborators in the context of serious games. On the other hand, rich social interactions can be established within temporary teams in games, as players are interested in collaborating with strangers [kou2014playing].

Another important element that can affect the behavior of players in the social context of games is competition [dondlinger2007educational]. In competition-based (adversarial) games, one player advances toward achieving a goal, while the other moves further away from it [hong2009playfulness]. Prior research has shown the potential of competition to draw the attention and excitement of players in games [cheng2009equal] that can, in turn, motivate players to put more effort into the in-game tasks. Playing a competitive video game activates the cognition associated with motivational pathways [katsyri2013just]. Research has shown that competitive video gaming can lead to enjoyment and positive affect [vorderer2003explaining]. Competition is also an important element of a computer game that can affect the motivation of players [dondlinger2007educational, melhart2019your]. Research has shown that although competition is not an essential game design element, it is nevertheless a critical motivator for gameplay experience and immersion [shaffer2006computer]. In this study, we argue that the awareness of the emojis of game opponents can affect the competition levels among players that can consequently lead to more enjoyment and positive affect in the game. In particular, as examined through hypothesis H4, we argue that the awareness of the opponent’s emotional state enhances the perceived competitiveness.

Research has shown that emotional expressions can convey a person’s cooperative tendencies that can, in turn, affect the decision-making of others [hoegen2017incorporating]. For example, smiles can indicate cooperative tendencies [stratou2015emotional] and temporal dynamics of smiles can shape trust among players of games [krumhuber2007facial]. Inexpressive opponents, instead, are generally viewed as untrustworthy [schug2010emotional]. Following this line of research Hoegen et al. [hoegen2017incorporating] have built a model that predict a player’s behavioral patterns based on the opponent’s emotional expressions and actions; such a model can be used for planning social interactions within the game. Decoding emotional signals of opponents has also been used in negotiation. According to [de2014reading], people can make inferences about others’ mental states from emotional expressions for decision making during a negotiation phase; it appears that facial expression contains critical cues for making such inferences. Further research in behavioral science has studied the impact of emotions on one’s own behavior and on another’s behavior in negotiations [van2010interpersonal]. For example, it is shown that manifesting anger during a negotiation can yield more concessive behaviors on one’s opponent [pietroni2008emotions], whereas expressing happiness may lead to fewer concessions. Emotions can further be used for the purpose of resolving conflicts in agent-agent or human-agent interactions [yannakakis2010siren]. Van Kleef et al. contend that “each discrete emotion has its own antecedents, appraisal components, relational themes, and action tendencies” [van2010interpersonal]. They argue that observing a particular emotion in a person provides “relatively differentiated information about how that person regards the situation” [van2010interpersonal]. Considering video gaming as a social interaction, our findings in this paper highlight that expressing emotion plays a core role in controlling the behaviors of opponents in this game-based form of social interaction.

Iii Experiment Design

To test our core hypothesis that opponent emotion awareness has an impact on the player experience we conducted a user study in a controlled environment (computer lab) and invited 20 participants to play variants of a two-player car racing game. The details of the experimental protocol we used and the game user study are detailed in the remainder of this section.

Iii-a Car Racing Game

Fig. 1: A screenshot of car racing game developed with the presentation of an emoji at the bottom-center of the player’s car.

A prototype of a car racing game was developed and used in all experiments reported in this paper; see Fig. 1. In this game, a player can control the car by accelerating, braking, and steering to right and left. This game provides adversarial gameplay through which two or more players can compete against each other to gain the first place in a racing route. After the initialization phase, the main loop of the game is started, where user inputs are gathered from the input devices (gaze and facial expression tracker) that are described below.

Iii-B Emotional Tracking via Facial Expression

This paper studies the relationship between a player’s and an opponent’s manifested emotions as displayed during the game. The particular set of emotional states chosen for this study is based on Ekman’s taxonomy of basic emotions as expressed through faces [ekman1999basic]. In this study, we focus on happiness, sadness, and anger as disgust and fear are not relevant emotions for this game genre and surprise (even though highly relevant for the racing genre) was not detected as frequently by our facial expression detection software and thus it was not included in our analysis.

We captured a player’s emotions through a facial expression recognition system consisting of a face detection module, a preprocessing module, a deep neural network module, and a mapping module. In particular, we use a supervised transform network 

[chen2016supervised] as a face detector. The captured face image is preprocessed to meet the constraints of a fine-tuned version of VGG-16 [simonyan2014very] which is used, in turn, to compute a player’s emotion with 13 convolutional layers, and 3 fully connected layers. To retrain the network for the purpose of facial expression recognition, we used the Extended Cohn-Kande dataset (CK+) [lucey2010extended]

, which provides a probability distribution over seven distinct emotional states, including neutral, anger, happiness, fear, sadness, disgust and surprise. The results of our training experiments yield a test accuracy of

on the CK+ dataset [yang2017facial]111The source code of our face recognition system is freely available at

Iii-C Participants

For this study we recruited 20 participants (16 males and 4 females) from undergraduate and graduate students of the faculty of Multimedia at Tabriz Art University. Participants’ ages range between 21 and 27 years old, with a mean of years. Upon recruitment, participants completed a brief questionnaire regarding their age, gender, video game skill level and time spent regularly playing. Those who reported no experience with video games were excluded. According to the background information captured through the pre-study questionnaire, all of the participants were frequent gamers. Based on the classification provided in [green2003action], frequent gamers are those who play video games at least 5 hours a week for a period of 6 months or more continuously. Subjects participated in the experiment in groups of two persons of the same sex.

Iii-D Experimental Protocol

To better distinguish between the settings with and without emotional awareness in a car racing game, we designed a within-subjects user study, where each participant played the game in both settings. In the emotion aware setting (EAW), the facial expression of each player is identified based on the image captured from the webcam installed above the monitor. Then, the emotional state based on this facial expression is presented to the opponent as expressed through the facial expression of an emoji. In our attempt to find the best possible position to place the emojis on the screen, we decided to display them just below the player’s car—which is always at the bottom-center of the screen—for two reasons. First, as the player always looks at her car while playing, the emojis are always visible in the area close to the player’s gaze. Second, there is a good contrast between the emoji and the solid color of the background (road) that makes it easy to recognize the emoji without additional cognitive burden (see Fig. 1). In the emotion agnostic (EAG) setting while the facial expression of opponents is extracted and logged, this information is not displayed to the players.

Fig. 2: The emojis used to express different emotions of the opponent in the game.

The emojis used in the prototypes were downloaded from vexels222Available at Based on these images, we created 2-second animations of facial expressions representing the transitions from one emotional state to another. The final frames of different emotional states are shown in Fig. 2. As illustrated in that figure we represent the happiness state by using a grinning face with smiling eyes; we used a slightly smiling face with neutral eyes to represent the neutral state.

Each emotional state was accompanied by a corresponding short sound effect. By using auditory effects in conjunction to the emojis we wish to ensure that the player is aware of the current emotional states of her opponent. The ambient background sound of the car racing game is a typical sound of a car engine. A change in the emotional state of the opponent was a trigger for playing the corresponding sound as well as for playing the transition animation (a change from one emoji to another). We selected short sounds (around 2 seconds) for each emotion among the tagged sounds available in zapsplat333 To avoid frequent and abrupt changes in the presentation of emotional states, we skipped the changes that were happening during the transition from one state to another.

In order to address potential order and learning effects, the order of exposure to the different settings was varied, resulting in the assignment of participants to two groups: the first group started the experiment with the EAW setting whereas the second group started playing on the EAG setting. This way, the order in which the participants play in different settings was eliminated as an independent variable.

Players answered a set of questions before and after the experiment. Before playing the game, each player was informed about the controls and the user interface of the game including the facial expression of the opponent (in the EAW mode). As discussed in [backlund2006computer], players who are not assigned any specific task during a driving game enjoy the game more than the players who were given a task. Consequently, we let players to freely drive their car, and play and enjoy the game in their own way. In groups of two players, each player is asked to drive a customized track for three laps. The first lap is a training lap, in which we did not measure the performance data and players had the opportunity to become familiar with the controls of the game. The driving path was specifically designed for this study and players did not have an experience of driving on this route before the study. After playing the game, participants were asked to answer some informal questions regarding the game.

Fig. 3: The overall setting of the user study. The facial expression module as well as the eye-tracking device is active in EAW and EAG modes. The emotional state of the opponent is shown to the player only in the EAW mode.
Fig. 4: Two players in separate rooms who are competing against each other in the car racing game. Webcams and eye trackers are placed, respectively, above and below the monitor.

As shown in Fig. 3 and Fig. 4, participants were asked to sit in front of a 24-inch monitor and play the car racing game with a standard keyboard. As the attachment of sensors to body parts is an intrusive process that may affect the experience of players [yannakakis2018artificial, yannakakis2014emotion] we used an unobtrusive eye tracker to capture gaze data. In particular, we installed a Tobii EyeX eye tracker to capture the eye movements of players and their focus on the screen during driving. Data from the game including speed, position, number of collisions, players’ actions (braking, speeding or steering), facial expression as well as eye gaze data was all collected during the game and time-stamped. Real-time performance metrics including speed and the position of the car in the racing track, as well as the facial expression of the opponent, were displayed on the monitor.

Before starting the experiments—when participants had an opportunity to freely play the game and become familiar with its controls—we explicitly informed participants that the emojis on the screen represent the real-time facial expression of their opponent while playing. This way, we made sure that players attribute the emoji to the opponent’s emotion and not to their own.

Iii-E Measures

To test each one of our five hypotheses we relied on a number of measurable parameters that are detailed in this section.

Iii-E1 Attention to Emotional Expression

In addition to using gaze as a sole modality to control the game [munoz2011towards], eye-tracking technology can offer information on where and what players are looking at while playing. Since in the EAW mode of the game the emotional state of the opponent is shown as a face icon on the screen, one would expect that players would pay attention to the icon during the game if it is relevant for their gameplay experience. As mentioned earlier we used two eye-tracker devices (one for each player) to obtain gaze data on the screen. The analysis of this data could reveal if the emotional states expressed via emojis are under the player’s attention.

Iii-E2 Player Behavioral and Performance Metrics

Since the aim of this paper is to study the degree to which the emotional state of one player may affect the behavior and the emotion of the other player, we capture a number of metrics regarding the behavior of players in the game as well as measures that indicate in-game performance. In particular, we log the position of a player, the speed of the car, the ranking of a player during the race, the state of focusing on the emoji (a binary value indicating if a user looks on the emoji or not in ea), the number of collisions with other cars on the road or some objects on the roadside (as an indicator of driving mistakes) and the emotional state of the player.

Iii-E3 Self-Reports

We measured two subjective factors using self-report questionnaires. First, participants were asked to answer some questions regarding Perceived Competitiveness. This parameter was measured by six statements adapted from [song2013effects] as follows: 1) I felt that this game was competitive; 2) I think other people would feel that this game is competitive; 3) I paid attention to my position in the route in relation to the position of my opponent; 4) I tried to be ranked first; 5) Ranking on the screen motivated me to drive better; 6) Other people’s performances motivated me to drive better and try harder. Participants were asked to represent their levels of agreement on a 5-point Likert-type scale. Participants also reported their social presence in the game. To this end, we measured the social presence using 17 questions [ijsselsteijn2013game] that assess the psychological and behavioral involvement of the player with other social entities. Finally, participants were asked to answer whether presenting the emotions of the opponent through the emojis had a positive effect on their enjoyment while playing.

A summary of the variables measured to support or reject each hypothesis and the corresponding method used to measure the variables are shown in Table I.

Hypothesis Variables Method
H1: Attracting the attention of players. Eye gaze data, fixation count, dwell time. Data collected by Tobii EyeX eye-trackers.
H2: Effect on the behavior of players. The number of braking, steering and throttling, the number of collisions, gameplay duration. Logged keyboard inputs. Captured in-game events and properties such as collisions with other cars and objects on the road, and racing times.
H3: Effect on the emotional states of players. The number of happy frames and angry frames, the number of emotionally active states, the rate of changes in the emotional states of players. Using a facial expression detection algorithm to detect emotions, finding relations between the emotional states of players and opponents, and comparing emotionally active states across different modes.
H4: Enhancing the perceived competitiveness. Perceived competitiveness. Six questions adapted from the perceived competitiveness questionnaire [song2013effects].
H5: Increasing the sense of social presence. Social presence. 17 questions based on social presence questionnaire  [ijsselsteijn2013game].
H6: Increasing reported enjoyment. Reported enjoyment. A single question about the effect of presenting the emotions of the opponent on game experience.
TABLE I: The variables used to test each of the six hypotheses in this paper and the corresponding method used to measure each variable

Iv Results

The collected quantitative and qualitative data was analyzed and the results of this analysis are presented in the following sections; each section is dedicated to each of the five hypotheses of this paper.

Iv-a Attention to Emotional Expression: H1

Before studying the effect of opponent emotional awareness on the behavioral and emotional states of players, we test the degree to which players pay attention to the emotional state of opponents displayed for them as an emoji at the bottom-center of their game screen. To this end, we used the data captured from the two Tobii EyeX eye-trackers.

A point captured by an eye-tracker is a single gaze that has a specific position. Gaze points close together can be grouped to form a fixation. Research has shown that fixations (i.e., moments when the eyes are relatively stationary), can reveal which parts of the displayed information are most salient, which is linked to attention 

[eckstein2017beyond]. Although a fixation does not necessarily mean that the participant perceived an element, generally speaking, elements that draw visual attention have a higher chance of being perceived [poole2006eye]. In our experiments, we computed fixations as areas on the screen where participants look for at least 200 ms following the guidelines proposed in [salvucci2000identifying]. Based on [hornof2002cleaning] we set the maximum distance that a point may vary from the average fixation point and still be considered part of the fixation at 50 pixels.

Heatmaps are time-aggregated density-based representations that are often used to visualize gameplay data [yannakakis2018artificial]. When applied to gaze data of players, heatmaps can reveal the areas where the players look at more frequently [raschke2014visual]. In our case study a heatmap of player fixations may offer the first insights on whether the player pays attention to the emotional awareness of his opponent while playing the game or not. For that purpose the heatmaps or eye fixation data for all players in both EAW and EAG states are visualized and are shown in Fig. 5 and Fig. 5, respectively. As it can be observed from these figures the emotional states of the opponent attract the attention of the player only in the EAW mode. Further, the fine-grained analysis of gaze data based on the three dissimilar emotional states of the opponent (see Fig. 6) shows that the emoji is always under attention by players regardless of the emotional state of the opponent. The results validate hypothesis H1 that the emotional state of the opponent as expressed by an emoji attracts the attention of the player in two-player car racing games.

Fig. 5: Heatmaps of eye fixations aggregated across all participants in the EAW mode (left) versus the EAG mode (right). The hotter (red) the area of the map the more gaze fixations are available in that area.
Fig. 6: Heatmap of accumulative gaze data across the three different emotional states examined in this work: happy (left), neutral (center), and angry (right).

Iv-B Behavior Analysis: H2

Fig. 7:

The average number of braking, steering and throttling across all players in the EAW and EAG game modes. Error bars display the 95% confidence interval of the average shown.

Through H2 we hypothesize that the emotional state of the opponent—as expressed via emojis—can affect the behavior of the player. Thus, we expect to observe differences in the way the player controls the car in the game in the EAW mode. In the examined game the possible actions a player can take during play include steering (right and left buttons), throttling (up button to speed up) and braking (down button to reduce the speed). Based on these actions we extract a number of core features that characterize the gameplay behavior in a car racing game and examine the degree to which they are affected by the presence of the opponent’s emotional manifestations. These include the total amount of steering (), speeding up () and reducing speed () but also the total number of collisions with other cars on the road or some objects on the roadside () and the gameplay duration ().

As shown in Fig. 7, participants were on average more active in terms of steering, braking and throttling when they played the EAW version of the game. To test H2 regarding the effect of one independent variable (EAW vs. EAG) on three dependent variables (, ,

), we used the repeated measures MANOVA test to cater to and assess multiple response variables simultaneously. The results showed that there was a statistically significant difference in players’ behavior based on emotional awareness,

, ; Wilk’s lambda is ( according to the post hoc power analysis). Such a finding validates H2 and suggests that the behavior of a player appears to be affected by the emotional state of the opponent in two-player car racing games.

In addition to car control extracted features, we also compared the behavior of players in terms of the number of collisions with other cars on the road or objects on the roadside, which is an indicator of driving mistakes in the car racing game. We argue that the stress of playing against an emotionally active opponent as well as the distractions coming from the changes in the emoji expressions on the screen can increase the number of these errors. A paired samples t-test was employed to test for any significant difference between the number of collisions in the EAW versus the EAG mode. The result reveals a significant difference between the EAW (

) and the EAG () modes (, ) which further indicates that the performance of a player in terms of the number of collisions in the car racing game can be affected by the emotional state of the opponent.

Finally, we compared the EAW and EAG modes in terms of gameplay duration. The gameplay duration is calculated as the total time needed to finish three laps of the path. The results show that participants spend more time in the EAW mode than in the EAG mode. According to the paired samples t-test conducted on the samples, there was a significant difference between EAW () and EAG () modes () indicating that the emotional awareness of the opponent seems to increase the level of challenge in the game, that can consequently affect the behavior of players.