It's Food Fight! Introducing the Chef's Hat Card Game for Affective-Aware HRI

by   Pablo Barros, et al.

Emotional expressions and their changes during an interaction affect heavily how we perceive and behave towards other persons. To design an HRI scenario that makes possible to observe, understand, and model affective interactions and generate the appropriate responses or initiations of a robot is a very challenging task. In this paper, we report our efforts in designing such a scenario, and to propose a modeling strategy of affective interaction by artificial intelligence deployed in autonomous robots. Overall, we present a novel HRI game scenario that was designed to comply with the specific requirements that will allow us to develop the next wave of affective-aware social robots that provide adequate emotional responses.


page 1

page 3

page 4


Affect-Driven Modelling of Robot Personality for Collaborative Human-Robot Interactions

Collaborative interactions require social robots to adapt to the dynamic...

Towards Learning How to Properly Play UNO with the iCub Robot

While interacting with another person, our reactions and behavior are mu...

Sensitive Pictures: Emotional Interpretation in the Museum

Museums are interested in designing emotional visitor experiences to com...

Generating Emotionally Aligned Responses in Dialogues using Affect Control Theory

State-of-the-art neural dialogue systems excel at syntactic and semantic...

Emotional Metaheuristics For in-situ Foraging Using Sensor Constrained Robot Swarms

We present a new social animal inspired emotional swarm intelligence tec...

Symbiotic Child Emotional Support with Social Robots and Temporal Knowledge Graphs

In current youth-care programs, children with needs (mental health, fami...

Data-driven emotional body language generation for social robotics

In social robotics, endowing humanoid robots with the ability to generat...

1. Introduction

Associating perception and action within a continuous adaptation mechanism could improve the way social robots interact and are accepted in real-world social scenarios (Crossman et al., 2018). An affective architecture, developed in the framework of cognitive computation, is the focus of several researchers over the past decade (Franklin et al., 2013; Tanevska et al., 2018). In particular, modulating behavior based on affect perception has been explored by the affective computing community (Hirokawa et al., 2018). Most of the presented solutions, however, developed social interaction strategies based on emotion contagion (Van de Perre et al., 2018)

by repeating what was perceived, or relied upon simple decision trees for generating behavior

(Tuyen et al., 2018).

Such solutions are suited for the simple interaction scenarios in which they were deployed. The limitation on designing, implementing and deploying realistic interaction scenarios, where affective information is more complex to capture and respond to is one of the problems that must be solved before the deployment of robots in real life. In a previous attempt without using a robot, a game especially designed to provoke to favor one and disfavor another player in a multiplayer interaction was shown to provoke emotions (Barakova et al., 2015), which lasted during the repetitive games between the same players (Gorbunov et al., 2017).

In this paper, we continue our efforts (refer to our previous work (Barros et al., 2019)) on designing a social interaction scenario, based on a multiple-players games, to be used in affective-aware Human- Robot Interaction (HRI), also illustrated in Figure 1

. The new game, entitled The Chef’s hat, was designed in an iterative process to guarantee that the display of affective behavior was natural and an integral part of the game dynamics. We first explored the social dynamics between human players as part of our game design. The integration of robots into the game, as active players, is of fundamental importance. To facilitate that, we included elements in the games requirements and final mechanics that allow robots to have an estimation of the game status and the person’s social behavior, and play without disturbing the game flow. As the game is intended to enforce teaming between the players, the occurrence of human- robot teaming behavior is an important part of our game design decisions. The proposed game scenario provides a platform where different learning systems for affective perception, decision making, and behavior generation can be integrated, deployed and evaluated properly and in a close-to-real-world scenario.

2. The Chef’s Hat Interaction Game

The design of the card-game was based on two main principles: 1) To provide restricted, but natural interaction, where affective behavior naturally arises; 2) to make possible that the game is played between people and a robot and provide a turn-taking opportunities, where a robot, such as (Metta et al., 2008), has a supportive infrastructure and capacity to process incoming information and generate behavior without breaking the fluidity of the interaction.

2.1. Interaction Requirements and Initial Assessments

To maintain the two main principles, as listed in the previous subsection, the card game was design to satisfy the following requirements:

  • R1 - The game should elicit a multitude of affective behavior that the robot can properly understand and model.

  • R2 - The game should be playable without the need of verbal expressions as part of the game mechanics to facilitate the affective understanding and processing.

  • R3 - The game should provide the possibility of creating strategies based on affective bonding between the players. This way, interacting with other players through the game should be part of the game’s natural flow.

  • R4 The game should have specific mechanics that, when used, cause affective reactions in the players.

  • R5 - The game should be easy to follow and to understand, having clear turns between the players. The number of actions to be made should be small and easy to process. This way, the limitations of the robot regarding decision-making, behavior display and processing time will be reduced.

  • R6 - The game status and players’ behavior should be easy to track and monitor using automatic mechanisms (cameras and microphones). This way we can create a knowledge repository to leverage the learning of gaming strategies and behavior understanding by the robot.

  • R7 - The game should give players enough opportunity to interact with all other players through game-mechanics. Actions taken should not only affect the next player in the game, as this could limit both the social and the competitive strategies.

Taking these requirements into consideration, we started our design process by evaluating common card games that are well-known to persons from different cultural backgrounds: e.g. UNO, Great Dalmuti and Quartet. Each of these games addresses some of the requirements we listed above, see Table 1. Once these games were selected, we performed several rounds of experiments with four participants playing each of these games.

Taking these requirements into consideration, we started our design process by evaluating common card games that somehow evoke the requirements that are well-known to persons from different cultural backgrounds: e.g. UNO, Great Dalmuti and Quartet. Each of these games addresses some of the requirements we listed above. Once these games were selected, we performed several rounds of empirical experiments with four players playing each of these games.

We performed a behavioral observation to evaluate which requirements each of these games addressed. We video-recorded and analyzed the behavior of all the players during the game and the behavior observation was carried out by four industrial design students , which were trained on evaluating user interactions.

Game R1 R2 R3 R4 R5 R6 R7
Great Dalmutti
Table 1. Summary of which requirements are addressed by each of the tested games.

The analysis showed that popular games, such as UNO and Great Dalmutti, fit most of the defined requirements, but not all. In particular, as discussed in our previous work (Barros et al., 2019), the possibility of causing adverse affective reactions throughout the gameplay was very present in UNO, but not in the other games. The disadvantage of UNO game was, the use of many words, and the lack of play elements that encourage bonding throughout the entire gameplay. Therefore, UNO was also disregarded as an appropriate game for the defined purposes.

Great Dalmuti, which is by itself based on a public domain game named Presidenten, or Daifugo in Japan, and UNO were taken as inspiration for creating the new card game that will fit the experimental purpose. The new game presents very simple game mechanics which is fairly easy to be understood and followed, similar to UNO, but without limiting strategic options as it is composed of different rounds with a continuing score. Furthermore, the hierarchy in the gameplay creates interesting group dynamics as well as opportunities for both competitive and cooperative play.

2.2. Game Mechanics

Chef‘s hat is a card game that Our game simulates a kitchen, and it has a role-based hierarchy: each player can either be a Chef, a Sous-Chef, a Waiter, or a Dishwasher. The players try to be the first to get rid of their ingredient cards and become the Chef. This happens for multiple rounds (or Shifts) until the first player to reach 15 Chef . During every Shift there are three phases:

  • Start of the Shift

  • Making Pizzas

  • End of the Shift

At the start of the Shift, the cards are divided. Then, the Dishwasher has to give the two cards with highest values to the Chef, who in return gives back two cards of their liking. The Waiter has to give their lowest- valued card to the Sous-Chef, who in return gives one card of their liking. If a player has two Jokers at the start of the Shift, they can choose to play their special action: in case of the Dishwasher this is ”Food Fight” (the hierarchy is inverted), in case of the other roles it is ”Dinner is served” (there will be no card exchange during that the Shift).

Then, the making of the pizzas starts. The person who possesses a Golden 11 card may start making the first pizza of the Shift. A pizza is prepared when ingredient cards are played on the pizza base on the playing field. A pizza is done when no one can (or wants to) put on any ingredients anymore. The rarest cards have the lowest numbers. A player can play cards by laying down their ingredient cards on the pizza base. To play cards, they need to be rarer (i.e. lowest face values) than the previously played cards. The ingredients are played from highest to the lowest number, so from 11 to 1. Players can play multiple copies of an ingredient at once, but always have to play an equal or greater amount of copies than the previous player did. If a player cannot (or does not want) to play, they pass until the next pizza starts.

At the end of the Shift, the new roles are distributed among the players according to the order of finishing, and every player gets the number of points related to their role.

Several pilot data collections have been conducted and videos recorded and the qualitative observation of these data showed that the requirements indicated before have been met. More specifically, we observed that the players exhibited clear emotional disruptions when the special actions were invoked. Also, we observed that some players developed group-based strategies against a player which had a higher score. Finally, the addition of the physical attributes was noted to give the players more motivations towards winning the game, which improved greatly their social engagement.

2.3. Game Elements

We started the design process of the “Chef’s Hat” card game taking Great Dalmuti as inspiration. In order to reduce the feeling of a controlled experiment and make the game feel more like a commercially available card game, a theme was chosen and incorporated throughout all game elements. We opted for a Kitchen theme, where players compete to become the chef of a pizza restaurant. This theme has also been used to clarify and give reasoning to certain rules, as a means to make the game even easier to follow.

Next to this, an important part of the design of the game elements was dictated by the game-state monitoring problem. The goal was to only have the need for one top-view camera to monitor the entire game-state.

The ingredient cards, illustrated in Figure 2, needed to be easily recognizable both when played on the playing field and when exchanged among players at the start of the Shift. For this, QR- codes were chosen, as they are hard for humans to recognize or memorize, but easy for the robot or intelligent camera to read.

Figure 2. Ingredient cards, and the Joker, with their corresponding face number. The lower the number, the rarer the card.

The cards are to be placed on the playing field, illustrated in Figure 3. While testing the game, a recurrent problem was that players did not play their cards one by one (which made it difficult for the camera to recognize what was played). To counter this, we redesigned the playing field to have 11 different marked places in which players could place their cards on the pizza.

Figure 3. Playing field where the cards are placed, representing a pizza board.

Role cards were designed to make it easy to see which player has a certain role. The role cards have an additional functionality to work as a special action card when turned over. To increase the engagement (and possibly competitiveness), it was opted to also have physical role-related attributes: a Chef’s hat, a Sous-Chef’s hat, a Waiter’s bow-tie, and a Dishwasher’s cloth. The attributes were color-coded per role (see figure 4).

Figure 4. Physical attributes used for each role: A - Chef, B - Sous-Chef, C - Waiter, D - Dish washer.

3. Next Steps: Robot Learning Workflow

The proposed game was designed to be used in an HRI scenario, in particular to 1) evoke natural and clear emotional reactions and social dynamics in the participants; and 2) to enable a robot participating in the game to sense and monitor both the game status and the affective responses of each participant and of the group as a whole.

The goal for the robot involved in the game will be to understand the affective behavior of the human participants and use this information, together with that of the game status, to select the best strategy to play the game. ”Best strategy” in this sense will acquire different meanings. The robot might either aim at winning the game or at selecting the behavior that maximizes a positive affective reaction in the other players.

One of the most challenging problems we expect to face is the mechanical interaction of the robot with the card game. We do not want to reduce the fluidity, or naturalness of the game from the player’s perspective because this would destroy our general ideal of having a very natural scenario. The iCub is a very advanced social robot, but still have limitations when interacting with real objects, in particular, with very thin objects such as cards. In this regard, we plan to adapt the way that the iCub interacts with the game. Our general idea is to have the iCub asking for help any time it needs to receive or withdraw a card, or put on and off a physical attribute at the beginning of the Shift and use the QR codes present in the card to generate an internal representation of which cards the robot has at hand. However, we are still evaluating the best way to make the robot place a card on the board during the game. Our current investigations involve weather by using a projector that will project the card of the robot on the board or using a specific cardholder that the robot can point at when it wants to place a card.

To achieve the intelligent and adaptive behavior that connects the learning of affective perception/display and game strategy, we envision a cognitive framework which will address the problem based on two perspectives: an adaptive affective perception, based on our previous work on artificial neural networks

(Barros and Wermter, 2016, 2017)

, which will use an unsupervised affective memory mechanism to overcome the problems of online learning; And an interactive reinforcement learning strategy based on learning the social and contextual impact of a certain action by measuring the affective responses of the players, avoiding the necessity of active human interaction in the learning loop.

4. Conclusion

In this paper, we described our vision on how to achieve an affective awareness in Human- robot interaction. In particular, we propose a new card game that encapsulates specific requirements which makes it suit- able for playing with robots. Moreover, the developed game enables research through design approach, since the game is specially designed to be used as a tool for gathering data and improving the emotional interaction between humans and robots.

Our Chef´s Hat card game has specific elements that integrate continual affective understanding into the game mechanics.Also, it was designed in a way to facilitate learning algorithms that can be used to understand and produce affective behavior, and learn game mechanics .

In the future we will focus on the development of the learning mechanisms to enable the robot to autonomously play the game and we will exploration how such a robot impacts the interaction between the players during the game.


  • E. I. Barakova, R. Gorbunov, and M. Rauterberg (2015) Automatic interpretation of affective facial expressions in the context of interpersonal interaction. IEEE transactions on human-machine systems 45 (4), pp. 409–418. Cited by: §1.
  • P. Barros, S. Wermter, and A. Sciutti (2019) Towards learning how to properly play uno with the icub robot. arXiv preprint arXiv:1908.00744. Cited by: Figure 1, §1, §2.1.
  • P. Barros and S. Wermter (2016) Developing crossmodal expression recognition based on a deep neural model. Adaptive behavior 24 (5), pp. 373–396. Cited by: §3.
  • P. Barros and S. Wermter (2017) A self-organizing model for affective memory. In Neural Networks (IJCNN), 2017 International Joint Conference on, pp. 31–38. Cited by: §3.
  • M. K. Crossman, A. E. Kazdin, and E. R. Kitt (2018) The influence of a socially assistive robot on mood, anxiety, and arousal in children.. Professional Psychology: Research and Practice 49 (1), pp. 48. Cited by: §1.
  • S. Franklin, T. Madl, S. D’mello, and J. Snaider (2013) LIDA: a systems-level architecture for cognition, emotion, and learning. IEEE Transactions on Autonomous Mental Development 6 (1), pp. 19–41. Cited by: §1.
  • R. Gorbunov, E. I. Barakova, and M. Rauterberg (2017) Memory effect in expressed emotions during long term group interactions. In International Work-Conference on the Interplay Between Natural and Artificial Computation, pp. 254–264. Cited by: §1.
  • M. Hirokawa, A. Funahashi, Y. Itoh, and K. Suzuki (2018) Adaptive behavior acquisition of a robot based on affective feedback and improvised teleoperation. IEEE Transactions on Cognitive and Developmental Systems. Cited by: §1.
  • G. Metta, G. Sandini, D. Vernon, L. Natale, and F. Nori (2008) The icub humanoid robot: an open platform for research in embodied cognition. In Proceedings of the 8th workshop on performance metrics for intelligent systems, pp. 50–56. Cited by: §2.
  • A. Tanevska, F. Rea, G. Sandini, and A. Sciutti (2018) Designing an affective cognitive architecture for human-humanoid interaction. In Companion of the 2018 ACM/IEEE International Conference on Human-Robot Interaction, pp. 253–254. Cited by: §1.
  • N. T. V. Tuyen, S. Jeong, and N. Y. Chong (2018) Emotional bodily expressions for culturally competent robots through long term human-robot interaction. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2008–2013. Cited by: §1.
  • G. Van de Perre, H. Cao, A. De Beir, P. G. Esteban, D. Lefeber, and B. Vanderborght (2018) Generic method for generating blended gestures and affective functional behaviors for social robots. Autonomous Robots 42 (3), pp. 569–580. Cited by: §1.