Building Second-Order Mental Models for Human-Robot Interaction

by   Connor Brooks, et al.
University of Colorado Boulder

The mental models that humans form of other agents—encapsulating human beliefs about agent goals, intentions, capabilities, and more—create an underlying basis for interaction. These mental models have the potential to affect both the human's decision making during the interaction and the human's subjective assessment of the interaction. In this paper, we surveyed existing methods for modeling how humans view robots, then identified a potential method for improving these estimates through inferring a human's model of a robot agent directly from their actions. Then, we conducted an online study to collect data in a grid-world environment involving humans moving an avatar past a virtual agent. Through our analysis, we demonstrated that participants' action choices leaked information about their mental models of a virtual agent. We conclude by discussing the implications of these findings and the potential for such a method to improve human-robot interactions.


Why robots should be technical: Correcting mental models through technical architecture concepts

Research in social robotics is commonly focused on designing robots that...

Active Dynamical Prospection: Modeling Mental Simulation as Particle Filtering for Sensorimotor Control during Pathfinding

What do humans do when confronted with a common challenge: we know where...

Modeling User Empathy Elicited by a Robot Storyteller

Virtual and robotic agents capable of perceiving human empathy have the ...

Understanding Human Context in 3D Scenes by Learning Spatial Affordances with Virtual Skeleton Models

Robots are often required to operate in environments where humans are no...

AI Challenges in Human-Robot Cognitive Teaming

Among the many anticipated roles for robots in the future is that of bei...

Establishing Appropriate Trust via Critical States

In order to effectively interact with or supervise a robot, humans need ...

Virtual Adversarial Humans finding Hazards in Robot Workplaces

During the planning phase of industrial robot workplaces, hazard analyse...

1 Introduction

Humans are known to form a “Theory of Mind” (ToM) when interacting with other people that models their knowledge of other’s beliefs and intentions [Baker et al.2017]. Research has shown that people are able to perform this inference on internal states even from low-level action information such as movement patterns [Bartlett et al.2019]. This may have implications for human-robot interactions: humans have been shown to readily anthropomorphize and form mental models about robots [Mutlu, Roy, and Šabanović2016] and even behave differently around robots for which they have different mental models. For instance, humans maintain varied proxemic distances for the same robot based on its likeability [Mumm and Mutlu2011] or its movement strategy [Mavrogiannis et al.2019]. These results suggest that humans use some type of ToM when interacting with robots and that these models may affect human actions around robots.

If humans form a ToM about robots, we could enable robots to estimate how a human is affected by a robot by using second-order mental modeling — a robot forming a belief over a human’s model of the robot. A robot or agent might acquire such a model by directly asking a human about their mental model of the agent, but this is an intrusive and potentially inefficient method of gathering information. Another option is to model the human’s belief updates from some initial belief state based on their assumed observations, forming an open-loop model of the human’s beliefs. However, this assumes knowledge of both what observations the human is making and, importantly, what their starting belief about the robot was before the observations began. We hypothesize that information about a human’s mental model of a robot might be gained from observing the human’s actions, enabling inference of second-order mental models.

Intuitively, a human’s low-level actions around another agent (such as a robot) may contain information that could be used to understand how they internally view the agent. To further illustrate this idea, consider a scenario in which people must walk past a robotic arm placing objects into a box. In this case, the interaction between the human and robot is quite brief; however, the human’s mental model of the robot can still affect their choice of actions. If a person is confident that the robot will not move outside the radius of the box in the near future, this person may choose a path that narrowly misses the robot in order to preserve efficiency of their own movements. Another person that believes the robot might soon change its task and swings its end effector to some unknown new location will likely give the robot a wider berth. In these cases, the humans’ differing models of the robot agent’s task-based intention and future movements cause them to choose different actions despite the humans sharing the same goal.

If humans form mental models of agents, their actions may contain information that could be used to infer these models. The question we address in this work is whether low-level human action choices during human-agent interactions “leak” information [Mutlu et al.2009]

that could be used in human-robot interactions to allow robots to perform Bayesian inference of a second-order ToM model. In this paper, we describe a framework for actively updating a belief of a human’s model of an agent based on their actions. Then, we conduct an online study in which humans move a virtual character past a virtual “robot” agent and test our framework’s inference of a person’s model of the virtual agent from their movement choices during this interaction. We also use the data from this study to evaluate if our framework improves estimation of the human’s goal. Our results show that information can be extracted from these low-level actions, although inference of mental models directly from observed actions may be noisy.

2 Background

This work is part of an emerging body of research focused on modeling human decision processes by extracting information from their actions. In particular, our work builds on research into human ToM and methods for learning representative models.

2.1 Modeling Humans for Action Understanding

Prior work has found that correctly predicting human actions through an understanding of human intent may produce safer human-robot interactions [Lasota et al.2017] and more efficient human-robot teams [Kanno, Nakata, and Furuta2003]. In addition, choudhury2019utility choudhury2019utility found that robot planning that utilizes a model of human planning may produce robot plans that generalize better than plans learned without such a model. Toward this end, past work has developed methods for predicting human intent based on low-level human motion [Thompson, Horiuchi, and Kagami2009, Kelley et al.2012, Pérez-D’Arpino and Shah2015] as well as higher-level models of human reasoning based on modeling social forces [Luber et al.2010] or using inverse planning to explain observed actions [Gray et al.2005, Baker, Saxe, and Tenenbaum2009, Sohrabi, Riabov, and Udrea2016].

Instead of directly inferring intent, other approaches perform inverse planning on a model of human decision making. For example, maximum entropy inverse reinforcement learning centers around finding the parameterization of a reward function that best explains a human’s actions, with the assumption that the human is acting approximately optimally to maximize that reward function

[Ziebart et al.2008]. Other frameworks for modeling human actions incorporate intention as part of a larger mental model of the human [Hiatt et al.2017]. The intention-driven dynamics model (IDDM) developed by wang2013probabilistic wang2013probabilistic models human intent as a latent variable that affects the transition function of the environment. trafton2013act trafton2013act create a cognitive architecture for modeling how humans process information, using this to help robots interacting with humans understand how humans will complete various tasks. hiatt2011accommodating hiatt2011accommodating use such an architecture to simulate different beliefs and goals that the human could hold in order to find which set of beliefs and goals best explain actions taken by the human that were unexpected by the robot.

Mental models of humans developed to explain human actions can also model factors in the human mental state beyond goal or intent. For instance, reddy2018you reddy2018you infer a human’s internal dynamics model by assuming suboptimal behaviors when controlling an agent are a result of incorrect internal dynamics. herman2016inverse herman2016inverse develop a gradient-based method for learning both the human’s internal dynamics and reward function simultaneously. tabrez2019explanation tabrez2019explanation identify issues in a human’s reward function that can then be used to provide coaching to a human teammate. Individualization can be done through modeling human psychological factors, such as personality factors [Bera et al.2017], grouping people into “types” [Nikolaidis et al.2015], or inferring people’s navigation strategies [Godoy et al.2016].

Models of a human’s mental state can also be used to estimate a human’s current knowledge or beliefs [Breazeal, Gray, and Berlin2009, Talamadupula et al.2014, Devin and Alami2016, Nikolaidis et al.2017]. Bayesian ToM [Baker, Saxe, and Tenenbaum2011] models how humans perform this inference, casting a human as an observer making inferences about an agent’s beliefs and intentions. One example of this is work by lee2019bayesian lee2019bayesian that estimates a child’s belief about a robot’s current state (listening vs not listening) during a storytelling interaction. Their approach accomplishes our goal of actively inferring a mental model, which they achieve through a filtering process that includes both a forward estimate of the person’s belief after a new observation and an update based on the person’s actions. However, their model is an example of a first-order ToM-based model — forming a model of a person’s belief about factual knowledge (in this case, the robot’s attentive state).

Each of these approaches use models of human reasoning to explain and predict human actions. An agent utilizing these models could be said to be practicing first-order ToM by attributing mental states to a human and modeling these states to understand human actions. However, this kind of modeling does not incorporate humans’ potential views regarding agents themselves as intelligent actors about which humans may form a ToM. Our interest is in performing Bayesian inference of second-order ToM-based models from human actions around an agent, thereby learning about a human’s mental model of the agent.

2.2 Modeling Humans’ Use of Theory of Mind

Second-order mental modeling relies on the premise that humans practice first-order ToM by forming mental models of others. Modeling a human’s use of first-order ToM can improve understanding of their actions, as demonstrated by baker2017rational baker2017rational. Further demonstrating this benefit, jara2016naive jara2016naive create a computational model of how humans reason about others’ decision making based on inferring costs and rewards. This principle has been used to include a ToM in simulation agents, creating better simulations of humans in social scenarios [Pynadath and Marsella2005, Doshi et al.2010, Stuhlmüller and Goodman2014]

. rabinowitz2018machine rabinowitz2018machine create a neural network inspired by ToM that is able to build models of various types of observed agents which can be used for applications such as predicting false beliefs. Models incorporating higher-order ToM better explain human actions in competitive games than models not including ToM

[Yoshida, Dolan, and Friston2008], and modeling how humans reason in multiplayer games can give insight into how people form social strategies such as cooperating or competing [Kleiman-Weiner et al.2016].

Moving beyond first-order ToM, yang2018bayes yang2018bayes found that using higher-order ToM for policy recognition in competitive games provides an advantage over an opponent using a lower-order ToM. In line with these findings, humans tend to use first-order ToM, but can raise this level based on identifying intelligent opponents that are using higher-order ToM in competitive games [Doshi et al.2010, de Weerd, Verbrugge, and Verheij2017]. In general, exploring beyond first-order ToM has shown that there are some benefits to humans using second-order ToM in competitive settings, but there are diminishing returns for higher orders beyond this [de Weerd, Verbrugge, and Verheij2014].

In order to perform second-order mental modeling, some recursive structure must be used to model another agent modeling oneself (for a survey of the many ways in which agents can model one another, see albrecht2018autonomous albrecht2018autonomous). gmytrasiewicz1995rigorous gmytrasiewicz1995rigorous established a framework for recursive agent models, which were expanded on using techniques for belief updating in these models [Gmytrasiewicz, Noh, and Kellogg1998]

. gmytrasiewicz2005framework gmytrasiewicz2005framework further developed these ideas into Interactive Partially Observable Markov Decision Processes (I-POMDPs), which provide a principled framework for finding policies while considering recursive beliefs about other agents. Following work by doshi2012modeling doshi2012modeling that models human recursive reasoning in a game through an I-POMDP, we leverage the I-POMDP framework to model a human’s internal planning around an agent.

An existing approach for building second-order mental models is what we have described above as open-loop mental modeling. baker2014modeling baker2014modeling use Bayesian ToM to simulate a human inferring an agent’s goal(s). huang2017enabling huang2017enabling model humans as using Bayesian IRL to learn about robot objectives, and use this model to plan for actions that will communicate to the human as desired. Similarly, mavrogiannis2018social mavrogiannis2018social anticipate human modeling of robot motion in order to create legible navigation strategies. peltola2018modelling peltola2018modelling use an assumed second-order mental model of a recommender system for interpreting feedback from a user, while gray2014manipulating gray2014manipulating instead use a second-order mental model in an adversarial manner by planning to fool a human competitor. Such research demonstrates that second-order mental modeling has the potential to improve performance of autonomous systems in interactive settings involving humans. These prior approaches are open-loop in nature: they use an assumed human mental model of the robot and update this model according to the human’s observations of the robot’s actions, but do not incorporate feedback into their model from the human’s actions.

The primary idea of our contribution is to create and test the feasibility of a framework for performing Bayesian inference of our estimate of the human’s mental model of an agent based on the human’s actions. We do this by framing the interaction between a human and an agent as an I-POMDP, then inferring the human’s belief about their interactive state. This allows us to estimate the human’s belief over the space of agent mental models without having to learn the human’s entire transition function — i.e., we can model the human’s uncertainty about the agent while still assuming the human knows the effect of actions on the environment. Furthermore, we demonstrate an approach for using this technique to update our estimate of the human’s goal or reward function. Thus, our contribution produces both an estimate of a human’s internal model of another agent and an estimate of human goals while around this other agent. We anticipate that this approach could be used to jointly provide a systematic way to update beliefs about how a human views a robot during an interaction and improve understanding of human actions around robots.

3 Inferring Human Beliefs and Goals from Actions

In order to approach this problem, we set up an Interactive-POMDP to model human planning during an interaction. We apply tools developed for inverse reinforcement learning to this framework in order to update our beliefs about the human’s mental model of an agent from observed human actions. Additionally, we derive a way to use this framework for estimating human goals while taking into account possible mental models of the agent that may be affecting their actions.

3.1 Interactive-POMDPs for Model Inference

The I-POMDP framework [Gmytrasiewicz and Doshi2005] modifies a POMDP to include other agents by creating the notion of an interactive state. Interactive states contain both environmental state and the states (or models) of other agents. We use a simplified version of an I-POMDP that assumes the environment is fully observed by both agents and that models of other agents are subintentional models which are not recursive and instead directly map states to actions.

Formally, let an I-POMDP representing the human’s perspective be given by the tuple , where is a set of interactive states, representing both environment state and the agent model type attributed to the other agent . is a set of actions available to both agents, is the transition function that updates the interactive state given human action and agent action via , and is the human’s reward function.

Given this framework, our desired contribution is to infer part of the human’s belief about their interactive state. Specifically, we want to infer given an environment state-action trajectory . That is, after observing

during a human-agent interaction, we want to determine the probability

that a given mental model is influencing the human’s decision making.

3.2 Inferring Human Mental Models of Other Agents

In order to perform inference from this trajectory, we first model the human’s planning through the state and state-action value functions of our I-POMDP. We model these value functions for an interactive state :


Following work in inverse reinforcement learning on estimating parameters of the Q-function from actions [Ramachandran and Amir2007], we model the human’s action selection as exponentially weighted toward actions with higher expected reward:


where is a normalizing factor.

We use this estimate along with an assumption of stationarity of belief (i.e., the human is not actively updating their mental model of the agent or reward function during this interaction) to obtain the likelihood of the observed trajectory:


In order to do inference on the human’s mental model type, we reverse this probability:


Since we use a uniform prior on all model types, for a given , the most likely model can be found by finding the model that maximizes the probability of the observed trajectory. Using the iterative update implied by Eq. 4, we can update our estimate of the human’s mental model type as we observe new actions during an interaction.

3.3 Joint Inference of Human Mental Model and Reward

The prior section assumes full knowledge of the human’s reward function in calculating the state-action value function. However, in many cases, we may not know either the human’s mental model of the agent or the human’s goal. In these cases, we can jointly reason over both possible mental models and possible goals.

To do this, let us assume we have some set of possible goals, , that are used to parameterize the human’s reward function: . We denote the state-action value function for a particular goal as . Rewriting Eq. 3 and Eq. 4 with produces the probability of a trajectory jointly conditioned on model and goal:


This gives us the joint probability of model type and goal:


This joint probability can be used to find either the probability of model type or goal by marginalizing out the other variable. In our experiment, we marginalize across a discrete set of model types to find the probability of a goal from an observed trajectory:


4 Evaluation

We conducted a within-subjects study to collect data on how humans incorporate mental models of autonomous agents into planning. We recruited participants from Mechanical Turk for this study. Turkers were required to have at least a approval rate, and we limited the participants to U.S. IP addresses to limit language translation issues with our experiment instructions. During the experiment, participants control an avatar that moves through a grid-world in order to reach a designated goal. This grid-world contains a second agent as well: a virtual “robot” agent with which the participant is required to avoid collisions. This grid-world and the interface used by participants is shown in Fig. 1. This task is meant to simulate a simple, passive human-robot interaction in which a human needs to move past a robot in order to continue along the human’s intended path.

4.1 Experimental Procedure

Participants complete 5 trials per condition. For each trial, participants are given one of three goals on the top row of the grid world. The positions of these three possible goals requires the participant to move their avatar past the agent, which starts randomly in the 3x3 section directly in the middle of the grid-world. Participant and agent movements are done in a turn-based manner at discrete time steps. There are 4 different strategies for agent movement, creating 4 conditions:

Stationary Agent: In this baseline condition, the virtual agent simply stays still for the duration of the trial.

Random Agent: The virtual agent chooses actions purely randomly.

Fixed-Goal Agent: The virtual agent moves toward its goal, which is at the participant’s starting location.

Chasing Agent: The virtual agent chases the participant by choosing actions at each state that move it closer to the human agent. Equivalently, the agent moves toward a goal at the participant’s current location.

Figure 1: The grid-world game played by participants. The participant’s character is the filled-in circle, while the robot agent is represented by the crossed circle. The current goal is represented as a green, filled-in square. For illustrative purposes, all three possible goal locations are labeled with “G” labels, and the three possible participant starting locations are denoted with “S” labels.

We design our experiment to ensure that participants have an accurate mental model of the agent before each trial begins so that we can test our hypotheses using this known mental model as the ground-truth. To accomplish this, before each condition’s trials begins, participants are informed of the strategy that the agent will be using to select actions. After this, participants complete a practice trial in which they move through the grid-world, including moving past the appropriate agent used in the next set of trials. Once participants end this practice trial, they answer a single question to ensure they correctly understand the agent’s movement strategy. This question consists of displaying a sample game state and asking the participant in which the direction the agent would move next based on the agent type for the upcoming trials. If participants incorrectly answer this question, they are taken back to the description of the “robot” agent model; this loop continues until participants answer correctly. The purpose of this procedure is to ensure the participants are internally using the correct mental model of the agent for each condition. Additionally, the practice trial serves to demonstrate the agent behavior to the participant so that they do not act according to surprise or reflexive behavior during the recorded trials.

If the participant collides with the agent during a trial, the trial is reset. Participants complete 5 successful trials with each condition before moving to the next (only trajectories from successful trials were used in our analysis). Finally, the ordering of conditions is randomized in order to mitigate biasing from ordering effects.

5 Hypotheses

We hypothesize that our framework for Bayesian inference of second-order mental models will enable improved estimation of both a human’s view of an agent and their goal:

Hypothesis I: Given a known goal, human actions will leak information that can be utilized by second-order mental modeling to infer the mental model the human holds of the agent.

Hypothesis II: Estimation of a human’s goal via marginalizing the joint probability of goal and agent mental model will lead to improved estimation compared with directly estimating the goal while assuming the human treats the agent merely as an obstacle.

6 Analysis

The purpose of our study setup is to record trajectories in which we know the human’s mental model of the agent is correct due to the explanation and practice period before each trial. This allows us to test our framework’s ability to infer this ground-truth mental model directly from the human action sequences.

First, we investigate our primary question of whether our framework can infer the human’s mental model of the agent from their actions by evaluating the accuracy of what our framework judges to be the most likely model based on an observed trajectory with a known goal. To do this, we set uniform priors over the four agent models used for our four conditions, then update these probabilities based on a trajectory. We calculate accuracy based on whether or not the agent model assigned the highest probability at the end of the trajectory matches the correct model for that trial.

Second, we compare the probability assigned to the correct goal over time using our method for joint estimation of agent model and goal compared to estimation of the goal while assuming the human views the agent as an obstacle (which is equivalent to using the stationary agent model at all times). We use a discrete goal space consisting of the three goals which we used in data collection: the top-left corner, top-middle, and top-right corner of the grid-world. After each action, we update and record the probability assigned to the correct goal by each method. The probabilities are presented by the percent of the trajectory observed rounded to intervals, averaged across all trials.

6.1 Implementation Modifications for Turn-Based Games

In our experiment, the human and agent take turns making movements, with each able to see the results of the other’s turn before choosing their own action. In order to accommodate this type of turn-based interaction into our framework, we expand the human’s state-action value functions to include the expected next turn of the agent. Essentially, this allows the Q-function to represent a full turn (both human and agent move) including both the given human action and an expectation over possible agent actions. For more compact notation, we represent the value function for a agent turn as , producing the following equations:




6.2 Tuning the Reward Function

In order to produce the state-action value function, we need to estimate the reward assigned to state-action pairs by participants. To do this, we create a simple reward function with three parameters: reward for reaching the goal , cost per action , and discount factor . We set aside data from participants for tuning these parameters (only the data from the other participants was used for testing), then perform a grid search to find the parameters that maximize the probabilities of the observed trajectories for this set of participants. This produced the following values: , , .

7 Results

7.1 Inference of the Participant’s Robot Model from Actions

The results of the Top-1 and Top-2 accuracy on the trajectories from the test participants are presented in Table 1. An improvement in classification accuracy over the accuracy due to chance demonstrates that the actions from participants in our experiment do leak information about their mental model of the agent.

Fig. 2

visualizes the confusion matrix of the top classification for each trial (for classifications in which there is a tie, the classification is split between all tied classes). As the confusion matrix shows, for each true model type, the correct model type is the most common classification by our method. Furthermore, some pairs of models are more frequently misclassified than other pairs — we discuss this result in more detail below.

Model Classification Accuracy
Top-1 Accuracy Top-2 Accuracy
I-POMDP State Inference 45.2% 78.5%
Chance 25.0% 50.0%
Table 1: Accuracies of model classification from observed trajectories.
Figure 2: Confusion matrix of model classification from observed trajectories.

7.2 Estimation of the Participant’s Goal

Fig. 3 visualizes the probability assigned to the correct goal over percent of the trajectory observed, averaged across all trials. An initial uniform probability of the three possible goals is assumed, which is then updated at each timestep according to the procedure laid out previously. In both cases, the correct goal is nearly always inferred with high confidence as the human approaches the goal. Our intent in visualizing these probabilities over time through the trajectories is to examine if joint inference of the model and goal has an effect on this inference as compared to assuming a fixed agent location at each action step (Hypothesis 2). We found that the two methods performed similarly, with the assumed stationary agent slightly outperforming joint inference in initial action observations, while the joint inference method assigns slightly higher probabilities to the correct goal as more actions have been observed.

8 Discussion

There are a few key takeaways from the results of our analysis. First, low-level actions do provide information about a human’s mental model of a local agent, even without considering the human’s observations. This is shown through our I-POMDP state inference method outperforming the accuracy due to chance at classification of the participant’s ground-truth mental model of the agent. However, our method is not robust in this scenario; the classifier only selects the correct model approximately half of the time. Nevertheless, the results demonstrate that there is potential to learn these second-order mental models from human actions around agents such as robots, even during brief, passive interactions. Furthermore, these second-order mental models are learned using a model of actions as independent choices (as opposed to considering trajectories which could implicitly encode the behavior of the agent). This shows that information about the participants’ view of the agent is contained directly in some subset of individual action selections, even without taking into account the agent’s actual movement or behavior. Thus, we found support for Hypothesis 1, although more work is needed to improve classification accuracy.

Second, an interesting pattern is shown by the confusion matrix from the classification (Fig. 2). The confusion matrix demonstrates that the classification errors are predominantly between the pairs of Stationary and Fixed-Goal models, and Random and Chasing models. Intuitively, these pairs of models represent the agent which are “safe” to pass directly next to (the Stationary and Fixed-Goal models) due to either not moving or having a pre-determined path that the participant can easily avoid, and the agents which are “unsafe” to pass directly adjacent to (the Random and Chasing models) due to the potential for collision from uncertain or hostile agent models. This clustering demonstrates the potential for actions to leak information not only about specific models that the human holds about an agent, but perhaps also higher-level information about more abstract mental models that might encompass multiple specific models.

Finally, the benefit to goal inference from our method is unclear. While our joint model and goal inference method produces marignally (but not significantly) higher final probabilities assigned to the correct goal compared with treating the agent as a fixed object in the view of the human while modeling action probabilities, the difference is quite small and overall the two methods perform similarly. Consequently, we do not find support for Hypothesis 2. Further testing on more complex interaction scenarios is required to determine whether or not there is a benefit to using this method for goal inference.

Figure 3: The probability assigned to the true goal by the percent of the trajectory observed.

9 Future Work

Through this paper, we have laid out a formulation for what we hope will be a useful way to improve understanding of how humans view robots during an interaction. Our experiment demonstrates that there is indeed information embedded in low-level human actions that can be used for second-order mental modeling. However, further work is needed to apply these ideas to real-world human-robot interactions.

First, in this paper, we have proposed a framework for performing Bayesian inference of second-order mental models and investigated the feasibility of doing so from low-level human actions. In order to establish this approach clearly, we have shown this on a simple task: a fully-observable grid-world with discrete state, action, reward, and model spaces. Relaxing these assumptions in future work would create a more powerful tool for understanding human actions without prior knowledge of possible human mental states.

Second, while we describe and evaluate our framework for Bayesian inference of second-order mental models in isolation in order to study its plausibility, an implementation in a more complex task ought to combine this framework with existing methods for open-loop modeling that take into consideration a human’s observations during the interaction. Thus, this framework could serve to close the loop in an existing open-loop modeling framework. This combined framework could also relax one of the assumptions made in our formulation: that the human’s belief about the agent’s model type is stationary. Such a framework would first model forward propagation of belief based on the human’s observations, then update this prior by taking into account the human’s action according to our method developed in this paper. In this way, our method and existing open-loop modeling could be used together to utilize both human observations and human actions in updating the belief about the human’s mental model without assuming either a certain initial model or a stationary model.

Finally, we evaluate our framework in this paper using a human-agent interaction scenario with greatly simplified dynamics and interaction possibilities. While we have developed this framework with human-robot interactions in mind, this initial study only directly demonstrates leaking of information about mental models in an online human-agent interaction. We look forward to conducting further studies with humans and physically-embodied robots pariticipating in passive, teaming, and adversarial interactions in order to further examine whether this leakage occurs and is useful in these scenarios.

10 Conclusion

If humans use a Theory of Mind about robots during interactions, second-order mental modeling might provide information that could be used by robots to be aware of their effect on the people around them. We have shown that human action choices around agents in a grid-world environment do leak information about a human’s view of such an agent. In doing so, we have provided a justification for closing the loop in open-loop mental models of humans in order to take into account this information as human actions around robots are observed.

A robot planning with an awareness of its effect on humans provides many possible benefits. For instance, such a robot could identify errors with a human’s view of the robot that might lead to sub-optimal interactions, then communicate to fix these errors in order to improve the remainder of the interaction. Actively assessing a human’s view of a robot provides a path for deepening and improving understanding between humans and robots during human-robot interactions.


  • [Albrecht and Stone2018] Albrecht, S. V., and Stone, P. 2018. Autonomous agents modelling other agents: A comprehensive survey and open problems. Artificial Intelligence 258:66–95.
  • [Baker and Tenenbaum2014] Baker, C. L., and Tenenbaum, J. B. 2014. Modeling human plan recognition using bayesian theory of mind. Plan, activity, and intent recognition: Theory and practice 177–204.
  • [Baker et al.2017] Baker, C. L.; Jara-Ettinger, J.; Saxe, R.; and Tenenbaum, J. B. 2017. Rational quantitative attribution of beliefs, desires and percepts in human mentalizing. Nature Human Behaviour 1(4):0064.
  • [Baker, Saxe, and Tenenbaum2009] Baker, C. L.; Saxe, R.; and Tenenbaum, J. B. 2009. Action understanding as inverse planning. Cognition 113(3):329–349.
  • [Baker, Saxe, and Tenenbaum2011] Baker, C.; Saxe, R.; and Tenenbaum, J. 2011. Bayesian theory of mind: Modeling joint belief-desire attribution. In Proceedings of the annual meeting of the cognitive science society, volume 33.
  • [Bartlett et al.2019] Bartlett, M.; Edmunds, C.; Belpaeme, T.; Thill, S.; and Lemaignan, S. 2019. What can you see? identifying cues on internal states from the movements of natural social interactions. Frontiers in Robotics and AI 6:49.
  • [Bera et al.2017] Bera, A.; Randhavane, T.; Prinja, R.; and Manocha, D. 2017. Sociosense: Robot navigation amongst pedestrians with social and psychological constraints. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 7018–7025. IEEE.
  • [Breazeal, Gray, and Berlin2009] Breazeal, C.; Gray, J.; and Berlin, M. 2009. An embodied cognition approach to mindreading skills for socially intelligent robots. The International Journal of Robotics Research 28(5):656–680.
  • [Choudhury et al.2019] Choudhury, R.; Swamy, G.; Hadfield-Menell, D.; and Dragan, A. D. 2019. On the utility of model learning in hri. In 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI), 317–325. IEEE.
  • [de Weerd, Verbrugge, and Verheij2014] de Weerd, H.; Verbrugge, R.; and Verheij, B. 2014. Agent-based models for higher-order theory of mind. In Advances in Social Simulation. Springer. 213–224.
  • [de Weerd, Verbrugge, and Verheij2017] de Weerd, H.; Verbrugge, R.; and Verheij, B. 2017. Negotiating with other minds: the role of recursive theory of mind in negotiation with incomplete information. Autonomous Agents and Multi-Agent Systems 31(2):250–287.
  • [Devin and Alami2016] Devin, S., and Alami, R. 2016. An implemented theory of mind to improve human-robot shared plans execution. In 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI), 319–326. IEEE.
  • [Doshi et al.2010] Doshi, P.; Qu, X.; Goodie, A.; and Young, D. 2010. Modeling recursive reasoning by humans using empirically informed interactive pomdps. In Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems: volume 1-Volume 1, 1223–1230. International Foundation for Autonomous Agents and Multiagent Systems.
  • [Doshi et al.2012] Doshi, P.; Qu, X.; Goodie, A. S.; and Young, D. L. 2012. Modeling human recursive reasoning using empirically informed interactive partially observable markov decision processes. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans 42(6):1529–1542.
  • [Gmytrasiewicz and Doshi2005] Gmytrasiewicz, P. J., and Doshi, P. 2005. A framework for sequential planning in multi-agent settings. Journal of Artificial Intelligence Research 24:49–79.
  • [Gmytrasiewicz and Durfee1995] Gmytrasiewicz, P. J., and Durfee, E. H. 1995. A rigorous, operational formalization of recursive modeling. In ICMAS, 125–132.
  • [Gmytrasiewicz, Noh, and Kellogg1998] Gmytrasiewicz, P. J.; Noh, S.; and Kellogg, T. 1998. Bayesian update of recursive agent models. User Modeling and User-Adapted Interaction 8(1-2):49–69.
  • [Godoy et al.2016] Godoy, J.; Karamouzas, I.; Guy, S. J.; and Gini, M. L. 2016. Moving in a crowd: Safe and efficient navigation among heterogeneous agents. In IJCAI, 294–300.
  • [Gray and Breazeal2014] Gray, J., and Breazeal, C. 2014. Manipulating mental states through physical action. International Journal of Social Robotics 6(3):315–327.
  • [Gray et al.2005] Gray, J.; Breazeal, C.; Berlin, M.; Brooks, A.; and Lieberman, J. 2005. Action parsing and goal inference using self as simulator. In ROMAN 2005. IEEE International Workshop on Robot and Human Interactive Communication, 2005., 202–209. IEEE.
  • [Herman et al.2016] Herman, M.; Gindele, T.; Wagner, J.; Schmitt, F.; and Burgard, W. 2016. Inverse reinforcement learning with simultaneous estimation of rewards and dynamics. In Artificial Intelligence and Statistics, 102–110.
  • [Hiatt et al.2017] Hiatt, L. M.; Narber, C.; Bekele, E.; Khemlani, S. S.; and Trafton, J. G. 2017. Human modeling for human–robot collaboration. The International Journal of Robotics Research 36(5-7):580–596.
  • [Hiatt, Harrison, and Trafton2011] Hiatt, L. M.; Harrison, A. M.; and Trafton, J. G. 2011. Accommodating human variability in human-robot teams through theory of mind. In IJCAI Proceedings-International Joint Conference on Artificial Intelligence, volume 22, 2066.
  • [Huang et al.2017] Huang, S. H.; Held, D.; Abbeel, P.; and Dragan, A. D. 2017. Enabling robots to communicate their objectives. Autonomous Robots 1–18.
  • [Jara-Ettinger et al.2016] Jara-Ettinger, J.; Gweon, H.; Schulz, L. E.; and Tenenbaum, J. B. 2016. The naïve utility calculus: Computational principles underlying commonsense psychology. Trends in cognitive sciences 20(8):589–604.
  • [Kanno, Nakata, and Furuta2003] Kanno, T.; Nakata, K.; and Furuta, K. 2003. A method for team intention inference. International Journal of Human-Computer Studies 58(4):393–413.
  • [Kelley et al.2012] Kelley, R.; Browne, K.; Wigand, L.; Nicolescu, M.; Hamilton, B.; and Nicolescu, M. 2012. Deep networks for predicting human intent with respect to objects. In 2012 7th ACM/IEEE International Conference on Human-Robot Interaction (HRI), 171–172. IEEE.
  • [Kleiman-Weiner et al.2016] Kleiman-Weiner, M.; Ho, M. K.; Austerweil, J. L.; Littman, M. L.; and Tenenbaum, J. B. 2016. Coordinate to cooperate or compete: abstract goals and joint intentions in social interaction. In CogSci.
  • [Lasota et al.2017] Lasota, P. A.; Fong, T.; Shah, J. A.; et al. 2017. A survey of methods for safe human-robot interaction. Foundations and Trends® in Robotics 5(4):261–349.
  • [Lee, Sha, and Breazeal2019] Lee, J. J.; Sha, F.; and Breazeal, C. 2019. A bayesian theory of mind approach to nonverbal communication. In 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI), 487–496. IEEE.
  • [Luber et al.2010] Luber, M.; Stork, J. A.; Tipaldi, G. D.; and Arras, K. O. 2010. People tracking with human motion predictions from social forces. In 2010 IEEE International Conference on Robotics and Automation, 464–469. IEEE.
  • [Mavrogiannis et al.2019] Mavrogiannis, C.; Hutchinson, A. M.; Macdonald, J.; Alves-Oliveira, P.; and Knepper, R. A. 2019. Effects of distinct robot navigation strategies on human behavior in a crowded environment. In 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI), 421–430. IEEE.
  • [Mavrogiannis, Thomason, and Knepper2018] Mavrogiannis, C. I.; Thomason, W. B.; and Knepper, R. A. 2018. Social momentum: A framework for legible navigation in dynamic multi-agent environments. In Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction, 361–369. ACM.
  • [Mumm and Mutlu2011] Mumm, J., and Mutlu, B. 2011. Human-robot proxemics: physical and psychological distancing in human-robot interaction. In Proceedings of the 6th international conference on Human-robot interaction, 331–338. ACM.
  • [Mutlu et al.2009] Mutlu, B.; Yamaoka, F.; Kanda, T.; Ishiguro, H.; and Hagita, N. 2009. Nonverbal leakage in robots: communication of intentions through seemingly unintentional behavior. In Proceedings of the 4th ACM/IEEE international conference on Human robot interaction, 69–76. ACM.
  • [Mutlu, Roy, and Šabanović2016] Mutlu, B.; Roy, N.; and Šabanović, S. 2016. Cognitive human–robot interaction. In Springer Handbook of Robotics. Springer. 1907–1934.
  • [Nikolaidis et al.2015] Nikolaidis, S.; Ramakrishnan, R.; Gu, K.; and Shah, J. 2015. Efficient model learning from joint-action demonstrations for human-robot collaborative tasks. In Proceedings of the tenth annual ACM/IEEE international conference on human-robot interaction, 189–196. ACM.
  • [Nikolaidis et al.2017] Nikolaidis, S.; Nath, S.; Procaccia, A. D.; and Srinivasa, S. 2017. Game-theoretic modeling of human adaptation in human-robot collaboration. In Proceedings of the 2017 ACM/IEEE international conference on human-robot interaction, 323–331. ACM.
  • [Peltola et al.2018] Peltola, T.; Celikok, M. M.; Daee, P.; and Kaski, S. 2018. Modelling user’s theory of ai’s mind in interactive intelligent systems. arXiv preprint arXiv:1809.02869.
  • [Pérez-D’Arpino and Shah2015] Pérez-D’Arpino, C., and Shah, J. A. 2015. Fast target prediction of human reaching motion for cooperative human-robot manipulation tasks using time series classification. In 2015 IEEE international conference on robotics and automation (ICRA), 6175–6182. IEEE.
  • [Pynadath and Marsella2005] Pynadath, D. V., and Marsella, S. C. 2005. Psychsim: Modeling theory of mind with decision-theoretic agents. In IJCAI, volume 5, 1181–1186.
  • [Rabinowitz et al.2018] Rabinowitz, N. C.; Perbet, F.; Song, H. F.; Zhang, C.; Eslami, S.; and Botvinick, M. 2018. Machine theory of mind. arXiv preprint arXiv:1802.07740.
  • [Ramachandran and Amir2007] Ramachandran, D., and Amir, E. 2007. Bayesian inverse reinforcement learning. In IJCAI, volume 7, 2586–2591.
  • [Reddy, Dragan, and Levine2018] Reddy, S.; Dragan, A. D.; and Levine, S. 2018. Where do you think you’re going?: Inferring beliefs about dynamics from behavior. arXiv preprint arXiv:1805.08010.
  • [Sohrabi, Riabov, and Udrea2016] Sohrabi, S.; Riabov, A. V.; and Udrea, O. 2016. Plan recognition as planning revisited. In IJCAI, 3258–3264.
  • [Stuhlmüller and Goodman2014] Stuhlmüller, A., and Goodman, N. D. 2014. Reasoning about reasoning by nested conditioning: Modeling theory of mind with probabilistic programs. Cognitive Systems Research 28:80–99.
  • [Tabrez, Agrawal, and Hayes2019] Tabrez, A.; Agrawal, S.; and Hayes, B. 2019. Explanation-based reward coaching to improve human performance via reinforcement learning. In 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI), 249–257. IEEE.
  • [Talamadupula et al.2014] Talamadupula, K.; Briggs, G.; Chakraborti, T.; Scheutz, M.; and Kambhampati, S. 2014. Coordination in human-robot teams using mental modeling and plan recognition. In Intelligent Robots and Systems (IROS 2014), 2014 IEEE/RSJ International Conference on, 2957–2962. IEEE.
  • [Thompson, Horiuchi, and Kagami2009] Thompson, S.; Horiuchi, T.; and Kagami, S. 2009. A probabilistic model of human motion and navigation intent for mobile robot path planning. In 2009 4th International Conference on Autonomous Robots and Agents, 663–668. IEEE.
  • [Trafton et al.2013] Trafton, J. G.; Hiatt, L. M.; Harrison, A. M.; Tamborello II, F. P.; Khemlani, S. S.; and Schultz, A. C. 2013. Act-r/e: An embodied cognitive architecture for human-robot interaction. Journal of Human-Robot Interaction 2(1):30–55.
  • [Wang et al.2013] Wang, Z.; Mülling, K.; Deisenroth, M. P.; Ben Amor, H.; Vogt, D.; Schölkopf, B.; and Peters, J. 2013. Probabilistic movement modeling for intention inference in human–robot interaction. The International Journal of Robotics Research 32(7):841–858.
  • [Yang et al.2018] Yang, T.; Meng, Z.; Hao, J.; Zhang, C.; and Zheng, Y. 2018. Bayes-tomop: A fast detection and best response algorithm towards sophisticated opponents. arXiv preprint arXiv:1809.04240.
  • [Yoshida, Dolan, and Friston2008] Yoshida, W.; Dolan, R. J.; and Friston, K. J. 2008. Game theory of mind. PLoS computational biology 4(12):e1000254.
  • [Ziebart et al.2008] Ziebart, B. D.; Maas, A. L.; Bagnell, J. A.; and Dey, A. K. 2008. Maximum entropy inverse reinforcement learning. In AAAI, volume 8, 1433–1438. Chicago, IL, USA.