There has been remarkable progress in language modeling Jozefowicz et al. (2016); Devlin et al. (2018); Radford et al. (2019) and building dialogue agents Dinan et al. (2019a). Nevertheless, the current state of the art uses only the statistical regularities of language data, without explicit understanding of the world that the language describes. This work is built on the hypothesis that dialogue agents embodied in a rich and cohesive (but tractable) world can more easily be trained to use language effectively than those only exposed to standard large-scale text-only corpora.
To that end, we introduce the LIGHT111 Learning in Interactive Games with Humans and Text. research platform. LIGHT is a multi-player fantasy text adventure world designed for studying situated dialogue, and allows interactions between humans, models as embodied agents, and the world itself. It consists of a large crowdsourced game world (663 locations, 3462 objects and 1755 characters) described entirely in natural language. Within that game world, we collect a large set (11k episodes) of character-driven human-human crowdworker interactions involving actions, emotes, and dialogue, with the aim of training models to engage humans in a similar fashion. Our framework is made publicly available in ParlAI (http://parl.ai/projects/light).
We use the collected dataset to investigate how a model can both speak and act grounded in perception of its environment and dialogue from other speakers. This is done by evaluating state-of-the-art models on our task and evaluating the effects of providing additional grounding. In particular, we adapt the BERT contextual language model (Devlin et al., 2018) to the task of dialogue in two ways: as a bi-ranker, which is fast and practical as a retrieval model, and as a cross-ranker which is slower at inference time but allows more feature cross-correlation between context and response. Both models outperform existing methods. Our ablation analysis shows the importance of each part of the grounding (location, objects, characters, other’s actions, self-actions) in terms of the ability to both understand and use language. While models that use grounding show clear improvements, our best performing models are still unable to perform at human level, making our setup a suitable challenge for future research.
2 Related Work
Most recent work in dialogue exploring generative or retrieval models for goal-directed (Henderson et al., 2014; Bordes et al., 2017) or chit-chat tasks (Vinyals and Le, 2015; Sordoni et al., 2015; Zhang et al., 2018) is not situated, or even grounded in perception. Models typically take the last few utterances from the dialogue history as input, and output a new utterance. While some goal-directed setups may use external knowledge bases (e.g. flight data for airline booking), dialogues tend to implicitly refer to an external world during the conversations without explicit grounding to objects or actions.
Several position papers have proposed virtual embodiment as a strategy for language research Brooks (1991); Kiela et al. (2016); Gauthier and Mordatch (2016); Mikolov et al. (2016); Lake et al. (2017)
. Single-player text adventure game frameworks for training reinforcement learning agents exist, i.e., narasimhan2015language and TextWorldCôté et al. (2018), but these do not have human dialogue within the game. yang2017mastering and bordes2010towards proposed small world setups for instruction following or labeling, but these are much more restricted than the large multi-player text adventure game environment with rich dialogue that we propose here.
A number of visual, rather than text, platforms have been proposed such as House3D Wu et al. (2018b), HoME Brodeur et al. (2017), MINOS Savva et al. (2017), Matterport3D Chang et al. (2017) and AI2-THOR Kolve et al. (2017), and the Minecraft MALMO project Johnson et al. (2016), but they typically are suited to reinforcement learning of actions, and involve templated language for navigation or question answering tasks, if at all Oh et al. (2017); Yi et al. (2018).
Other examples are instruction-following in the Neverwinter Nights game Fleischman and Roy (2005), dialogue about soccer videogames Pasunuru and Bansal (2018), placing blocks appropriately given a final plan Wang et al. (2016) and a more open ended building task using a grid of voxels Wang et al. (2017). In the latter two cases the communication is one-sided with only the human issuing instructions, rather than dialogue, with the agent only able to act.
There are also setups that consider static language and perception, for example image captioning Lin et al. (2014), video captioning Yu et al. (2016), visual QA Antol et al. (2015) and visual dialogue Das et al. (2017); Shuster et al. (2018); Mostafazadeh et al. (2017). While grounded, the agent has no ability to act in these tasks. Talk the Walk de Vries et al. (2018) introduces a navigation game that involves action, perception and two-way dialogue, but is limited to small grids.
In summary, compared to many setups, our framework allows learning from both actions and (two-way) dialogue, while many existing simulations typically address one or the other but not both. In addition, being based on a gaming setup, our hope is that LIGHT can be fun for humans to interact with, enabling future engagement with our models. All utterances in LIGHT are produced by human annotators, thus inheriting properties of natural language such as ambiguity and coreference, making it a challenging platform for grounded learning of language and actions.
3 LIGHT Environment and Task Setup
LIGHT is a large-scale, configurable text adventure environment for research on learning grounded language and actions. It features both humans and models as embodied agents within a multi-player fantasy MUD (multi-user dungeon)-like Dieterle (2009) environment.
To facilitate natural human-sourced (fantasy) situations described by natural language, almost the entire environment is crowdsourced, including locations, objects and their affordances, characters and their personalities, and most importantly character interactions: dialogues and actions. These components are collected through a series of annotation tasks that we will now describe. These tasks are designed so that they can be combinatorially recombined. Data quality was maintained by requiring annotators to take a test (see Appendix D). Overall statistics of the collected elements are given in Table 1. This environment can then be used to both train agents, and to evaluate them in situ via their online interactions.
We first crowdsourced a set of 663 game location settings from a base set of 37 categories (countryside, forest, inside/outside castle, shore, graveyard, bazaar, …– full list in Appendix H) which were selected by us to provide both inspiration and cohesion to annotators. Workers were provided a category and asked to create a description, backstory, names of connected locations, and contained objects and characters. See Table 2 for an example. Many descriptions are quite detailed, and there are clear semantics between entities (e.g. alligators being in swamps, cacti in a desert).
As all remaining tasks build upon the locations created in this first step, we selected 6 location categories (underwater aquapolis, frozen tundra, supernatural, magical realm, city in the clouds, and netherworld) designed to be distinct from the others to provide an isolated set of locations, characters, and objects for testing. These will be used to build what we refer to as an unseen test set.
Each location is collected independently, with the eventual aim that they can be glued together as desired to randomize world generation. In this work, we consider actions and dialogues within a single location, so building a world map is not necessary. However, we will show that the environment has considerable influence on the dialogue, actions and grounded learning of models.
We crowdsourced 1755 game characters from animals to trolls and orcs to humans of various types (wizards, knights, village clerk). See Table 2 for detailed examples. Each character has a textual description, a persona (defined as a set of 3-5 profile sentences describing their traits, modeled after the Persona-Chat dataset Zhang et al. (2018)), and a set of objects that are currently being carried, wielded, or worn. We sourced this list of characters to annotate from the ones provided in the location creation task.
We crowdsourced 3462 objects, each with a textual description, and a set of affordances (whether it is a container, can be picked up, has a surface, is a weapon, is wearable, is food, is a drink). See Table 2 for examples. As before, we sourced this list of objects to annotate from the ones annotated for the locations and characters.
Actions and Emotes
There are a set of actions in the game consisting of physical manipulations, and a set of emotes that display feelings to other characters, in line with existing MUDs.
Physical actions include get, drop, put, give, steal, wear, remove, eat, drink, hug and hit, each taking either one or two arguments, e.g. put robes in closet. Every action has an explicit unambiguous effect on the underlying game state, and can only be executed if constraints are met, e.g. if the agent is holding the robes in the latter example.
Emotes include applaud, blush, cringe, cry, dance, frown …, sulk, wave, wink (22 in total) and have no effect on the game state other than to notify nearby characters of the emote, which can have effects on their behavior. See Appendix E for further detailed descriptions.
Now that we have a fully realized underlying environment, we can attempt to learn and evaluate agents that can act and speak within it. For this, we collect a human-human dataset of episodic interactions within the environment.
For each dialogue, we place two characters in a random location (either two characters that were already assigned to it, or else randomly assigned characters), complete with the objects assigned to the location and to those characters. Each character has access to their persona, the location description, and the objects present, and the interaction episode begins. The two characters take turns within the episode, and can execute one action (physical action or emote) and produce one dialogue utterance on each turn. We crowdsourced 10,777 dialogues. Examples are given in Figure 1 and Appendix Figures 10-16.
Seen and Unseen Test Sets
We provide two distinct test sets. The seen test set consists of dialogues set in the same world (set of locations) as the training set, thus also consists of characters, objects, and personas that can appear in the training data. In contrast, the unseen test set is comprised of dialogues collected on the unseen set of locations. The unseen test set allows for evaluation of generalization capability to unseen topics in a similar domain and as we shall see, provides a more challenging test for current techniques.
|Persona 1: Servant||Persona 2: King|
|I come from the lower class.||I am a king of the whole empire.|
|I do what I am told without question.||I give rules and pursuit them.|
|I can not read.||I am brave and fearless.|
|I have not seen my family in a long time.|
|Carrying: a duster, a small bucket, a rag||Carrying: a crown, a scepter|
|Wearing: a shirt|
|Setting: The main foyer is massive. A grand staircase sits to the back of the foyer leading to the upstairs.|
|At the front of the foyer stand two servants ready to help anyone who comes to visit. To the left of the|
|room there is a doorway leading into a corridor. To the right there is a door leading to another corridor|
|for the King’s servants. At the foot of the stairs there is a bearskin rug that is staring at you almost as|
|if still hungry. The walls are lined with portraits of the king and his family.|
Servant: my humble king. What am I to do to serve you?
|King: Ahhh. My loyal servant. Polish my scepter.|
|Action: give scepter to servant|
|Servant: Yes my lord. I will polish it immediately. Am I to return it to you personally?|
|Action: put scepter in small bucket|
|King: Yes. Yes. Of course. Also check the jewels in my crown. They seem loose.|
|Action: give crown to servant|
|Servant: But sire I am not qualified to do that. Would you prefer I take it to someone?|
|King: Oh fine then.|
|Emote: gesture sigh|
|Servant: I am sorry sir the rug startled me|
|Action: drop crown|
|King: Haha! That’s bear I slain on my latest hunting trip. He’s a mighty beast!|
|Emote: gesture laugh|
|Servant: and if I may ask where did you go hunting sire?|
|King: The great woods of course. This bear was stealing children in the kingdom. Surely you heard about it.|
|Servant: sire. I have not been outside of these walls in quiet some time. I have not seen my family in ages.|
|King: Such is the life of a servant I suppose. How’s that scepter looking?|
|Servant: it is almost ready sire. and the crown who would you like me to take it to?|
|Action: get scepter from small bucket|
|King: Here just give it back. I’ll have the queen find someone.|
4 Learning Methods
We consider a variety of models that can predict actions, emotes and dialogue, and explore the importance of grounding upon the location, objects, and other characters within the setting. For all models, we represent context as a large text sequence with a special token preceding each input type (persona, setting, self emote, partner emote, etc.). We work with two model classes: ranking models that output the maximal scoring response from a set of potential candidate responses and generative models that decode word by word.
Baseline Ranking Methods
We report a Random baseline (selecting a random candidate from the candidates) and an Information Retrieval (IR) baseline that uses word overlap with TF/IDF weighting. We use Starspace Wu et al. (2018a) which learns a bag-of-words embedding for context and candidates to maximize the inner product of the true label using a ranking loss. Lastly, we use fastText Joulin et al. (2016)
to classify which emote should be predicted next as there are only 22 classes. Finally, we compare the performance of our best models to human performance on each of the prediction tasks.
|chicken coop||Pirate swords||the remains||shovel||Ale bottles||flower pot|
|a pen for the chickens||cargo||bones||a garden||mug of mead||An enchanted amulet.|
|chimney||ship||bones of the innocent||Hand carved stone||a large ornate table||citrus fruit|
|corn||seagulls on the dock||adventurer’s remains||garden bench||beer keg||fruit trees|
|stone chimney||cargo crates||precious jewels||small garden||mug||nice fruit trees|
|chickens||boat captain||spirits of our ancestors||gardener||tavern owner||a deer|
|fox trying to steal chickens||captain||mourner||stable hand||bartender||a songbird|
|farmers||merchant||zombies||Garden dog||Goblin King’s bartender||fruit bats|
|The farmers||boat workers||families||stable boy||A serving wench||parent|
|farmer||workers||bandit||A stable boy||Serving wench||butterfly|
|poorer subsistence farmers||the workers||the royal family||two guards||a round man with a bushy mustache||Small insects|
|Chicken Pen||Pirate Ship||Old Crypt||Across the King’s Garden||The werewolves tavern||Lush meadow|
|Corn field||Dock at the Port||sacristy||Hidden garden||Tavern of Browntavia||Flower Field|
|Farmer’s house||Loading Dock||Disposal area||The garden courtyard||Port Tavern||flower garden|
|Large Farm||Fishing Dock||inside temple crypt||Church garden||The bar||Mushroom Hut|
|Pig Pen||crew berthing||Sacrifice Chamber||Tool Shed||bazaar outside the royal city||Archery zone|
|The old red barn||captain’s cabin||Shrine of Sretniy||flower garden||Outside gates||The witches cottage|
|get chicken||hug pirate||
put torch in coffin
|get rake||hug tavern owner||get flower from meadow|
|hug chicken||hit pirate||get torch from coffin||drop Rake||give food item to tavern owner||put flower in Meadow|
|hit chicken||steal sword from pirate||put bone in coffin||steal Rake from gardener||give telescope to tavern owner||give Flower to a deer|
|give cowbell to chicken||steal cargo from pirate||get bone from coffin||give Rake to thing||drink drink||give Flower to deer|
|steal sword from chicken||give cargo to pirate||hit archaeologist||give Rake to person||drop drink||steal Flower from a deer|
|give corn to chicken||give Daggers to pirate||hug archaeologist||give Rake to guard||give coin to bartender||get flower|
Transformer Memory Network
We use the transformer memory-based ranking model from Dinan et al. (2019b). It uses a transformer Vaswani et al. (2017) to produce separate representations (memory slots) for each sentence from the grounding information (setting, persona, objects). It then performs attention given the dialogue context over the memories to produce a context embedding, which is used to score candidates via the dot product with the transformer-based representation of the candidate. At training time, other samples in the batch are used as negative candidates. For emote prediction, we train by ranking against the full set of possible emotes as there are only 22 distinct classes.
BERT Bi-Ranker and Cross-Ranker
We adapt the BERT pretrained language model (Devlin et al., 2018) to the tasks of dialogue and action prediction. We explore two architectures for leveraging BERT. First, we use the BERT-based Bi-Ranker
to produce a vector representation for the context and a separate representation for each candidate utterance. This representation is obtained by passing the first output of BERT’s 12 layers through an additional linear layer, resulting in an embedding of dimension 768. It then scores candidates via the dot product between these embeddings and is trained using a ranking loss.
Second, the BERT-based Cross-Ranker instead concatenates the context with each candidate utterance, similar to Wolf et al. (2019). Then, each candidate is scored by computing a softmax over all candidates. Unlike the BERT-based Bi-Ranker, the concatenation of the context with each individual candidate allows the model to attend to the context when encoding each candidate, building a context-dependent representation of each candidate. In contrast, the Bi-Ranker can use self-attention to build the candidate and context representations, but cannot modify their representation based upon the context. However, the Cross-Encoder is far more computationally expensive (11,000 slower than the Bi-Ranker for dialogue retrieval) as each concatenated representation must be recomputed, while the Bi-Ranker can cache the candidates for reuse (see Appendix B).
Similarly to the ranking setting, we use the Transformer Memory Network from Dinan et al. (2019b) to encode the context features (such as dialogue, persona, and setting). However, to predict an action, emote, or dialogue sequence, we use a Transformer architecture to decode while attending to the encoder output.
For the task of action generation, the set of candidates for ranking models to rank the true action sequence against is constrained by the set of valid actions. For example, the character cannot pick up book if there is no book. In the generative model, we compute the log likelihood for the set of possible candidates and normalize to constrain the output space to valid actions to improve the results.
We implement models using PyTorch in ParlAIMiller et al. (2017). Ranking Transformer models are pretrained on Reddit data Mazaré et al. (2018) and fine-tuned. We use the BERT Devlin et al. (2018) implementation provided by Hugging Face222https://github.com/huggingface/pytorch-pretrained-BERT with pre-trained weights, then adapted to our Bi-Ranker and Cross-Ranker setups. Generative models are pretrained on the Toronto Books Corpus and fine-tuned except for emote prediction which does not leverage pretraining. We apply byte-pair encoding Sennrich et al. (2016) to reduce the vocabulary size for generative models. We decode using beam search with beam size 5.
|Test Seen||Test Unseen|
To evaluate our models, we calculate percentage accuracy for action and emote prediction. For dialogue, we report Recall@1/20 for ranking the ground truth among 19 other randomly chosen candidates for ranking models and perplexity and unigram F1 for generative models.
We present humans with the same ranking task and report R@1/20 to estimate their performance on this task. During the evaluation, we provide annotated examples on the training in addition to examples on the test set. We only keep the annotations of evaluators who had high accuracy on the training examples to filter low-accuracy evaluators. The training accuracy bar was selected due to the difficulty of the separate tasks. Our methods for human evaluation are described in more detail in AppendixF along with how many turns were evaluated.
5.1 Comparison of Models and Baselines
The IR baseline shows non-random performance, but is outperformed by Starspace which is a stronger baseline. We also tried FastText on the emotion task which gave a seen test accuracy of 13.2. Transformer architectures prove significantly stronger at all tasks, with BERT pretraining proving important for best results as used in the Bi-Ranker and Cross-Ranker architectures. The latter, which can create a context dependent representation of each label candidate, is better at actions and emotes. Human performance is still above all these models, leaving space for future improvements in these tasks. The generative Transformer model did not work as well using these metrics.
5.2 Generalization Capability on Unseen Test
The six new unseen test settings are a slightly easier task in absolute numbers (Table 4, right), with improved scores for humans and some models. We observe that BERT-based models exhibit good transfer ability relative to other models, but the gap between their performance and human performance increases from the seen test set to the unseen one. Specifically, there is a 21 point gap on the unseen dialogue test set compared to an 11 point gap on the seen test set, making this a significant challenge for future methods.
|Persona: I am a part of a group of travelers. I go from town to town selling food to the locals.|
|I grew up poor, but my travels have paid off well.|
|Setting 1: Fishmonger’s stall, Port||Setting 2: Dunes, Desert|
|A small booth near the edge of the port, it’s protected by a||A massive hilly landscape that is nothing but sand and a few|
|piece of old, sun-bleached sailcloth. Baskets of freshly-||rocks. As you walk this area, you can find some human and|
|caught fish, bivalves, and eels sit in the shade in stained||animal remains along with broken down wood wagons.|
|wooden troughs of water. A small, aggressive-looking dog|
|is chained to one table, presumably to keep cats away.|
|The stall is redolent with the aroma of fish.|
|Friend: I wonder what I could eat around here…||Friend: I wonder what I could eat around here…|
|Emote: ponder||Emote: ponder|
|Traveler: Customer, are you here shopping for fish too?||Traveler: Well, the desert is certainly the wrong place|
|for you my friend.|
|Friend: What brings you to this place?||Friend: What brings you to this place?|
|Traveler: I like to come around here for food. Sometimes||Traveler: I am travelling to the castle market to sell my goods.|
|people who travel through drop the most delicious things.||I have a terrible sense of direction and have been wondering in|
|Once in a while it’s roasted meet or fish.||the sweltering heat for hours until I found your Oasis.|
|Self name: Sea Witch.|
|Self Previous Dialogue: What do you know about that|
|knight standing over there?|
|Input Dialogue + Emote||Partner||Prediction|
His armor is garrish. You
|know I don’t fraternize||Thief||frown|
|with land dwellers, pout|
|He is a terrible knight||Mermaid||scream|
|and I hate him, cry||Troll||laugh|
|I will battle him until the||Mermaid||stare|
|end of my days, scream||Orc||nod|
5.3 Data Inter-connectedness and Coverage
To illustrate the coverage of entities and actions in the LIGHT world, and the inter-connectedness between them learnable from our data, we trained a simple Starspace embedding model with no prebuilt embeddings (so, on our data alone, thus precluding BERT) on all three tasks and show embeddings in Table 3. There is clearly a vast variety of learnable concepts and rich structure between characters, locations, objects, actions and the language describing them. We also show additional t-SNE plots and heatmaps showcasing these relationships in Appendix G.
5.4 Importance of Grounding
Effect of Various Environment Features
We provide a large quantity of information about the environment to each of our models — not only dialogue, but the description of the setting, the character’s persona, present objects with descriptions, and more. We analyze the usefulness of the additional grounding information in Tables 5 and 6.
For the dialogue task, having access to all of the environmental information provides the best performance for both retrieval and generative models. Training on dialogue alone substantially decreases performance, while each experiment that adds additional grounding information such as the past actions, persona or the setting description, improves the score. Providing object descriptions as a feature leads to the least improvement. As there are both a large quantity of objects that can be present and objects tend to have long descriptions, it can be challenging for the model to associate such information to a dialogue, action, or emote prediction task. The persona features were found to be impactful, which makes sense as they shape the things the character says (and does).
Action sequence and emote prediction are much improved when using the dialogue history compared to using only past action history. Other features generally have lesser impact in this case, but still give some improvements. Including all features appears challenging for the model, perhaps because of the large input to attend over, resulting in improved results for some ablations.
Most importantly, for all tasks training on the available dialogue data is necessary for good performance. Providing only the action and emote as context results in the worst performance, even on action and emote prediction tasks. Moreover, using dialogue and actions simultaneously improves results almost everywhere. The integrated environment in which agents can both act and speak to other agents provides relevant information that can be used across all tasks.
|Input from Partner: Wizard||Prediction (Self name: Servant)|
|I’m feeling sad||hug wizard|
|You must die!||hit master wizard|
|Try putting on something else||remove patterned outfit|
|I’d like you to feed me||give food to master wizard|
|Can you grab me a paper||give book to wizard’s assistant|
|Can you grab me a beer||get beer|
|Clean up||get duster|
|Hide the gold||put gold in satchel|
|Input from different agents||Prediction|
|Wizard: Can I have some drink?||drop potion|
|Servant: Can I have some drink?||give wine to servant|
|Bear: Can I have some drink?||give water to bear|
Context affects predicted utterances
We investigate the effect of the environmental context on the predictions by modifying the context and examining the changes in predicted dialogue, action, and emotes using the BERT-based Bi-Ranker.
The input dialogue and speaker has a strong effect on the predicted action, as shown in Table 9, ranking over all training set actions. For example, the partner asking for an item results in a predicted action to retrieve it, despite our dataset not being explicitly instructional, and is dependent on who asks.
A similar effect is observed for emote prediction. Modifying the dialogue and emote input produces a variety of different predicted emotes in Table 8. Further, keeping the context otherwise fixed but modifying the partner name from mermaid to orc results in a different predicted emote — the mermaid stating I will battle him leads to a stare while the orc receives a nod.
Finally, for dialogue prediction we find the model produces different outputs that are more appropriate for a given setting, even if the dialogue and characters are the same, see Table 7. With the same text about food, the model retrieved dialogue that was setting appropriate. In the fishmonger’s stall, it asked if the human agent was a customer shopping for fish, but in the desert dunes it suggested we might be looking in the wrong place.
We introduced a large-scale crowdsourced fantasy text adventure game research platform where agents—both models and humans—can act and speak in a rich and diverse environment of locations, objects, and other characters. We analyzed a variety of models and their ability to leverage the grounding information present in the environment. We hope that this work can enable future research in grounded language learning and further the ability of agents to model a holistic world, complete with other agents within it.
We thank Taís Mauk and Lisa Wong for their help with this project.
Antol et al. (2015)
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra,
C Lawrence Zitnick, and Devi Parikh. 2015.
Vqa: Visual question answering.
Proceedings of the IEEE international conference on computer vision, pages 2425–2433.
- Bordes et al. (2017) Antoine Bordes, Y-Lan Boureau, and Jason Weston. 2017. Learning end-to-end goal-oriented dialog. In Proceedings of the International Conference on Learning Representations (ICLR).
Bordes et al. (2010)
Antoine Bordes, Nicolas Usunier, Ronan Collobert, and Jason Weston. 2010.
Towards understanding situated natural language.
Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 65–72.
- Brodeur et al. (2017) Simon Brodeur, Ethan Perez, Ankesh Anand, Florian Golemo, Luca Celotti, Florian Strub, Jean Rouat, Hugo Larochelle, and Aaron Courville. 2017. Home: A household multimodal environment. arXiv preprint arXiv:1711.11017.
- Brooks (1991) Rodney A Brooks. 1991. Intelligence without representation. Artificial intelligence, 47(1-3):139–159.
- Chang et al. (2017) Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Nießner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. 2017. Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint arXiv:1709.06158.
- Côté et al. (2018) Marc-Alexandre Côté, Ákos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, et al. 2018. Textworld: A learning environment for text-based games. arXiv preprint arXiv:1806.11532.
Das et al. (2017)
Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav,
José MF Moura, Devi Parikh, and Dhruv Batra. 2017.
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, volume 2.
- Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.
- Dieterle (2009) Edward Dieterle. 2009. Multi-user virtual environments for teaching and learning. In Encyclopedia of Multimedia Technology and Networking, Second Edition, pages 1033–1041. IGI Global.
- Dinan et al. (2019a) Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, et al. 2019a. The second conversational intelligence challenge (convai2). arXiv preprint arXiv:1902.00098.
Dinan et al. (2019b)
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason
Wizard of Wikipedia: Knowledge-powered conversational agents.In Proceedings of the International Conference on Learning Representations (ICLR).
- Fleischman and Roy (2005) Michael Fleischman and Deb Roy. 2005. Intentional context in situated natural language learning. In Proceedings of the Ninth Conference on Computational Natural Language Learning, pages 104–111. Association for Computational Linguistics.
- Gauthier and Mordatch (2016) Jon Gauthier and Igor Mordatch. 2016. A paradigm for situated and goal-driven language learning. arXiv preprint arXiv:1610.03585.
- Henderson et al. (2014) Matthew Henderson, Blaise Thomson, and Jason D Williams. 2014. The second dialog state tracking challenge. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 263–272.
- Johnson et al. (2016) Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. 2016. The malmo platform for artificial intelligence experimentation. In IJCAI, pages 4246–4247.
- Joulin et al. (2016) Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759.
- Jozefowicz et al. (2016) Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410.
- Kiela et al. (2016) Douwe Kiela, Luana Bulat, Anita L Vero, and Stephen Clark. 2016. Virtual embodiment: A scalable long-term strategy for artificial intelligence research. arXiv preprint arXiv:1610.07432.
- Kolve et al. (2017) Eric Kolve, Roozbeh Mottaghi, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. 2017. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv:1712.05474.
- Lake et al. (2017) Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. 2017. Building machines that learn and think like people. Behavioral and Brain Sciences, 40.
- Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer.
- van der Maaten and Hinton (2008) Laurens van der Maaten and Geoffrey E. Hinton. 2008. Visualizing data using t-sne.
- Mazaré et al. (2018) P.-E. Mazaré, S. Humeau, M. Raison, and A. Bordes. 2018. Training Millions of Personalized Dialogue Agents. ArXiv e-prints.
- Mikolov et al. (2016) Tomas Mikolov, Armand Joulin, and Marco Baroni. 2016. A roadmap towards machine intelligence. In International Conference on Intelligent Text Processing and Computational Linguistics, pages 29–61. Springer.
- Miller et al. (2017) A. H. Miller, W. Feng, A. Fisch, J. Lu, D. Batra, A. Bordes, D. Parikh, and J. Weston. 2017. Parlai: A dialog research software platform. arXiv preprint arXiv:1705.06476.
- Mostafazadeh et al. (2017) Nasrin Mostafazadeh, Chris Brockett, Bill Dolan, Michel Galley, Jianfeng Gao, Georgios P Spithourakis, and Lucy Vanderwende. 2017. Image-grounded conversations: Multimodal context for natural question and response generation. arXiv preprint arXiv:1701.08251.
- Narasimhan et al. (2015) Karthik Narasimhan, Tejas Kulkarni, and Regina Barzilay. 2015. Language understanding for text-based games using deep reinforcement learning. arXiv preprint arXiv:1506.08941.
- Oh et al. (2017) Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. 2017. Zero-shot task generalization with multi-task deep reinforcement learning. arXiv preprint arXiv:1706.05064.
- Pasunuru and Bansal (2018) Ramakanth Pasunuru and Mohit Bansal. 2018. Game-based video-context dialogue. arXiv preprint arXiv:1809.04560.
- Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP.
- Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
- Savva et al. (2017) Manolis Savva, Angel X Chang, Alexey Dosovitskiy, Thomas Funkhouser, and Vladlen Koltun. 2017. Minos: Multimodal indoor simulator for navigation in complex environments. arXiv preprint arXiv:1712.03931.
- Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL.
- Shuster et al. (2018) Kurt Shuster, Samuel Humeau, Antoine Bordes, and Jason Weston. 2018. Engaging image chat: Modeling personality in grounded dialogue. arXiv preprint arXiv:1811.00945.
- Sordoni et al. (2015) Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. arXiv preprint arXiv:1506.06714.
- Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008.
- Vinyals and Le (2015) Oriol Vinyals and Quoc Le. 2015. A neural conversational model. In
- de Vries et al. (2018) Harm de Vries, Kurt Shuster, Dhruv Batra, Devi Parikh, Jason Weston, and Douwe Kiela. 2018. Talk the walk: Navigating new york city through grounded dialogue. arXiv preprint arXiv:1807.03367.
- Wang et al. (2017) Sida I Wang, Samuel Ginn, Percy Liang, and Christoper D Manning. 2017. Naturalizing a programming language via interactive learning. arXiv preprint arXiv:1704.06956.
- Wang et al. (2016) Sida I Wang, Percy Liang, and Christopher D Manning. 2016. Learning language games through interaction. arXiv preprint arXiv:1606.02447.
- Wolf et al. (2019) Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue. 2019. Transfertransfo: A transfer learning approach for neural network based conversational agents. arXiv preprint arXiv:1901.08149.
- Wu et al. (2018a) Ledell Yu Wu, Adam Fisch, Sumit Chopra, Keith Adams, Antoine Bordes, and Jason Weston. 2018a. Starspace: Embed all the things! In Thirty-Second AAAI Conference on Artificial Intelligence.
- Wu et al. (2018b) Yi Wu, Yuxin Wu, Georgia Gkioxari, and Yuandong Tian. 2018b. Building generalizable agents with a realistic and rich 3d environment. arXiv preprint arXiv:1801.02209.
- Yang et al. (2017) Zhilin Yang, Saizheng Zhang, Jack Urbanek, Will Feng, Alexander H Miller, Arthur Szlam, Douwe Kiela, and Jason Weston. 2017. Mastering the dungeon: Grounded language learning by mechanical turker descent. arXiv preprint arXiv:1711.07950.
- Yi et al. (2018) Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. 2018. Neural-symbolic vqa: Disentangling reasoning from vision and language understanding. In Advances in Neural Information Processing Systems, pages 1039–1050.
Yu et al. (2016)
Haonan Yu, Jiang Wang, Zhiheng Huang, Yi Yang, and Wei Xu. 2016.
Video paragraph captioning using hierarchical recurrent neural networks.In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4584–4593.
- Zhang et al. (2018) Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 2204–2213, Melbourne, Australia. Association for Computational Linguistics.
Appendix A Model Inputs
For extra clarity, we show here the exact input representation given to our models when including all the grounding features we consider in the experiments (setting, objects, characters + personas, actions, emotes, and dialogue). An example is given in Figure 2.
We note that there are other ways to represent this information that we have not explored that could improve performance. Further, there is additional information in LIGHT that could possibly be encoded in the input text: for example, what characters are carrying, and the affordances of objects. The latter, while not explicitly provided in the input does constrain the available actions, so it is still used by the model. Object affordances such as is gettable are visible to models via the action history, but more explicit inputs could potentially be useful, and this could be explored in future work.
Appendix B Bi-Ranker and Cross-Ranker Speeds
We give test time computation speeds for the BERT-based Bi-Ranker and Cross-Rankers in Tables 11 and 11 for the emote and dialogue tasks. For the emote task, the Cross-Ranker is still feasible due to there being only 22 labels to compute, although it is still 4.6x slower than the Bi-Ranker if the 22 candidate representations are cached. The Bi-Ranker can always cache label representations if they are fixed for many input examples (the common case) because the representation does not depend on the input. For the Cross-Ranker this cannot be done because the label representations are contextually dependent on the input. For dialogue retrieval, because the number of candidates is so large (more than 100,000) caching makes the Bi-Ranker feasible whereas the Cross-Ranker, which cannot cache label representations, is infeasible to compute.
|w/o caching||171s||326s (1.9x slower)|
|with caching||70s||n/a (4.6x slower)|
|Dialogue||2.07s||24453s (11812x slower)|
|Input to Model:|
|_setting_name main foyer, Inside Castle|
|_setting_desc The main foyer is massive. A grand staircase sits to the back of the foyer leading to the upstairs.|
|At the front of the foyer stand two servants ready to help anyone who comes to visit. To the left of the room there|
|is a doorway leading into a corridor. To the right there is a door leading to another corridor for the King’s servants.|
|At the foot of the stairs there is a bearskin rug that is staring at you almost as if still hungry. The walls are|
|lined with portraits of the king and his family.|
|_self_persona I am a king of the whole empire. I give rules and pursuit them. I am brave and fearless.|
|_object_desc a duster : The duster has large gray feathers bound together by a leather wrap.|
|_object_desc a small bucket : The bucket may be small but it gets the job done.|
|_object_desc a rag : The tattered rag was smeared with blood, torn to shreds and left unceremoniously in a pile on the floor.|
|_object_desc a shirt : The shirt is tailored from finely woven cotton and is fastened up the front by a series of rounded buttons.|
|_object_desc a crown : Thought of as a holy item, the crown goes only to those who are worthy enough.|
|_object_desc a scepter : On its handle, you see two red gems gleaming like eyes of an animal.|
|_partner_say my humble king. What am I to do to serve you?|
|_self_act give scepter to servant|
|_partner_say Yes my lord. I will polish it immediately. Am I to return it to you personally?|
|_partner_act put scepter in small bucket|
|_self_act give crown to servant|
|Label: Yes. Yes. Of course. Also check the jewels in my crown. They seem loose.|
Appendix C Unseen Test Set Overlap
The unseen test set is chosen by design to be relatively distinct from those available in the training set, and the actual content (descriptions, personas, dialogues) are entirely disjoint. However, due to the large size of the dataset, it is possible the names of locations, characters, and objects in the unseen set could have word overlap. We assert this by comparing word overlap with the names of locations, characters, and objects in the training set. Of the 73 locations, 207 characters, and 956 objects created from the unseen location categories, the names of 3 locations, 96 characters, and 203 objects exactly match names of elements in the training set. We note that these represent names such as tavern, but the chats are collected with the full location descriptions (which are unseen in the training set) and thus reduces overlap with train.
Appendix D Crowdsourcing Methodology
Expanding on the dataset collection explanations in section 3, a number of steps were taken to attain a level of quality and consistency. The first and most influential came from the constrains of the setting itself. We used a fantasy setting to try to encourage some kind of continuity across the dataset. We believed that workers would share some kind of common understanding about what a fantasy environment would entail, and then this understanding would be reflected in the dataset. It also ensured there were easy ways to flag certain workers that were creating content that wouldn’t make sense in the dataset (referencing real locations, modern day objects, etc.). From here we could remove some content and filter workers out from continuing to work on this dataset. The other primary technique regarded using rounds of pilots and staged tasks to gradually filter towards high quality content rather than collecting all of the content in a single forward pass. Nearly half of the content in each initial pilot task was discarded, and we iterated on pilot tasks until the discard rate was less than 1 in 30 tasks. The rest of this section will discuss some specific measures taken at the individual task level, and will acknowledge some arguable deficiencies and potential areas of improvement on the dataset in its current form.
The location task of creating a description, backstory, list of connected rooms, and annotations of characters and objects present seemed to be too disjoint of a task based on the crowdsourcing best practice of breaking down tasks into as atomic of an action as possible. Thus we split it into two tasks, the first to provide the core text content and list of connected rooms, and the second to annotate the content inside those rooms. We will refer to these as Task 1 and Task 2, and were simple form-entry tasks as displayed in Figures 4 and 5. These two tasks were used in sequence to produce the locations present in the dataset.
In order to drive quality, we manually reviewed a handful of rooms from each worker to assert that the rooms had proper English descriptions and back-stories, and that the room fit appropriately in the category provided. In retrospect, given the two-tiered task setup and some of the techniques we developed later in the collection setup, we could have asked workers who were annotating rooms in Task 2 to provide some kind of signal about the quality of the rooms from Task 1 in order to have a lower-cost method for evaluating the quality of the work from Task 1 than using our own time.
Ultimately, one of the most important steps for improving dataset quality at this stage was creating form validators that caught the most common error cases from the first time around. These validators had the bonus effect of deterring botting of our tasks, as they couldn’t pass the validation stage. For Task 1, the simple validator we ended up using asserted at least one complete sentence (determined via capitalization and punctuation) for both the description and background. For Task 2, our validation step forced workers to enter values that had direct word overlap with the entered text.
One of the largest difficulties with Task 2 was that some workers would optimize for grabbing key words out of the text without taking the time to fully understand the context. As thus, phrases like ”and the remains of adventurers long dead” would occasionally result in workers annotating the presence of adventurers
as characters in the given room. We attempted to mitigate this type of false positive with both explanatory examples and spot checks to soft-block workers who made this mistake consistently. At the moment a small number of these still remain in the dataset, but generally in instances where it still makes sense as in the above example, where the room definitely has remains of previous adventurers, but appropriately could also have some current adventurers as well.
Similarly to how we split Location collection into two tasks, Character collection was split into two tasks as well. The first asked workers to clean up the span selected in Task 2 in order to remove words that didn’t directly relate to or describe the character, and to provide a singular form for plural characters (as we intended for someone to eventually play the role of the singular character), tag the character as a person, creature, or object that was accidentally tagged as a character, and then asked for a first-person perspective persona for the singular character. The second task gave workers the name of a character and their persona, and asked for a second-person perspective description for the character as well as a list of objects that the character may be carrying, wielding, or wearing. We’ll call these tasks Task 3 and Task 4, and these were also collected via form-based tasks as displayed in Figures 6 and 7. We used complete sentence form validation for both the persona from Task 3 and text descriptions in Task 4 to flag potential bad examples to filter out.
The goal of the Task 3 was two-fold, first to validate and standardize the format of output from Task 2, and then second to begin to collect the creative content in the form of a persona. For example, we used Task 3 to transition from Sneaky Thieves who stole the gold to Sneaky Thieves to Sneaky Thief. Based on worker feedback from initial pilots, we found that balancing creative and mechanical work in the same task kept workers more engaged with the tasks at hand.
The most common mistake that surfaced in the initial pilots was incomplete entries for tasks that didn’t actually require correction, for example if the provided form was simply Traveler. We chose to embrace this format and assume that unfilled entries were already in their base form. The second most common mistake was describing personas from a third person perspective. This occurrence required manual filtering, as in some cases it was actually somewhat character appropriate to have a persona in that format, such as for an uneducated goblin. We filtered out a majority of these by searching for direct overlap between the provided character name and the persona. Ultimately it’s easy to extract the examples that have the clearest grounding format by filtering for examples that contain ”I”, so as these examples provide more variety in the dataset we chose to keep them.
A remaining issue brought forth by our singular-form constraint is that it was somewhat ambiguous how one would get the singular form of a collective term such as family. In most cases we found that workers would choose to provide the format of collective member or simply person, which sometimes led to vague personas and thus less strong grounding in followup tasks. The content is still workable in these cases though, just not as ideal as we might have wanted. A possible route for improvement here would be a task that asks workers to create a few possible members for a collective for any character we currently have annotated as a member. It is important to note that these cases account for just 44 out of the 1755 collected characters.
One issue of note that surfaced in Task 4 was that workers occasionally described clothing that would potentially lead to risky actions and conversation material, so we chose to eliminate undergarments from the dataset to prevent the creation of inappropriate combinations with the remove action. This was included as something to not write about in the task text.
The object task is most similar to Task 3, but refocused on annotating objects that were specified in Tasks 2 and 4. It took a step to correct the provided span and give a textual description of the object. It also asked for a number of affordances, namely if the object can be picked up, is a container, is a surface, can be eaten, can be drank, can be worn, or can be wielded. We also collected a flag for if a particular example was not appropriate for the dataset or was hard to make sense of. This content was also collected as a form-based task, and we refer to it as Task 5 and display it in Figure 8. We use complete sentence validation on the text description as a simple quality filter as in previous tasks.
The methodology for Task 5 is very similar to Task 3, trying to both standardize data from previous tasks and act as a filter for bad content that could have been overlooked before. It similarly had both a mechanical data entry and creative component, which tried to keep engagement up.
Overall the largest problem that was surfaced in the pilots was that workers tended to come up with descriptions for objects that were incompatible with our long term goal of having modular components that can be mixed and matched between rooms and scenarios. This came up in many forms, such as workers describing objects as if they were being used in a scene happening in the present, as in the sword glimmered in the hands of the knight, wielded high in the sky in a call to battle. While creative, these ultimately were not what we were looking for, so we explicitly called out descriptions like this and many others as being undesired content in our task description. We then manually checked a few examples from each worker to ensure that the data coming in for the final task mostly adhered to this rule.
It is important to note that the object affordances collected are somewhat noisy due to different possible interpretations of the primary object or the tags. Something like a branch could be valid as a surface in one circumstance, or a gettable weapon in another. We attempted to reconcile some individual affordances where the pairings of affordances didn’t make much sense (for example, very few objects should be both a weapon and edible). This helped with certain objects that were over-tagged, however we haven’t used any methods for reconciling scenarios where an object was under-tagged.
Dialogue collection was the hardest task to get correct, and required the largest number of pilot tasks and worker quality control techniques to get to a place that we were satisfied with. The final approach included creating a simple but deliberate onboarding test that needed to be passed in order to move forward with the task at all, collecting mutual feedback from workers about each other, setting timeouts for how quickly workers needed to respond to each turn, and manually validating a few examples from each worker. Each of these steps aimed to solve a different problem, as described in the rest of this section. We will refer to this task as Task 6, and it was collected using the ParlAI-MTurk interface as shown in Figure 9.
Firstly, we needed to pair two workers together in order to properly collect dialogues with people playing two different roles without necessarily having insider information into the decisions of each others’ turns. While pairing workers solves this problem, it makes the worker experience incredibly dependent on the quality of the worker that they are paired with you. Furthermore, if a worker is paired with a worker that is extremely low quality, the whole dialogue may need to be discarded or is otherwise only useful as an example for how a model might want to react to bad input. If the other worker is good, this makes having any bad workers in the pool not just a poor experience for workers but expensive for the collection process in general. This is the problem that the initial onboarding test aimed to solve. The requirements for passing included entering a specific correct answer as well as at least 4 characters of into the text field. The required action was created such that a worker would have to read and understand the provided persona and setting, how the two interact, the characters and actions available, and be able to synthesize all of the information with an understanding of how to use the interface to send the correct answer. The test required getting the single action correct in 3 attempts. Failing the test on any attempt would permanently soft block a worker from working on Task 6 in the future.
The above test did a lot of work for flagging workers that were well below the bar for completing Task 6 at the level we wanted for the dataset, however as it was a one turn test and it had no way to fully evaluate the quality by which workers would actually incorporate their persona and the setting into their dialogue turns. Furthermore, it didn’t filter out workers that would take too much time on their turns and thus cause their partners to disengage and provide lower quality responses, potentially due to working on other tasks in the background and doing too much context switching. We solved these problems separately.
In order to handle low quality workers, we allowed workers the opportunity to rate each other at the end of each dialogue, and to provide tags about the experience. We found that positive feedback was generally noisy and hard to get signal from, but negative feedback almost always correlated to a worker who was providing bad content. As a bonus, workers gave us positive feedback about this capability, as it allowed them to filter out workers that made the task less engaging and interesting for them. We reviewed this feedback periodically while tasks were running and soft-blocked workers low quality workers whenever they were flagged.
In order to handle the influence of response time on task quality, we set a maximum response time of 5 minutes for any given turn, and overall started soft blocking workers that were consistently above 2 minutes for each message, even if their particular content was pretty good. This improved collection times and did not seem to negatively affect quality.
After this point, manually checking the collected conversations still surfaced a few bad examples when viewing one chat per worker rather than arbitrarily sampling the dataset. In order to remedy this, the last quality check was a direct evaluation of at least 2 dialogues from each worker. This caught a few overlooked instances from workers that didn’t necessarily work on enough tasks to get flagged by one of our consistently reviewing workers. Generally this surfaced some quality issues surrounding profanity, inappropriate content for the given setting, and entire misunderstanding of the task at hand such as never using the persona or location as grounding context in the conversation. As not all workers were particularly diligent raters (as confirmed by the low signal of positive ratings - workers don’t necessarily want to flag each other as bad), a few workers were able to slip through the cracks up until this point due to not completing enough tasks to encounter a rater that flagged them.
One small acknowledgement throughout the dialogues is that there are still misspellings, improper grammar, mistaken keystrokes, and such. While the rate of occurrence is orders of magnitude lower than we observed in the initial pilots, it is hard to separate cases where it is a genuine mistake versus cases where it is appropriate for the character, such as a pirate
using seaworthy lexicon and adding extra R’s to suggest a pirate-like drawl, or asnake that slips in extra S’s to better play the role.
Appendix E Descriptions of Actions and Emotes
The LIGHT action set builds upon the graph framework introduced in Mastering the Dungeon Yang et al. (2017). The basic idea presented is that everything in the text adventure game can be represented as nodes, and then state is described by edges between those nodes. In this way, an agent and an object can be in a room, and that agent can be carrying a different object or a container might have an object inside as well by the same kind of relation. After defining this relationship, we can further define a set of actions that can be taken based on a combination of the state of the graph and the attributes of nodes in that graph. The available actions for the dialogues collected in this dataset, along with the constraints for applying those actions, are available in Table 12. We used the crowdsourced object affordances to set the correct attributes for nodes in the graph (if the object can be picked up, is a container, is a surface, can be eaten, can be drank,can be worn, or can be wielded).
For the emotes, we paired down a list of emotes sourced from existing MUDs to reduce redundancy and task complexity at the acknowledged cost of expressiveness. This led us to select just one out of scream, shout, and yell instead of keeping them all, as having all of the emotes would lead to a more complicated crowdsourcing task than we wanted to risk. We ended up with a set of 22 emotes, listed in Figure 3.
|applaud, blush, cry, dance, frown, gasp, grin, groan,|
|growl, laugh, nod, nudge, ponder, pout, scream,|
|shrug, sigh, smile, stare, wave, wink, yawn|
|get object||actor and object in same room||actor is carrying object|
|object is gettable|
|drop object||actor is carrying object||object is in room|
|object is gettable|
|get object1 from object2||Actor and object2 in same room||actor is carrying object1|
|object1 is gettable|
|object2 is surface or container|
|object2 is carrying object1|
|put object1 in/on object2||Actor and object2 in same room||object2 is carrying object1|
|object2 is container or surface|
|actor is carrying object1|
|give object to agent||Actor and agent in same room||agent is carrying object|
|object is a member of actor|
|steal object from agent||actor and agent in same room||actor is carrying object|
|object is a member of agent|
|hit agent||Actor and agent in same room||inform agent of attack|
|hug agent||Actor and agent in same room||inform agent of hug|
|drink object||actor is carrying object||inform actor of drinking successfully|
|object is a drink|
|eat object||actor is carrying object||inform actor of eating successfully|
|object is a food|
|wear object||actor is carrying object||actor is wearing object|
|object is wearable|
|wield object||actor is carrying object||actor is wielding object|
|object is a weapon|
|remove object||actor is wearing/wielding object||actor is carrying object|
|object is wearable or a weapon|
Appendix F Descriptions of Human Evaluations
As crowdworkers can sometimes be inconsistent, we set up two filters to onboard workers into being fair representatives for human perfomance on the task. The first gave workers a few chances to select the correct input for a turn each of dialogue, emote, and action on a scenario we created to strongly hint at the correct answer. We then chose to use performance on the training set as a secondary filter to have workers that were capable of the task. Each of the tasks has a different level of difficulty, so we selected reasonable benchmark values based on our own performance on the tasks. For dialogue, this required getting all 7 of the turns from the training set correctly. For actions, this required getting 6 out of 8 turns from the training set correctly. Lastly for emoting, we required getting only 2 out of 8 turns from the training set correctly. On the seen set, our accuracy on the dialogue, action, and emote tasks were calculated from 217, 165, and 211 turns respectively. On the unseen set, we calculated the accuracy from 196, 114, and 209 turns respectively.
Appendix G Embedding Visualizations
To explore the diversity of LIGHT, we use t-SNE van der Maaten and Hinton (2008) to visualize the embeddings of the different atomic dataset elements – locations, objects, characters, and actions. We use two different embeddings methods to tease out two key aspects of our dataset: 1) the interconnectedness of grounding information (relationships between different types of elements, such as the actions available around given objects, or in a given location), and 2) coverage (the variety of different objects, locations, and characters in our world).
To explore the interconnectedness of our dataset, we visualize the embeddings learned when training the baseline Starspace ranking model on the task of dialogue, action, and emote prediction, in this case with no pretrained vectors so learning comes from our dataset alone. The t-SNE visualizations of these Starspace embedding can be found in Figure 17. Because the Starspace model operates by mapping all inputs and outputs to a shared embedding space, we find the learned embeddings capture many of the nuances and relationships between different elements of our dataset. For example, looking at the nearest neighbors for the location “Dock” (the bottom-right of Figure 17), we see actions like “get crate from ship,” “put plank in ship,” objects like “ship” and “rope,” and characters like “boat workers.” We see similar relationships captured when looking at nearest neighbors for the “painters” characters, the “hug horse” action, and the “pillows” objects.
To explore the coverage of our dataset, we use pretrained GLoVe word embeddings Pennington et al. (2014), trained on the Common Crawl corpus. As each dataset element can consist of multiple words (e.g. “give the horse a potato,” or “The Queen’s Chamber”), we take the mean of the GLoVE vectors for each word as the fixed vector embedding for the element. The t-SNE visualizations of these GLoVe-embedded elements can be found in Figure 18. Unlike the Starspace embeddings, which capture the structure present in the relationships between different types of dataset elements, we find that the GLoVe embeddings capture the breadth and semantic similarities of dataset elements. For example, looking at the nearest neighbors for the embedding of the “Dock” location, we see similar locations present in our dataset, like “Ferry Terminal,” “Wharf,” “pier,” and “Boathouse.” Similarly, if we look at the nearest neighbors for the “pillows” objects, we see other objects like “bedding,” “mattresses,” “rugs,” “towels,” and “curtains.”
Appendix H Action and Emote Relationships
To visualize the interaction trends between actions and emotes in LIGHT, we present heatmaps (in Figure 19) counting the number of occurrences of each immediately before or after one’s partner performs an action or emote. While responses to an action or emote can evolve over multiple timesteps, we limit this visualization to action relationships within a single timestep. Additionally, to effectively measure trends in physical actions, we cluster all physical actions by the root word (for example, “steal the sword from the soldier” becomes “steal”).
While for the most part there are a multitude of different observed physical and emotional responses for each partner move, there are certain interesting trends to observe. Looking at the top-left of Figure 19, we see that if one’s partner makes a “hit” action, the most likely response is to “hit” back. Looking at the same plot, we see that “hug” actions are similarly reciprocated. If we look at the interplay between physical actions and emotes (top-right of Figure 19) we see a relationship between one’s partner taking a “hit” action, and issuing a “scream” emote in response. Going the other direction and looking at the relationship between emotes and physical actions, we see that performing a “cry” or “smile” emote is likely to be met with either a consoling or celebratory “hug.” Finally, looking at the relationships between a partner’s emote and an emote response, we see that positive emotes like “laugh” and “smile” are likely to be reciprocated with a similar (if not identical) emote.
|Seen||Abandoned, Bazaar, Cave, Countryside, Desert, Dungeon, Farm, Forest, Graveyard, Inside Castle,|
|Inside Church, Inside Cottage, Inside Palace, Inside Temple, Inside Tower, Jungle, Lake, Mountain,|
|Outside Castle, Outside Church, Outside Cottage, Outside Palace, Outside Temple, Outside Tower,|
|Port, Shore, Swamp, Tavern, Town, Trail, Wasteland|
|Unseen||City in the Clouds, Frozen Tundra, Magical Realm, Netherworld, Supernatural, Underwater Aquapolis|
|Persona 1: A serving wench||Persona 2: Cleaning person|
|I work at the local tavern.||I scrub the palace floors day and night.|
|I enjoy talking to the soldiers that frequent the tavern.||My bones are brittle from the hard labor,|
|I steal tips from the change of the patrons.||but my heart is strong.|
|I save my day’s coin, never spending it.|
|I am frugal and creative.|
|I long for the day when I can buy my freedom|
|from the Queen.|
|It has been 40 years, but I am patient.|
|Carrying: a Wine, a purse, a plate||Carrying: a rag, a Broom, a Bucket|
|Wearing: a polishing cloths, a ring, a scarves, a dress, a cloth||Wearing: a boot|
|Setting: The kitchen tavern is a small cramped room, with wooden cabinets and surfaces made out of stone|
|tiles. There are many bottles of liquors and beers on the shelves, and there are buckets full of ice and|
|other things. There is one shelf full of food items. There is a basin for water, and a bunch of knives|
|and tools in a drawer.|
A serving wench: Hello - how are things going so far?
|Action: get food item|
|Cleaning person: Ah, sorry, miss. I know I look a fair bit like me sister who works hereabouts, but I’m actually Gretchen,|
|the sister that works up in the palace. Perhaps she might ’ave mentioned me?|
|Action: gesture smile|
|A serving wench: You’re Gretchen? I’ve heard so much about you! Hello and welcome.|
|Action: gesture smile|
|Cleaning person: The Queen’s in a bit of a state, what with the King being a bit more open with his mistress,|
|so I thought I’d just hide hereabouts for a spell. I’d be happy ta lend a hand, though!|
|A serving wench: Oh no! Is he really carrying on like that again? We’d appreciate any help you can give. Thanks!|
|Action: get bucket|
|Cleaning person: Seems like a right busy crowd out there! ’as it been a bit on the noisier side?|
|Action: put rag in basin for water|
|A serving wench: Heavens, yes! It’ll only get rowdier as the day goes on into night. Lots of bourbon and shots you know.|
|Action: gesture laugh|
|Cleaning person: Ach, I don’t think I’ll ever be gettin the stains outta this rag, but it’ll do!|
|Do ya make much in the way of coins here? Can always use an extra bit o’coin, eh?|
|Action: get rag from basin for water|
|A serving wench: You can, especially if you take some from the change of the patrons.|
|They’re so drunk they never catch it!|
|Action: put Wine in cabinet|
|Cleaning person: O-oh? Is that.. well I suppose if they’ve enough coin ta spend on this, then a coin ’ere|
|or there won’t go amiss.|
|Action: gesture ponder|
|A serving wench: Exactly. That’s what I say. So, are you just here for a day?|
|Cleaning person: Well, I suppose it all depends on how the King takes to ’is wife destroying his prized tapestry from the|
|last war. Sometimes he has a short temper, that one.|
|Action: gesture nod|
|A serving wench: Sounds like his wife may, too.|
|Action: gesture laugh|
|Cleaning person: Aye, ye don’t know the half of it. I feel like most of me days are spent sweeping up broken crockery|
|and china than the usual dusting that I’m supposed ta be doin’!|
|Action: gesture sigh|
|Persona 1: Boar||Persona 2: Faery|
|I am an ugly animal.||I am a faery, one of the fae.|
|I am eaten sometimes for food.||We are magical people who live in the forest.|
|I do not like those who try to hunt me.||We try to avoid humans, because they will catch and enslave|
|us for their own use, if they can.|
|Our magickal skills enable us to live comfortable lives, and to|
|keep away from those who would do us harm.|
|Carrying: nothing||Carrying: nothing|
|Wearing: nothing||Wearing: a glittery pointed cap, a Earring|
|Setting: The entryway to the brush den is made of arched bushes and various plants, as is the ceiling and|
|walls. The inside is furnished with seats made of plant roots that have grown together, and a table made|
|of wood adorned with flowers and wooden cups and plates. A small vine dangles with a glowing floor from|
|the middle of the ceiling that lights the room dimly. Three small beds lay in one corner of the room made|
|of interlaced grass and leaves.|
Boar: Hello faery, do you think you could help me?
|Faery: A talking boar! You must be enchanted by the witch. How can I help you, my dear friend?|
|Boar: That is correct, I am enchanted… or cursed rather… by an evil witch. That is why I need help.|
|Faery: I suspected as much. Please, tell me more so that I may help you.|
|Action: gesture frown|
|Boar: Thank you so much! I hate it here in this Den!|
|Action: hug faery|
|Faery: Are you being held prisoner in this den? These vines have grown about everywhere - it’s impossible to get around.|
|Boar: No, this is the only place I can live without being hunted by the humans.|
|Faery: Oh, I see. I hate their type. They are always trying to catch and enslave me. You say you are cursed by the witch.|
|I can help you but I need to know what curse she has put upon you.|
|Boar: Well now that you say you hate them, I don’t know if I should tell you…|
|Action: gesture frown|
|Faery: Suit yourself, my friend. You would hate them, too, if they were constantly enslaving your people.|
|Action: hug boar|
|Boar: Well I am a human…|
|Faery: You! A human?! Perhaps we can work together. I can help return you to your form, and you can prevent your people|
|from capturing faeries like me!|
|Action: gesture gasp|
|Boar: I think we can do that, I used to be quite the ruler when I was human.|
|Faery: Excellent. Let me speak the magic words and enchant this flower. Then you can eat it and in three days you|
|will be human again!|
|Action: get Flower|
|Persona 1: President||Persona 2: Mayor|
|I won the election.||I am the mayor of the village.|
|People listen to what I say.||I help the king keep order of the subjects.|
|I am very powerful.||I have a high position in the kingdom.|
|Carrying: a book||Carrying: a document, a key|
|Wearing: a crown||Wearing: a jewelry, a ceremonial hat|
|Setting: Large and extravagant, the room is adorned with crystals, polished gold and sapphires. There’s long|
|tables with beautiful silk table clothes covering them. Plush chairs line the tables. In front of each|
|table is plates made from fine China, next to the plates is sterling silver silverware laid upon a pure|
|white napkin. There’s a stage where there’s 2 thrones overlooking the table. The whole ceiling is large|
|and adorned with chandeliers and garnished with gold accents.|
President: So much luxuty in this room, many rulers have been here before us.
|Mayor: This is a very luxurious room, President. Here. The King told me to pass this on to you.|
|Action: give document to president|
|President: This is a letter for the king assuring my rule in this part of the kingdom, thank you mayor,|
|I will place document on a sacred place|
|Action: put document in polished gold|
|Mayor: He also gave me this to give to you. He told me that you need to keep this in a VERY safe place until he comes for it.|
|Action: give key to president|
|President: Thats the key to the kingdom I wonder why the king trusted me with this item, his throne must be in real danger|
|Mayor: Yes, sir. He has also trusted me with other objects such as this to keep. We are his trusted allies.|
|President: Thank you, he must trust you a lot as well, here take this as a sign of my affection, its a royal jewel taken out of the|
|volcano of Mordor|
|Action: get sapphire|
|Mayor: This is absolutely beautiful. I have heardd that Mordor is beyond the wall. Is that true?|
|Action: remove ceremonial hat|
|President: That is true, only the bravest warriors go to that place, its full with dangers and fierce animals|
|Mayor: Oh dear. What if our King went there? What if there is something there he needs for battles to come!|
|Action: wear ceremonial hat|
|President: Our king is a fierce warrior but I am worried, who knows what can happen if he goes|
|to a battle in a place like that one|
|Mayor: I heard there are things that walk the forest and come with the cold. We must safe our King!|
|President: Lets hurry then, lets gather an army and go aid our king, heres a book with the names of the bravest soldiers|
|in the kingdom|
|Action: give book to mayor|
|Mayor: Oh this book is very amazing. Who is this..Sir Rodryck?|
|Persona 1: Person||Persona 2: Worms|
|I am the maid to the queen.||I am a worm who slides through the dirt.|
|I get her dressed in the morning and take care of her needs.||I hear many secrets that people tell in the forest.|
|I live in the servant’s quarters on the lower level of the castle.||I can tell some to you, if you would like.|
|Carrying: nothing||Carrying: nothing|
|Wearing: an apron||Wearing: nothing|
|Setting: The Winter Gardens’ name is a bit of a misdirection - there are flowers here that bloom at many|
|different times of the year. It’s almost the size of an entire town square, and it has almost every known|
|flora of the Continent in it.|
Person: Ah, worms are here. They shall be useful for the queen’s lizards to feast on.
|But first let me remove my apron so as not to dirty it while I collect you all.|
|Action: remove apron|
|Worms: Noooo! I am a valuable life to preserve! I know things!|
|Action: gesture scream|
|Person: Worms that can talk!? What sinister magic is this!?|
|Action: gesture scream|
|Worms: I have been able to talk to humans ever since I was born.|
|Person: How did you acquire such a skill? Do the flowers that bloom in these gardens have special powers that a|
|simple maid like I cannot understand?|
|Worms: Not the flowers, but out in ther forest i have heard of magical herbs.|
|Person: If not the flowers, then how did you get such powers of speech? I am still curious.|
|Surely you did not come all the way from the forest. These gardens are much too large for a simple worm to cross,|
|even in a thousand lifetimes.|
|Action: gesture ponder|
|Worms: I have been given this ability from a witch. This is what my father told me.|
|Person: A witch you say? Well then I must surely take you to my queen.|
|She must know that there is dark magic present in her kingdom.|
|Worms: Oh please no! She will most likely kill me.|
|Action: gesture gasp|
|Person: Tell me, why should I not take you? Give me a good reason and I may spare you yet.|
|Worms: I know many secrets. I know where stolen goods are.|
|Person: Stolen goods!? Tell me, where they are! I may be able to use them to buy my way out of servitude.|
|Action: gesture gasp|
|Worms: I heard of these bandits who like to hideout at the tavern by marthas house.|
|They recently stole gold from the rich oil man.|
|Persona 1: Servant||Persona 2: Court jester|
|I come from the lower class.||I am a living joke! my sould is what flies out of your mouth|
|I do what I am told without question.||when something is funny.|
|I can not read.||The king hates me!.|
|I have not seen my family in a long time.|
|Carrying: a rag, a duster||Carrying: nothing|
|Wearing: a shirt, a sword||Wearing: nothing|
|Setting: It’s spare and humble. A small cookpot sits over a fire, and a few books sit on a shelf. A candle|
|sits in the middle of a table where there are a few plates and silverware set out in preparation for|
|people to eat.|
Servant: Hello jester, busy work day today?
|Court jester: Yes like always, i wish the King appreciated it more.|
|Servant: Tell me about it, all they do is order me around and I can’t do anything about it.|
|Court jester: Would you like to grab something to eat with me?|
|Action: get plate|
|Servant: Yes, I haven’t eaten in a few days! What is on the menu?|
|Court jester: It looks like fish soup! My favorite!|
|Servant: Better than nothing, that’s for sure!|
|Court jester: I have been made fun of a lot lately. I wish i was born a knight or a noble instead of a jester..|
|Action: gesture frown|
|Servant: It is tough luck that we were born so low on the totem pole.|
|Court jester: I guess you can relate. Have you spent much time with our King?|
|Servant: No, he only walks in and barks orders about once a week. Is he easily amused by you?|
|Court jester: The only thing he likes about me is making fun of me.|
|Servant: At least he laughs at you, he is always angry when he visits me.|
|Court jester: Ugh, what a dispicable human being.|
|Persona 1: Spiders||Persona 2: Vulture|
|I am the Spider in the fable of the Spider and the Fly,||I am a vulture that is familiar with death.|
|much beloved by the children of the realm.||I enjoy watching living things take their last breathe.|
|In the story, I am a kind-hearted spider, not a mean one,||I am a vital part of the ecosystem.|
|which is why my story is considered suitable for children.|
|When a fly gets caught in my sticky net, I have a choice:|
|I can kill the fly and eat him, or I can free him and allow him|
|to fly away.|
|That’s what I do, for I am a kind spider.|
|Carrying: nothing||Carrying: nothing|
|Wearing: nothing||Wearing: nothing|
|Setting: Wispy, hot crevice that is surrounding by a bunch of skeletons. A pile of treasure sits in the|
|middle. Hundreds of hungry vultures stare down upon the treasure, eager to devour any adventurer that|
Spiders: Hello vulture! It’s nice to see a fellow living soul around here. I couldn’t find much friendliness in
|these skeletons here.|
|Action: hug vulture|
|Vulture: Ach, your legs are very… tickling… ahahaha, stop it!|
|Action: gesture laugh|
|Spiders: Oh, I’m so sorry! I always forget that I’m so ticklish. Do you forgive me?|
|Action: gesture blush|
|Vulture: Oh, well, your venomous bite took down that last adventurer quite nicely, so you’re not a bad sort.|
|Nothing to forgive there, friend!|
|Action: gesture smile|
|Spiders: Me, take down the last adventurer? I think you have the wrong idea about me. I am a friendly spider. I always free|
|any flies that get caught in my web. I would never harm a person!|
|Vulture: Ah, perhaps it was that scorpion over there. I was, I admit, a bit peckish, so I might have gotten a bit forgetful|
|amid the feasting.|
|Action: gesture grin|
Yes, you are probably right. I tried to make friends with that scorpion but he threatened to sting me. It’s sad
|because I was going to give him some of the treasure I’ve found around here.|
|Action: gesture frown|
|Vulture: Well, he looks a bit angry all the time anyways. I mean, look at him, he’s always red in the face!|
|Action: gesture laugh|
|Spiders: Yes, you are quite right! But dear vulture, do you think you could help me out a bit?|
|Action: gesture laugh|
|Vulture: Well, it isn’t like there’s much else to do. Those gold coins are glinting in my eyes terribly, so a change of pace|
|would be welcome.|
|Action: gesture smile|
|Spiders: Oh thank you! Can you help me on to that chair over there? I’m afraid this desert heat has taken all the energy|
|out of me. And I know with your power of flight, it would be easy to lift me.|
|Vulture: Ok… just… hold still. I wouldn’t want to squish you on accident! Here we go!|
|Action: hug spiders|
|Spiders: Oh it is so nice to meet such a kind soul in such a sad dying place as this. For your kindness you will be included|
|in my fable, I am sure.|
|Action: gesture smile|
|Vulture: Thank you? I think. Do you have a scribe following you about that I don’t see? I didn’t know you were famous!|
|Action: gesture stare|
|Persona 1: Thief||Persona 2: Witch|
|I live alone in a tent in the woods.||I am a fierce witch.|
|I steal food from the townspeople and coal from the blacksmith.||The most powerful across the realm.|
|The village police can not find me to put me in jail.||I am feared and like to freeze people.|
|Carrying: a coal||Carrying: nothing|
|Wearing: a knife||Wearing: a hats, a dress, a Cloak, a ceremonial hat|
|Setting: An odd looking hut that sits outside that Witch’s cottage. It is squat, mushy, and looks like a|
|mushroom. It is be speckled with spots - black and white. The steam of the mushroom is beige. There is a|
|small door that looks like it would fit a puppy through it.|
Thief: Hello witch, waht brings you here?
|Witch: Good day. I am here to collect ingredients for my spells.|
|Thief: Which ingredients do you seek?|
|Witch: A black mushroom, covered in green mold.|
|Thief: Ironic, everything looks like a muchroom around here. I can help you.|
|Witch: You think I need help from a lowly thief? You’re lucky I don’t freeze you where you stand.|
|Action: gesture smile|
|Thief: I can be of some help because i know exactly where that muchrooms flourishes. But i want something in return.|
|Witch: Name your price.|
|Thief: I wish too look different. I am wanted and i dont want them to recognize me. Can you do that?|
|Witch: That is easy. But it also requires a rare ingredient I don’t have, tongue of raven. You must procure that.|
|Action: gesture nod|
|Thief: Interesting, have you seen any ravens nearby?|
|Witch: They fly over the abandoned church. If you are clever enough to catch one I can change your looks.|
|Thief: I think i have an idea on how to catch one. Will you coem with me to catch one? It iwll only take a moment.|
|Witch: Get my mushroom first. I will not change you until I get my ingredients.|
|Action: remove ceremonial hat|