Computational Models of Tutor Feedback in Language Acquisition

by   Jens Nevens, et al.

This paper investigates the role of tutor feedback in language learning using computational models. We compare two dominant paradigms in language learning: interactive learning and cross-situational learning - which differ primarily in the role of social feedback such as gaze or pointing. We analyze the relationship between these two paradigms and propose a new mixed paradigm that combines the two paradigms and allows to test algorithms in experiments that combine no feedback and social feedback. To deal with mixed feedback experiments, we develop new algorithms and show how they perform with respect to traditional knn and prototype approaches.



There are no comments yet.


page 5


Learning Robust Recommender from Noisy Implicit Feedback

The ubiquity of implicit feedback makes it indispensable for building re...

The Markup Language for Designing Gaze Controlled Applications

The Gaze Interaction Markup Language (GIML) is presented, which is new l...

Programming Paradigms, Turing Completeness and Computational Thinking

The notion of programming paradigms, with associated programming languag...

Neuromorphic Control

Neuromorphic engineering is a rapidly developing field that aims to take...

Provably More Efficient Q-Learning in the Full-Feedback/One-Sided-Feedback Settings

We propose two new Q-learning algorithms, Full-Q-Learning (FQL) and Elim...

Relevance Feedback with Latent Variables in Riemann spaces

In this paper we develop and evaluate two methods for relevance feedback...

Integrating Prior Knowledge in Mixed Initiative Social Network Clustering

We propose a new paradigm—called PK-clustering—to help social scientists...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Understanding lexicon learning in children, robots and artificial agents is one of the prime targets for developmental robotics as is evidenced by a steady stream of models, experiments and publications

[1, 2, 3]. A particular focus has been on agent-based models that study language learning using artificial software agents where one agent (the tutor) is teaching another agent (the learner) a particular language. One of the key factors that influences the complexity of the learning task is the presence or absence of social feedback. Consequently, work on lexicon learning traditionally falls into two sets of models: cross-situational learning models and interactive learning.

In cross-situational learning models, the learner sees a context of objects, observes a word (or multiple) and over many interactions has to figure out which object the word refers to. The key problem solved by the learner in these models is which word refers to which object (or which aspects of objects). This paradigm has been studied with infants [4], adults [5, 6] and in simulation [7, 8, 9, 10]. Hypothesis based models store a single hypothesized referent, until evidence for the contrary is presented [11, 12]. However, associative models (e.g. [9]), using statistical analysis of words and objects, align better with empirical findings [3, 13].

The second type of models is called interactive learning. Here the learner not only observes a number of objects and an utterance but additionally receives social feedback (often in the form of pointing or gaze) that clearly restricts the possible referents of the word to one object in the context. Interactive learning models have been proposed for the learning of spatial language [14], color [15] and other domains. They are also a prime paradigm for models of lexicon evolution [16, 17, 18].

Almost all models in lexicon learning fall into either of the two categories. However, very little work has been done to compare these two paradigms systematically, except for [19] who show that cross-situational learning is slower than interactive learning, because the social feedback in interactive learning helps the learner. [19] compare two algorithms: K Nearest Neighbor (KNN) for cross-situational learning and

prototype estimation

for interactive learning. Importantly, real life interactions between caregiver and child probably do not fall into either of the two categories but are really combinations of social feedback, little or no feedback and/or unreliable feedback. Consequently, learning algorithms must be able to deal with situations where there is feedback interleaved with interactions where there is no feedback.

In this paper, we extend traditional approaches such as KNN and prototype learning to mixed interaction scenarios that allow to control the amount of social feedback in each interaction and we propose a number of algorithms for solving mixed cases including the extreme cases of cross-situational learning and interactive learning. We quantify how our algorithms perform with respect to learning success.

The paper proceeds as follows. We first introduce the experimental paradigm and discuss its relation to interactive and cross-situational learning. We then introduce four algorithms and test whether and how they deal with cross-situational, interactive and mixed learning scenarios. We quantify the performance of the algorithms in experiments and draw some conclusions about the impact of tutor feedback in language learning.

Ii Experimental Setup

In this paper, we examine the role of tutor feedback in agent-based models of continuous-valued meaning lexicon learning in consecutive interactions between a learner and a tutor agent. The tutor is given a lexicon of word meaning mappings before the experiment, and the learner has to pick up these words and their meaning in situated interactions. Each experiment has two phases: training and testing. Both take place in the same environment.

Ii-a World

A world consists of a pool of objects of size (also called the world size). Each object is made up of features. In the experiments reported here, the objects consist of three continuous features with . can represent various sensorimotor spaces, e.g. three color channels of the YUV color space. The world is initialized at the start of each experiment by uniformly sampling objects from . Every interaction between tutor and learner takes place in a different environment – the context . The context consists of a set of objects, randomly drawn from (, ).

Ii-B Training

In training, the tutor teaches the learner the language by providing examples of words in particular environments. Training interactions happen in the following way:

  1. Choose a set of objects of size ()

  2. The tutor chooses a random object from the context, called the speaker topic .

  3. The tutor chooses a word to describe or distinguish the topic. How this is done depends on the tutor’s production strategy (explained later). The tutor passes the word to the learner.

  4. Depending on the particular paradigm the learner might or might not receive pointing to the topic from the tutor. The frequency of pointing is determined by a binomial distribution

    with . is a parameter of the experiment.

  5. Word, context and pointing feedback (if available) are used by the learner to update his internal representations.

Ii-C Testing

The goal of the training phase is for the learner to become successful in interacting with the tutor. We check the performance of the learner using testing interactions:

  1. Randomly assign the roles of speaker and hearer to the learner and the tutor.

  2. Choose a set of objects in the context .

  3. The speaker chooses a random object from the context, called the speaker topic .

  4. The speaker chooses a word to describe or distinguish the topic and passes it to the hearer.

  5. The hearer interprets the word and identifies the topic of the utterance. The hearer points to .

  6. The interaction is a success iff

In testing, the learner can be both speaker and hearer. The learner has to show how well he is capable of understanding the tutor (interpretation) and how well he is understood by the tutor (production). Success is confirmed through pointing. That is, a test interaction succeeds if both speaker and hearer think that the word chosen by the speaker refers to the same object.

Ii-D Tutor language

An important aspect of these experiments is how the tutor represents the target language that needs to be acquired by the learner. We here take a prototype approach. The tutor randomly generates words. The meaning of each word is a prototype

. The prototypes are randomly generated from a uniform distribution over


In a particular context and some randomly chosen topic , the tutor as the speaker chooses the prototype closest to the topic and most far away from the closest other object in the context as in the following formula:


with all prototypes known by the tutor, and being the Euclidian distance (). Each prototype corresponds to exactly one word (bijective mapping). So choosing a prototype immediately results in the tutor knowing which word to utter to the learner. This strategy is called discriminative production, because the tutor chooses words that are most discriminative – optimizing distance between the topic and other objects in the context.

The tutor can also be hearer in test interactions. In interactions where the learner is producing words as the speaker, the tutor hears a word and retrieves the prototype . He then has to decide which of the objects in the context the word refers to. He does this by retrieving the object in the context with the shortest distance to :


with being the Euclidian distance.

Ii-E Analysis of the setup

Often work on language learning takes either an interactive (always feedback) or a cross-situational learning (no feedback) point of view. Our setup allows to test algorithms with respect to various forms of feedback. If the tutor always points to the object he has in mind then we are in an interactive learning paradigm (). If the tutor does not point to the topic () then we are in a cross-situational learning paradigm. If the tutor sometimes points and sometimes does not point () then we are in a mixed learning environment.

We evaluate various learning algorithms in the same way using the test interactions. However, for successful test interactions, the representations of tutor and learner do not have to be the same. In fact, our formulation of the learning problem turns word learning into a multi-class, online machine learning problem with noisy labels (depending on pointing frequency). This means that we can use and test a variety of machine learning algorithms.

Notice also that we are in a continuous meaning domain with especially if the tutors prototypes . The tutor lexicon can be larger or smaller than the actual number of objects that the learner encounters. The tutor lexicon does not aim to uniquely identify each object, but instead represents more generic colour categories. Notice also that depending on the training time and the tutor lexicon size, the learner may not have encountered all objects and/or words in training.

The experiments have a number of parameters: - pointing frequency, - dimensionality of the object space, - the number of objects in the world, - the context size for training and testing interactions, and - the words and prototypes (number and exact location in ) that have to be learned.

Iii Learning algorithms

The goal of the learner is to become a proficient speaker and listener with respect to the target language of the tutor. We developed four algorithms able to deal with mixed feedback scenarios, each of them using a different strategy to estimate the tutor language. In particular, we employ a variant of Nearest Neighbor (KNN) adjusted to deal with mixed cases. We also propose three new prototype-based (centroid) algorithms. Both KNN and centroid algorithms have been used for language learning in continuous semantic domains. KNN is the only tested algorithm for continuous meaning cross-situational learning [19]. Centroid algorithms are the goto method for interactive learning [15, 14, 20].

Iii-a k Nearest Neighbor (KNN)

One algorithm that has been applied to continuous meaning domains [19] is -Nearest Neighbor (KNN) classification technique. This is a non-parametric technique that is one of the simplest and most effective methods available in Machine Learning. The learner stores all samples of naming events without computing any abstractions.


The learner keeps a word-object memory that contains the observed objects and words associated with these objects.


In training, the learner observes a word , the context and possibly pointing. For KNN the learner first computes the topic set


and then simply adds a sample for each to his word-object memory. If (there was pointing), then we add times to the word-object memory with .

Production and Interpretation

In testing, the learner has to produce and interpret words. In order to produce a word for the topic , the learner finds the nearest neighbors to the topic using the samples in his word-object lexicon. From these word-object entries, the learner chooses the word that occurs most frequently. Interpretation works similarly. To interpret a given word , the learner will take the

nearest neighbors in its word-object memory for each object in the context and classify each object as the most occurring word in its

neighbors. The interpreted topic is the object which has been classified with . If multiple objects are classified with , one of the objects is chosen at random.


This method has the hyper-parameter . Here we use .

Iii-B Prototype Estimation (PE)

In interactive learning researchers often propose to use prototypes for the learner’s lexicon[15, 14, 20] and to update these prototypes iteratively as new samples of word object mappings are collected by the learner.


The learner stores prototypes .


The prototype is updated when observing the word in a training interaction using the previous prototype and the topic set (see Equation 3) with the following equation.


This update shifts the prototype of the learner more towards the encountered objects mediated by . If the learner has not heard before, then we assume and .


This method has the hyper-parameter with typical .

Production and Interpretation

Since the learner has the same representation as the tutor, we can apply the same production and interpretation equations as the tutor (Equations 1 and 2).


In interactive learning (), the algorithm behaves like a vanilla centroid model with weighted update. The equation then reduces to


which is an update rule proposed in [15]. The update will incorporate all objects in the context in interactions without feedback. The intuition behind this learning rule is the following. Imagine a series of interactions with a context of two objects, one always being the same topic, the other being a randomly chosen distractor object. Suppose further the tutor is always using the same word . By always adding objects to of which one is the correct object, and averaging over them, the prototype for this word should converge towards the prototype known by the tutor, because the distraction objects cancel each other out – if they are generated uniformly.

Iii-C Averaging Prototypes (AP)

This algorithm is a variant of PE that keeps around full sets of samples for words and estimates prototypes based on this set (not just the last sample as in PE).


The learner stores prototypes . He also keeps a list of samples of all past context indexable by the word as .


Suppose the learner has observed the context , he can then compute the topic set (as in Equation 3). The learner estimates the new prototype using and which is all samples for word .


where denotes the concatenation of new samples to the list of samples . If the learner has not heard before, then we assume is empty.

Production and Interpretation

Since the learner has the same representation as the tutor, we can apply the production and interpretation of Equations 1 and 2.


This method has no hyper-parameters.


AP behaves similar to PE but with the difference being that all samples have equal influence, whereas in PE samples are weighted by recency.

Iii-D Co-occurrence Weighted Prototypes (CWP)

CWP is another algorithm that uses prototypes as basic representation. However, here we add additional information from co-occurrence of words and objects into the update rule.


The learner estimates prototypes (centroids) from a set of samples by accumulating examples from the tutor. He also keeps a matrix for tracking co-occurrences between words and object with . is initialized all zeros.


Suppose the learner is in context and has observed . He then estimates topic set as in Equation 3. The learner then updates and using the following two Equations


with being a normalization factor, . We set if the learner hears word for the first time. The rule weights the impact of each object in the topic set on the prototype update by how often word/prototype and object co-occurred. Objects that co-occurred more frequently with the word and are in the topic set have more influence on the prototype . controls how much influence each interaction has on the prototype update.

Production and Interpretation

Since the learner has the same representation as the tutor, we can apply the same production and interpretation (Equations 1 and 2).


In interactive learning (), and . It follows that which means that this rule in interactive learning is the same as for PE. In cross-situational learning, , and consequently the impact of each object in the context is weighted by its overall co-occurrence with word with respect to all other objects in the context. Suppose that we are in a context , and furthermore suppose that has co-occurred always with and never, then will be updated more with . However, if and have co-occurred with roughly the same time, then the update of is averaged over and .

Iv Results

Iv-a Interactive Learning ()

Figure 1 shows the communicative success obtained in testing interactions for the various learning algorithms, all using the interactive learning paradigm (). The prototype-based algorithms (PE, AP and CWP) achieve high levels of communicative success. Also, the communicative success increases very quickly (50% after only 10 interactions). This points to the fact the learner acquires the lexicon very rapidly. We argue that this rapid learning is due to the availability of social feedback which allows these algorithms to update the prototypes using a single object.

This high level of success is not achieved by the KNN algorithm. One reason for this might be the difference in lexicon representation between tutor and learner when using this algorithm. However, as we will discuss later on, the production strategy of the tutor also plays a role.

Fig. 1: Communicative success for increasing training interactions using the interactive learning paradigm (

). World size = 32, context size = 4, tutor lexicon size = 50, 100 testing interactions, 20 repetitions. Error bars show standard deviation.

Iv-B Cross-Situational Learning ()

Fig. 2: Communicative success for increasing training interactions using the cross-situational learning paradigm (). World size = 32, context size = 4, tutor lexicon size = 50, 100 testing interactions, 20 repetitions. Error bars show standard deviation.

Figure 2 compares the communicative success for the four algorithms in a cross-situational learning setting (). When comparing the cross-situational learning setting (Fig. 2) to the interactive learning setting (Fig. 1), we see that removing the social feedback causes the agents to reach lower communicative success, irrespective of the amount of training. Also, the final communicative success achieved after 10000 interactions lies much lower for the cross-situational learning setting. This shows that the availability of social feedback not only speeds up the learning process, but also improves upon it.

While all algorithms show a decrease in success when removing social feedback, this decrease is much larger for PE and AP when compared to CWP, even though they performed similarly in interactive learning. This is because PE and AP assign equal importance to all objects in the context (and former contexts, in the case of AP) when updating the prototypes. CWP estimates the importance of each object, using the co-occurrence counts, and uses this to update the prototypes. We conclude that CWP can handle the absence of social feedback best.

Iv-C Mixed Feedback Environments ()

Figures 3, 4, 5 and 6 show the communicative success for mixed environments. The behavior of these algorithms for and is already discussed above. For the mixed cases (), the availability of feedback in some interactions does give the learner an advantage. Not only does it reduce the amount of training needed to reach a certain level of success, but also the final performance is higher with increasing social feedback.

While PE and AP perform similarly in the extreme settings ( and ), the PE learner benefits more from the availability of some feedback. For example, the increase in success between and is much larger for PE than it is for AP. This is because the presence of feedback allows the PE learner to update its prototype using a single object, while the AP learner still takes previous contexts into account.

Finally, we turn to Fig. 3. As discussed above, the KNN algorithm reaches approximately the same communicative success as the other algorithms in the cross-situational learning setting (). The availability of some feedback () immediately increases the success of the learner, close to the performance of interactive learning. However, the performance with full feedback () is not comparable to that of the other algorithms.

Fig. 3: Communicative success for mixed environments using KNN learning algorithm. World size = 32, context size = 4, tutor lexicon size = 50, 100 testing interactions, 20 repetitions. Error bars show standard deviation.
Fig. 4: Communicative success for mixed environments using PE learning algorithm. World size = 32, context size = 4, tutor lexicon size = 50, 100 testing interactions, 20 repetitions. Error bars show standard deviation.
Fig. 5: Communicative success for mixed environments using AP learning algorithm. World size = 32, context size = 4, tutor lexicon size = 50, 100 testing interactions, 20 repetitions. Error bars show standard deviation.
Fig. 6: Communicative success for mixed environments using CWP learning algorithm. World size = 32, context size = 4, tutor lexicon size = 50, 100 testing interactions, 20 repetitions. Error bars show standard deviation.

Iv-D Impact Production Strategy

One of the parameters in the experimental setup is the tutor strategy for selecting word given a context and a topic . We assumed what is called a discrimination strategy (Equation 1) where the tutor selects the most discriminative word - a word that is closest to the topic and most far away from all other objects in the context. Such strategies are commonly used in word learning models (e.g. [14])

The tutor can also use another word choice strategy that does not presume the word to be discriminative. This is called descriptive production strategy and it is easily described by the following equation where is the prototypes known to the tutor.


We test the descriptive production rule in the same settings as discussed before. Everything else, such as learner algorithms, tutor interpretation mechanism (Equation 2), training and test paradigm stays the same.

Figures 7 and 8 show the effect of the descriptive production rule on cross-situational learning () and interactive learning environments (), respectively. It is clear that in cross-situational learning environments (Figure 7) all algorithms benefit from the descriptive production rule. Similarly, the descriptive production rule also has positive effects on early convergence in interactive learning environments (Figure 8).

This is especially true for the KNN algorithm, which is able to reach similar performance as the other algorithms given descriptive tutors. KNN benefits from the descriptive production rule since this rule will always produce the same word for the same object, whereas the discriminative production strategy may more often produce a different word for the same object (because of the influence of distractor objects). KNN exploits the use of the same label in description to better learn the correct labels for each object. The prototype-based algorithms benefit both from singling out the topic from the context (discrimination) and consistent object naming (description).

Fig. 7: Showing the effect of the tutor’s production strategy on the communicative success in cross-situational learning (). World size = 32, context size = 4, tutor lexicon size = 50, 1000 training interactions, 100 test interactions, 20 repetitions. Error bars show standard deviation.
Fig. 8: Showing the effect of the tutor’s production strategy on the communicative success in interactive learning (). World size = 32, context size = 4, tutor lexicon size = 50, 1000 training interactions, 100 test interactions, 20 repetitions. Error bars show standard deviation.

V Conclusion

Our results show a double effect of tutor feedback on learner success. On the one hand, introducing social feedback in the learning process of the agent allows the agent to acquire the lexicon faster. Across learning algorithms, the learner needs less training to reach the same level of communicative success when there is feedback compared to when there is less or none at all. On the other hand, more feedback also improves the communicative success for the same amount of training. In other words, agents are better at communicating when being trained with more feedback. However, as we have shown in Section IV-D, this is also dependent on the strategy used by the tutor to provide learning opportunities to the learner. Finally, we can note that our novel algorithms, especially CWP, perform well in the continuous semantic domain – a domain which has not been explored extensively in the cross-situational learning paradigm.

A next step is to apply these mechanisms to a data set with real robots. This will allow us to further study the effect of tutor feedback in more challenging environments. Indeed, in a grounded environment, each agent has its own view on the objects, causing perceptual deviation. How the agents will respond to this additional difficulty is part of future work.

Similarly is it important to study the quality of the tutor feedback. In the experiments here, the feedback given by the tutor in the form of pointing is handled implicitly. The agents are able to point to the exact centre of an object and they can perfectly interpret the pointing of other agents. In real life, this may not always be the case. Gaze and pointing are noisy reference mechanisms. Therefore, it is interesting to study how well the learner acquires the lexicon in the case of unreliable social feedback.


  • [1] D. Roy and A. Pentland, “Learning words from sights and sounds: A computational model,” Cognitive science, vol. 26, no. 1, pp. 113–146, 2002.
  • [2] J. Fontanari, V. Tikhanoff, A. Cangelosi, R. Ilin, and L. Perlovsky, “Cross-situational learning of object–word mapping using neural modeling fields,” Neural Networks, vol. 22, no. 5, pp. 579–585, 2009.
  • [3] G. Kachergis, C. Yu, and R. M. Shiffrin, “Cross-situational word learning is better modeled by associations than hypotheses,” in Development and Learning and Epigenetic Robotics (ICDL), 2012 IEEE International Conference on.   IEEE, 2012, pp. 1–6.
  • [4] L. Smith and C. Yu, “Infants rapidly learn word-referent mappings via cross-situational statistics,” Cognition, vol. 106, no. 3, pp. 1558–1568, 2008.
  • [5] C. Yu and L. B. Smith, “Rapid word learning under uncertainty via cross-situational statistics,” Psychological Science, vol. 18, no. 5, pp. 414–420, 2007.
  • [6] Y. Chen and D. Filliat, “Cross-situational noun and adjective learning in an interactive scenario,” in Development and Learning and Epigenetic Robotics (ICDL-EpiRob), 2015 Joint IEEE International Conference on.   IEEE, 2015, pp. 129–134.
  • [7] J. M. Siskind, “A computational study of cross-situational techniques for learning word-to-meaning mappings,” Cognition, vol. 61, no. 1, pp. 39–91, 1996.
  • [8] K. Smith, A. D. Smith, R. A. Blythe, and P. Vogt, “Cross-situational learning: a mathematical approach,” Lecture Notes in Computer Science, vol. 4211, pp. 31–44, 2006.
  • [9] A. Fazly, A. Alishahi, and S. Stevenson, “A probabilistic computational model of cross-situational word learning,” Cognitive Science, vol. 34, no. 6, pp. 1017–1063, 2010.
  • [10] M. C. Frank, N. D. Goodman, and J. B. Tenenbaum, “A bayesian framework for cross-situational word-learning.” in NIPS, 2007, pp. 457–464.
  • [11] T. N. Medina, J. Snedeker, J. C. Trueswell, and L. R. Gleitman, “How words can and cannot be learned by observation,” Proceedings of the National Academy of Sciences, vol. 108, no. 22, pp. 9014–9019, 2011.
  • [12] J. C. Trueswell, T. N. Medina, A. Hafri, and L. R. Gleitman, “Propose but verify: Fast mapping meets cross-situational word learning,” Cognitive psychology, vol. 66, no. 1, pp. 126–156, 2013.
  • [13] G. Cassani, R. Grimm, S. Gillis, and W. Daelemans, “Constraining the search space in cross-situational word learning: Different models make different predictions,” in Proceedings of the 38th Annual Conference of the Cognitive Science Society.   Philadelphia, PA: Cognitive Science Society, 2016.
  • [14] M. Spranger, “Grounded lexicon acquisition - case studies in spatial language,” in Development and Learning and Epigenetic Robotics (ICDL-Epirob), 2013 Joint IEEE International Conferences on.   IEEE, 2013, pp. 1–6.
  • [15] J. Bleys, M. Loetzsch, M. Spranger, and L. Steels, “The Grounded Color Naming Game,” in Proceedings of the 18th IEEE International Symposium on Robot and Human Interactive Communication (Ro-man 2009), 2009.
  • [16] L. Steels, “Language games for autonomous robots,” IEEE Intelligent systems, vol. 16, no. 5, pp. 16–22, 2001.
  • [17] J. Bleys, Language strategies for the domain of colour.   Language Science Press, 2015.
  • [18] M. Spranger, The evolution of grounded spatial language.   Language Science Press, 2016.
  • [19] T. Belpaeme and A. Morse, “Word and category learning in a continuous semantic domain: Comparing cross-situational and interactive learning,” Advances in Complex Systems, vol. 15, no. 3 & 4, p. 1250031, 2012.
  • [20] M. Spranger and K. Beuls, “Referential uncertainty and word learning in high-dimensional, continuous meaning spaces,” in Development and Learning and Epigenetic Robotics (ICDL-Epirob), 2016 Joint IEEE International Conferences on, 2016, V. Hafner and A. Pitti, Eds.   IEEE, 2016.