DeepAI AI Chat
Log In Sign Up

Learning to Recognise Words using Visually Grounded Speech

05/31/2020
by   Sebastiaan Scholten, et al.
Radboud Universiteit
Delft University of Technology
0

We investigated word recognition in a Visually Grounded Speech model. The model has been trained on pairs of images and spoken captions to create visually grounded embeddings which can be used for speech to image retrieval and vice versa. We investigate whether such a model can be used to recognise words by embedding isolated words and using them to retrieve images of their visual referents. We investigate the time-course of word recognition using a gating paradigm and perform a statistical analysis to see whether well known word competition effects in human speech processing influence word recognition. Our experiments show that the model is able to recognise words, and the gating paradigm reveals that words can be recognised from partial input as well and that recognition is negatively influenced by word competition from the word initial cohort.

READ FULL TEXT

page 3

page 4

03/14/2022

Modelling word learning and recognition using visually grounded speech

Background: Computational models of speech recognition often assume that...
09/18/2019

Word Recognition, Competition, and Activation in a Model of Visually Grounded Speech

In this paper, we study how word-like units are represented and activate...
09/09/2019

Language learning using Speech to Image retrieval

Humans learn language by interaction with their environment and listenin...
03/28/2022

Word Discovery in Visually Grounded, Self-Supervised Speech Models

We present a method for visually-grounded spoken term discovery. After t...
02/21/2019

Towards Visually Grounded Sub-Word Speech Unit Discovery

In this paper, we investigate the manner in which interpretable sub-word...
06/16/2021

Semantic sentence similarity: size does not always matter

This study addresses the question whether visually grounded speech recog...
09/01/2020

Hearings and mishearings: decrypting the spoken word

We propose a model of the speech perception of individual words in the p...