research
∙
09/19/2023
AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Models
Audio-visual representation learning aims to develop systems with human-...
share
research
∙
06/27/2023
Style-transfer based Speech and Audio-visual Scene Understanding for Robot Action Sequence Acquisition from Videos
To realize human-robot collaboration, robots need to execute actions for...
research
∙
06/14/2023
When to Use Efficient Self Attention? Profiling Text, Speech and Image Transformer Variants
We present the first unified study of the efficiency of self-attention-b...
research
∙
05/24/2023
Unit-based Speech-to-Speech Translation Without Parallel Data
We propose an unsupervised speech-to-speech translation (S2ST) system th...
research
∙
05/21/2023
Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech Pre-Training for Adaptation to Unseen Languages
Recent models such as XLS-R and Whisper have made multilingual speech te...
research
∙
05/19/2023
Syllable Discovery and Cross-Lingual Generalization in a Visually Grounded, Self-Supervised Speech Mode
In this paper, we show that representations capturing syllabic units eme...
research
∙
05/18/2023
Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalization
We investigate the emergent abilities of the recently proposed web-scale...
research
∙
12/03/2022
Unsupervised Fine-Tuning Data Selection for ASR Using Self-Supervised Speech Models
Self-supervised learning (SSL) has been able to leverage unlabeled data ...
research
∙
12/02/2022
Continual Learning for On-Device Speech Recognition using Disentangled Conformers
Automatic speech recognition research focuses on training and evaluating...
research
∙
11/02/2022
Phoneme Segmentation Using Self-Supervised Speech Models
We apply transfer learning to the task of phoneme segmentation and demon...
research
∙
11/02/2022
M-SpeechCLIP: Leveraging Large-Scale, Pre-Trained Models for Multilingual Speech to Image Retrieval
This work investigates the use of large-scale, pre-trained models (CLIP ...
research
∙
11/01/2022
Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionality
Recent visuolinguistic pre-trained models show promising progress on var...
research
∙
10/07/2022
C2KD: Cross-Lingual Cross-Modal Knowledge Distillation for Multilingual Text-Video Retrieval
Multilingual text-video retrieval methods have improved significantly in...
research
∙
10/03/2022
SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Model
Data-driven speech processing models usually perform well with a large a...
research
∙
03/30/2022
MAE-AST: Masked Autoencoding Audio Spectrogram Transformer
In this paper, we propose a simple yet powerful improvement over the rec...
research
∙
03/28/2022
Word Discovery in Visually Grounded, Self-Supervised Speech Models
We present a method for visually-grounded spoken term discovery. After t...
research
∙
03/21/2022
Automated detection of foreground speech with wearable sensing in everyday home environments: A transfer learning approach
Acoustic sensing has proved effective as a foundation for numerous appli...
research
∙
02/07/2022
Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling
In this paper, we describe our submissions to the ZeroSpeech 2021 Challe...
research
∙
12/08/2021
Everything at Once – Multi-modal Fusion Transformer for Video Retrieval
Multi-modal learning from video data has seen increased attention recent...
research
∙
12/01/2021
Routing with Self-Attention for Multimodal Capsule Networks
The task of multimodal learning has seen a growing interest recently as ...
research
∙
11/08/2021
Cascaded Multilingual Audio-Visual Learning from Videos
In this paper, we explore self-supervised audio-visual models that learn...
research
∙
09/16/2021
Fast-Slow Transformer for Visually Grounding Speech
We present Fast-Slow Transformer for Visually Grounding Speech, or FaST-...
research
∙
06/14/2021
Learning Audio-Visual Dereverberation
Reverberation from audio reflecting off surfaces and objects in the envi...
research
∙
05/10/2021
Spoken Moments: Learning Joint Audio-Visual Representations from Video Descriptions
When people observe events, they are able to abstract key information an...
research
∙
04/26/2021
Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videos
Multimodal self-supervised learning is getting more and more attention a...
research
∙
12/31/2020
Text-Free Image-to-Speech Synthesis Using Learned Segmental Units
In this paper we present the first model for directly synthesizing fluen...
research
∙
06/16/2020
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos
Current methods for learning visually grounded language from videos ofte...
research
∙
11/21/2019
Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech
In this paper, we present a method for learning discrete linguistic unit...
research
∙
07/09/2019
Transfer Learning from Audio-Visual Grounding to Speech Recognition
Transfer learning aims to reduce the amount of data required to excel at...
research
∙
02/21/2019
Towards Visually Grounded Sub-Word Speech Unit Discovery
In this paper, we investigate the manner in which interpretable sub-word...
research
∙
04/09/2018
Vision as an Interlingua: Learning Multilingual Semantic Embeddings of Untranscribed Speech
In this paper, we explore the learning of neural network embeddings for ...
research
∙
04/04/2018
Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input
In this paper, we explore neural network models that learn to associate ...
research
∙
12/11/2017
Learning Modality-Invariant Representations for Speech and Images
In this paper, we explore the unsupervised learning of a semantic embedd...
research
∙
01/25/2017
Learning Word-Like Units from Joint Audio-Visual Analysis
Given a collection of images and spoken audio captions, we present a met...
research
∙
11/11/2015