Exploring Fine-Grained Audiovisual Categorization with the SSW60 Dataset

07/21/2022
by   Grant Van Horn, et al.
4

We present a new benchmark dataset, Sapsucker Woods 60 (SSW60), for advancing research on audiovisual fine-grained categorization. While our community has made great strides in fine-grained visual categorization on images, the counterparts in audio and video fine-grained categorization are relatively unexplored. To encourage advancements in this space, we have carefully constructed the SSW60 dataset to enable researchers to experiment with classifying the same set of categories in three different modalities: images, audio, and video. The dataset covers 60 species of birds and is comprised of images from existing datasets, and brand new, expert-curated audio and video datasets. We thoroughly benchmark audiovisual classification performance and modality fusion experiments through the use of state-of-the-art transformer methods. Our findings show that performance of audiovisual fusion methods is better than using exclusively image or audio based methods for the task of video classification. We also present interesting modality transfer experiments, enabled by the unique construction of SSW60 to encompass three different modalities. We hope the SSW60 dataset and accompanying baselines spur research in this fascinating area.

READ FULL TEXT

page 6

page 26

page 27

research
04/18/2020

Feathers dataset for Fine-Grained Visual Categorization

This paper introduces a novel dataset FeatherV1, containing 28,272 image...
research
04/04/2018

Fine-grained Video Attractiveness Prediction Using Multimodal Deep Learning on a Large Real-world Dataset

Nowadays, billions of videos are online ready to be viewed and shared. A...
research
06/23/2020

Facing the Hard Problems in FGVC

In fine-grained visual categorization (FGVC), there is a near-singular f...
research
06/04/2023

CDLT: A Dataset with Concept Drift and Long-Tailed Distribution for Fine-Grained Visual Categorization

Data is the foundation for the development of computer vision, and the e...
research
06/30/2015

A Large-Scale Car Dataset for Fine-Grained Categorization and Verification

Updated on 24/09/2015: This update provides preliminary experiment resul...
research
04/29/2020

Asking without Telling: Exploring Latent Ontologies in Contextual Representations

The success of pretrained contextual encoders, such as ELMo and BERT, ha...
research
04/06/2021

IndoFashion : Apparel Classification for Indian Ethnic Clothes

Cloth categorization is an important research problem that is used by e-...

Please sign up or login with your details

Forgot password? Click here to reset