DeepAI AI Chat
Log In Sign Up

Multiscale Audio Spectrogram Transformer for Efficient Audio Classification

by   Wentao Zhu, et al.

Audio event has a hierarchical architecture in both time and frequency and can be grouped together to construct more abstract semantic audio classes. In this work, we develop a multiscale audio spectrogram Transformer (MAST) that employs hierarchical representation learning for efficient audio classification. Specifically, MAST employs one-dimensional (and two-dimensional) pooling operators along the time (and frequency domains) in different stages, and progressively reduces the number of tokens and increases the feature dimensions. MAST significantly outperforms AST <cit.> by 22.2%, 4.4% and 4.7% on Kinetics-Sounds, Epic-Kitchens-100 and VGGSound in terms of the top-1 accuracy without external training data. On the downloaded AudioSet dataset, which has over 20% missing audios, MAST also achieves slightly better accuracy than AST. In addition, MAST is 5x more efficient in terms of multiply-accumulates (MACs) with 42% reduction in the number of parameters compared to AST. Through clustering metrics and visualizations, we demonstrate that the proposed MAST can learn semantically more separable feature representations from audio signals.


page 1

page 2

page 3

page 4


Audio Classification from Time-Frequency Texture

Time-frequency representations of audio signals often resemble texture i...

MAST: Multiscale Audio Spectrogram Transformers

We present Multiscale Audio Spectrogram Transformer (MAST) for audio cla...

HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection

Audio classification is an important task of mapping audio samples into ...

SpecTNT: a Time-Frequency Transformer for Music Audio

Transformers have drawn attention in the MIR field for their remarkable ...

A Time-Frequency Perspective on Audio Watermarking

Existing audio watermarking methods usually treat the host audio signals...

End-to-End Probabilistic Inference for Nonstationary Audio Analysis

A typical audio signal processing pipeline includes multiple disjoint an...

SoundSieve: Seconds-Long Audio Event Recognition on Intermittently-Powered Systems

A fundamental problem of every intermittently-powered sensing system is ...