Improving Audio Captioning Using Semantic Similarity Metrics

10/29/2022
by   Rehana Mahfuz, et al.
0

Audio captioning quality metrics which are typically borrowed from the machine translation and image captioning areas measure the degree of overlap between predicted tokens and gold reference tokens. In this work, we consider a metric measuring semantic similarities between predicted and reference captions instead of measuring exact word overlap. We first evaluate its ability to capture similarities among captions corresponding to the same audio file and compare it to other established metrics. We then propose a fine-tuning method to directly optimize the metric by backpropagating through a sentence embedding extractor and audio captioning network. Such fine-tuning results in an improvement in predicted captions as measured by both traditional metrics and the proposed semantic similarity captioning metric.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/10/2021

Can Audio Captions Be Evaluated with Image Caption Metrics?

Automated audio captioning aims at generating textual descriptions for a...
research
09/06/2023

Detecting False Alarms and Misses in Audio Captions

Metrics to evaluate audio captions simply provide a score without much e...
research
11/14/2022

Is my automatic audio captioning system so bad? spider-max: a metric to consider several caption candidates

Automatic Audio Captioning (AAC) is the task that aims to describe an au...
research
07/31/2023

Guiding Image Captioning Models Toward More Specific Captions

Image captioning is conventionally formulated as the task of generating ...
research
04/08/2022

On Distinctive Image Captioning via Comparing and Reweighting

Recent image captioning models are achieving impressive results based on...
research
05/31/2019

What does a Car-ssette tape tell?

Captioning has attracted much attention in image and video understanding...
research
09/16/2022

Belief Revision based Caption Re-ranker with Visual Semantic Information

In this work, we focus on improving the captions generated by image-capt...

Please sign up or login with your details

Forgot password? Click here to reset