The Influence of Audio on Video Memorability with an Audio Gestalt Regulated Video Memorability System

04/23/2021
by   Lorin Sweeney, et al.
2

Memories are the tethering threads that tie us to the world, and memorability is the measure of their tensile strength. The threads of memory are spun from fibres of many modalities, obscuring the contribution of a single fibre to a thread's overall tensile strength. Unfurling these fibres is the key to understanding the nature of their interaction, and how we can ultimately create more meaningful media content. In this paper, we examine the influence of audio on video recognition memorability, finding evidence to suggest that it can facilitate overall video recognition memorability rich in high-level (gestalt) audio features. We introduce a novel multimodal deep learning-based late-fusion system that uses audio gestalt to estimate the influence of a given video's audio on its overall short-term recognition memorability, and selectively leverages audio features to make a prediction accordingly. We benchmark our audio gestalt based system on the Memento10k short-term video memorability dataset, achieving top-2 state-of-the-art results.

READ FULL TEXT
research
12/31/2020

Leveraging Audio Gestalt to Predict Media Memorability

Memorability determines what evanesces into emptiness, and what worms it...
research
12/10/2019

Listen to Look: Action Recognition by Previewing Audio

In the face of the video data deluge, today's expensive clip-level class...
research
11/22/2017

Integrating both Visual and Audio Cues for Enhanced Video Caption

Video caption refers to generating a descriptive sentence for a specific...
research
06/26/2022

State of the Art of Audio- and Video-Based Solutions for AAL

The report illustrates the state of the art of the most successful AAL a...
research
11/01/2019

Multimodal Video-based Apparent Personality Recognition Using Long Short-Term Memory and Convolutional Neural Networks

Personality computing and affective computing, where the recognition of ...
research
09/14/2020

Themes Informed Audio-visual Correspondence Learning

The applications of short-term user-generated video (UGV), such as Snapc...
research
04/11/2023

Audio Bank: A High-Level Acoustic Signal Representation for Audio Event Recognition

Automatic audio event recognition plays a pivotal role in making human r...

Please sign up or login with your details

Forgot password? Click here to reset