Learning Spatio-temporal Features with Partial Expression Sequences for on-the-Fly Prediction

11/29/2017
by   Wissam J. Baddar, et al.
0

Spatio-temporal feature encoding is essential for encoding facial expression dynamics in video sequences. At test time, most spatio-temporal encoding methods assume that a temporally segmented sequence is fed to a learned model, which could require the prediction to wait until the full sequence is available to an auxiliary task that performs the temporal segmentation. This causes a delay in predicting the expression. In an interactive setting, such as affective interactive agents, such delay in the prediction could not be tolerated. Therefore, training a model that can accurately predict the facial expression "on-the-fly" (as they are fed to the system) is essential. In this paper, we propose a new spatio-temporal feature learning method, which would allow prediction with partial sequences. As such, the prediction could be performed on-the-fly. The proposed method utilizes an estimated expression intensity to generate dense labels, which are used to regulate the prediction model training with a novel objective function. As results, the learned spatio-temporal features can robustly predict the expression with partial (incomplete) expression sequences, on-the-fly. Experimental results showed that the proposed method achieved higher recognition rates compared to the state-of-the-art methods on both datasets. More importantly, the results verified that the proposed method improved the prediction frames with partial expression sequence inputs.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/16/2018

Mode Variational LSTM Robust to Unseen Modes of Variation: Application to Facial Expression Recognition

Spatio-temporal feature encoding is essential for encoding the dynamics ...
research
05/10/2022

Spatio-Temporal Transformer for Dynamic Facial Expression Recognition in the Wild

Previous methods for dynamic facial expression in the wild are mainly ba...
research
04/16/2019

LBVCNN: Local Binary Volume Convolutional Neural Network for Facial Expression Recognition from Image Sequences

Recognizing facial expressions is one of the central problems in compute...
research
03/25/2013

Spatio-Temporal Covariance Descriptors for Action and Gesture Recognition

We propose a new action and gesture recognition method based on spatio-t...
research
09/19/2020

Recognizing Micro-expression in Video Clip with Adaptive Key-frame Mining

As a spontaneous expression of emotion on face, micro-expression is rece...
research
03/20/2020

Predicting Real-Time Locational Marginal Prices: A GAN-Based Video Prediction Approach

In this paper, we propose an unsupervised data-driven approach to predic...

Please sign up or login with your details

Forgot password? Click here to reset