Temporal Action Localization with Variance-Aware Networks

by   Ting-Ting Xie, et al.

This work addresses the problem of temporal action localization with Variance-Aware Networks (VAN), i.e., DNNs that use second-order statistics in the input and/or the output of regression tasks. We first propose a network (VANp) that when presented with the second-order statistics of the input, i.e., each sample has a mean and a variance, it propagates the mean and the variance throughout the network to deliver outputs with second order statistics. In this framework, both the input and the output could be interpreted as Gaussians. To do so, we derive differentiable analytic solutions, or reasonable approximations, to propagate across commonly used NN layers. To train the network, we define a differentiable loss based on the KL-divergence between the predicted Gaussian and a Gaussian around the ground truth action borders, and use standard back-propagation. Importantly, the variances propagation in VANp does not require any additional parameters, and during testing, does not require any additional computations either. In action localization, the means and the variances of the input are computed at pooling operations, that are typically used to bring arbitrarily long videos to a vector with fixed dimensions. Second, we propose two alternative formulations that augment the first (respectively, the last) layer of a regression network with additional parameters so as to take in the input (respectively, predict in the output) both means and variances. Results in the action localization problem show that the incorporation of second order statistics improves over the baseline network, and that VANp surpasses the accuracy of virtually all other two-stage networks without involving any additional parameters.



There are no comments yet.


page 1

page 2

page 3

page 4


Boundary Uncertainty in a Single-Stage Temporal Action Localization Network

In this paper, we address the problem of temporal action localization wi...

Second-order statistics of fermionic Gaussian states

We study the statistical behavior of entanglement in quantum bipartite s...

Second order Stein: SURE for SURE and other applications in high-dimensional inference

Stein's formula states that a random variable of the form z^ f(z) - div ...

Learning Affinity-Aware Upsampling for Deep Image Matting

We show that learning affinity in upsampling provides an effective and e...

A light-weight method to foster the (Grad)CAM interpretability and explainability of classification networks

We consider a light-weight method which allows to improve the explainabi...

Dynamic Boltzmann Machines for Second Order Moments and Generalized Gaussian Distributions

Dynamic Boltzmann Machine (DyBM) has been shown highly efficient to pred...

Approximate Program Smoothing Using Mean-Variance Statistics, with Application to Procedural Shader Bandlimiting

This paper introduces a general method to approximate the convolution of...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.