Fast Retinomorphic Event Stream for Video Recognition and ReinforcementLearning

05/16/2018
by   Wanjia Liu, et al.
0

Good temporal representations are crucial for video understanding, and the state-of-the-art video recognition frame- work is based on two-stream networks. In such framework, besides the regular ConvNets responsible for RGB frame inputs, a second network is introduced to handle the temporal representation, usually the optical flow (OF). However, OF or other task-oriented flow is computationally costly, and is thus typically pre-computed. Critically, this prevents the two-stream approach from being applied to reinforcement learning (RL) applications such as video game playing, where the next state depends on current state and action choices. Inspired by the early vision systems of mammals and in- sects, we propose a fast event-driven input representation (EDR) that models several major properties of early retinal circuits1: (1) logarithmic input response, (2) multi-timescale temporal smoothing to filter noise, and (3) bipolar (ON/OFF) pathways for primitive event detection. Trading off the directional information for fast speed (> 9000 fps), EDR enables fast real-time inference/learning in video applications that require interaction between an agent and the world such as game-playing, virtual robotics, and domain adaptation. In this vein, we use EDR to demonstrate performance improvements over state-of-the-art reinforcement learning algorithms for Atari games, something that has not been possible with pre-computed OF. Moreover, with UCF-101 video action recognition experiments, we show that EDR performs near state-of-the-art in accuracy while achieving a 1,500x speedup in input representation processing ( 9K frames/sec), as compared to optical flow.

READ FULL TEXT

page 1

page 4

page 6

page 9

page 12

page 13

research
05/16/2018

Fast Retinomorphic Event Stream for Video Recognition and Reinforcement Learning

Good temporal representations are crucial for video understanding, and t...
research
06/07/2022

TadML: A fast temporal action detection with Mechanics-MLP

Temporal Action Detection(TAD) is a crucial but challenging task in vide...
research
12/07/2021

E^2(GO)MOTION: Motion Augmented Event Stream for Egocentric Action Recognition

Event cameras are novel bio-inspired sensors, which asynchronously captu...
research
05/08/2018

FFNet: Video Fast-Forwarding via Reinforcement Learning

For many applications with limited computation, communication, storage a...
research
06/17/2018

Task-Relevant Object Discovery and Categorization for Playing First-person Shooter Games

We consider the problem of learning to play first-person shooter (FPS) v...
research
01/12/2017

Ordered Pooling of Optical Flow Sequences for Action Recognition

Training of Convolutional Neural Networks (CNNs) on long video sequences...
research
04/12/2017

Predictive-Corrective Networks for Action Detection

While deep feature learning has revolutionized techniques for static-ima...

Please sign up or login with your details

Forgot password? Click here to reset