Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos

04/16/2021
by   Yanghao Li, et al.
0

We introduce an approach for pre-training egocentric video models using large-scale third-person video datasets. Learning from purely egocentric data is limited by low dataset scale and diversity, while using purely exocentric (third-person) data introduces a large domain mismatch. Our idea is to discover latent signals in third-person video that are predictive of key egocentric-specific properties. Incorporating these signals as knowledge distillation losses during pre-training results in models that benefit from both the scale and diversity of third-person video data, as well as representations that capture salient egocentric properties. Our experiments show that our Ego-Exo framework can be seamlessly integrated into standard video models; it outperforms all baselines when fine-tuned for egocentric activity recognition, achieving state-of-the-art results on Charades-Ego and EPIC-Kitchens-100.

READ FULL TEXT

page 1

page 3

page 5

page 7

page 8

page 15

page 16

research
12/07/2020

Unsupervised Pre-training for Person Re-identification

In this paper, we present a large scale unlabeled person re-identificati...
research
06/21/2021

Pre-training also Transfers Non-Robustness

Pre-training has enabled many state-of-the-art results on many tasks. In...
research
05/02/2019

Large-scale weakly-supervised pre-training for video action recognition

Current fully-supervised video datasets consist of only a few hundred th...
research
02/06/2021

SM+: Refined Scale Match for Tiny Person Detection

Detecting tiny objects ( e.g., less than 20 x 20 pixels) in large-scale ...
research
03/24/2023

Application-Driven AI Paradigm for Person Counting in Various Scenarios

Person counting is considered as a fundamental task in video surveillanc...
research
04/25/2018

Actor and Observer: Joint Modeling of First and Third-Person Videos

Several theories in cognitive neuroscience suggest that when people inte...
research
03/31/2016

The Open World of Micro-Videos

Micro-videos are six-second videos popular on social media networks with...

Please sign up or login with your details

Forgot password? Click here to reset