DeepAI AI Chat
Log In Sign Up

Social Behavior Prediction from First Person Videos

by   Shan Su, et al.
University of Minnesota
KAIST 수리과학과
University of Pennsylvania

This paper presents a method to predict the future movements (location and gaze direction) of basketball players as a whole from their first person videos. The predicted behaviors reflect an individual physical space that affords to take the next actions while conforming to social behaviors by engaging to joint attention. Our key innovation is to use the 3D reconstruction of multiple first person cameras to automatically annotate each other's the visual semantics of social configurations. We leverage two learning signals uniquely embedded in first person videos. Individually, a first person video records the visual semantics of a spatial and social layout around a person that allows associating with past similar situations. Collectively, first person videos follow joint attention that can link the individuals to a group. We learn the egocentric visual semantics of group movements using a Siamese neural network to retrieve future trajectories. We consolidate the retrieved trajectories from all players by maximizing a measure of social compatibility---the gaze alignment towards joint attention predicted by their social formation, where the dynamics of joint attention is learned by a long-term recurrent convolutional network. This allows us to characterize which social configuration is more plausible and predict future group trajectories.


page 1

page 3

page 4

page 5

page 6

page 7

page 8


Am I a Baller? Basketball Performance Assessment from First-Person Videos

This paper presents a method to assess a basketball player's performance...

A gaze driven fast-forward method for first-person videos

The growing data sharing and life-logging cultures are driving an unprec...

Detecting Attended Visual Targets in Video

We address the problem of detecting attention targets in video. Specific...

Attention Flow: End-to-End Joint Attention Estimation

This paper addresses the problem of understanding joint attention in thi...

Future Person Localization in First-Person Videos

We present a new task that predicts future locations of people observed ...

Actor and Observer: Joint Modeling of First and Third-Person Videos

Several theories in cognitive neuroscience suggest that when people inte...

Using Cross-Model EgoSupervision to Learn Cooperative Basketball Intention

We present a first-person method for cooperative basketball intention pr...