DeepAI AI Chat
Log In Sign Up

Learning Interactions and Relationships between Movie Characters

by   Anna Kukleva, et al.
Max Planck Society

Interactions between people are often governed by their relationships. On the flip side, social relationships are built upon several interactions. Two strangers are more likely to greet and introduce themselves while becoming friends over time. We are fascinated by this interplay between interactions and relationships, and believe that it is an important aspect of understanding social situations. In this work, we propose neural models to learn and jointly predict interactions, relationships, and the pair of characters that are involved. We note that interactions are informed by a mixture of visual and dialog cues, and present a multimodal architecture to extract meaningful information from them. Localizing the pair of interacting characters in video is a time-consuming process, instead, we train our model to learn from clip-level weak labels. We evaluate our models on the MovieGraphs dataset and show the impact of modalities, use of longer temporal context for predicting relationships, and achieve encouraging performance using weak labels as compared with ground-truth labels. Code is online.


page 1

page 7

page 12

page 13

page 14

page 15

page 16

page 17


Movienet: A Movie Multilayer Network Model using Visual and Textual Semantic Cues

Discovering content and stories in movies is one of the most important c...

Modeling Memory Imprints Induced by Interactions in Social Networks

Memory imprints of the significance of relationships are constantly evol...

How you feelin'? Learning Emotions and Mental States in Movie Scenes

Movie story analysis requires understanding characters' emotions and men...

Generative Proxemics: A Prior for 3D Social Interaction from Images

Social interaction is a fundamental aspect of human behavior and communi...

Inferring Interpersonal Relations in Narrative Summaries

Characterizing relationships between people is fundamental for the under...

A Model-Based Approach to Predicting Predator-Prey & Friend-Foe Relationships in Ant Colonies

Understanding predator-prey relationships among insects is a challenging...

Learning Human-Human Interactions in Images from Weak Textual Supervision

Interactions between humans are diverse and context-dependent, but previ...