Task Programming: Learning Data Efficient Behavior Representations

by   Jennifer J. Sun, et al.

Specialized domain knowledge is often necessary to accurately annotate training sets for in-depth analysis, but can be burdensome and time-consuming to acquire from domain experts. This issue arises prominently in automated behavior analysis, in which agent movements or actions of interest are detected from video tracking data. To reduce annotation effort, we present TREBA: a method to learn annotation-sample efficient trajectory embedding for behavior analysis, based on multi-task self-supervised learning. The tasks in our method can be efficiently engineered by domain experts through a process we call "task programming", which uses programs to explicitly encode structured knowledge from domain experts. Total domain expert effort can be reduced by exchanging data annotation time for the construction of a small number of programmed tasks. We evaluate this trade-off using data from behavioral neuroscience, in which specialized domain knowledge is used to identify behaviors. We present experimental results in three datasets across two domains: mice and fruit flies. Using embeddings from TREBA, we reduce annotation burden by up to a factor of 10 without compromising accuracy compared to state-of-the-art features. Our results thus suggest that task programming can be an effective way to reduce annotation effort for domain experts.


page 1

page 11

page 12

page 17


Automatic Synthesis of Diverse Weak Supervision Sources for Behavior Analysis

Obtaining annotations for large training sets is expensive, especially i...

VIANA: Visual Interactive Annotation of Argumentation

Argumentation Mining addresses the challenging tasks of identifying boun...

Learning from both experts and data

In this work we study the problem of inferring a discrete probability di...

Knowledge Elecitation for Factors Affecting Taskforce Productivity using a Questionnaire

In this paper we present the process of Knowledge Elicitation through a ...

Interpreting Expert Annotation Differences in Animal Behavior

Hand-annotated data can vary due to factors such as subjective differenc...

Learning to Generalize One Sample at a Time with Self-Supervision

Although deep networks have significantly increased the performance of v...

DEXA: Supporting Non-Expert Annotators with Dynamic Examples from Experts

The success of crowdsourcing based annotation of text corpora depends on...

Code Repositories


Learning trajectory representations using self-supervision and programmatic supervision.

view repo