VideoPro: A Visual Analytics Approach for Interactive Video Programming

by   Jianben He, et al.

Constructing supervised machine learning models for real-world video analysis require substantial labeled data, which is costly to acquire due to scarce domain expertise and laborious manual inspection. While data programming shows promise in generating labeled data at scale with user-defined labeling functions, the high dimensional and complex temporal information in videos poses additional challenges for effectively composing and evaluating labeling functions. In this paper, we propose VideoPro, a visual analytics approach to support flexible and scalable video data programming for model steering with reduced human effort. We first extract human-understandable events from videos using computer vision techniques and treat them as atomic components of labeling functions. We further propose a two-stage template mining algorithm that characterizes the sequential patterns of these events to serve as labeling function templates for efficient data labeling. The visual interface of VideoPro facilitates multifaceted exploration, examination, and application of the labeling templates, allowing for effective programming of video data at scale. Moreover, users can monitor the impact of programming on model performance and make informed adjustments during the iterative programming process. We demonstrate the efficiency and effectiveness of our approach with two case studies and expert interviews.


page 1

page 4


Instance Selection Mechanisms for Human-in-the-Loop Systems in Few-Shot Learning

Business analytics and machine learning have become essential success fa...

Data Programming by Demonstration: A Framework for Interactively Learning Labeling Functions

Data programming is a programmatic weak supervision approach to efficien...

OneLabeler: A Flexible System for Building Data Labeling Tools

Labeled datasets are essential for supervised machine learning. Various ...

Using ScrutinAI for Visual Inspection of DNN Performance in a Medical Use Case

Our Visual Analytics (VA) tool ScrutinAI supports human analysts to inve...

EventAnchor: Reducing Human Interactions in Event Annotation of Racket Sports Videos

The popularity of racket sports (e.g., tennis and table tennis) leads to...

Towards Automated Infographic Design: Deep Learning-based Auto-Extraction of Extensible Timeline

Designers need to consider not only perceptual effectiveness but also vi...

Sparse Transfer Learning for Interactive Video Search Reranking

Visual reranking is effective to improve the performance of the text-bas...

Please sign up or login with your details

Forgot password? Click here to reset