OpenPifPaf: Composite Fields for Semantic Keypoint Detection and Spatio-Temporal Association

03/03/2021
by   Sven Kreiss, et al.
7

Many image-based perception tasks can be formulated as detecting, associating and tracking semantic keypoints, e.g., human body pose estimation and tracking. In this work, we present a general framework that jointly detects and forms spatio-temporal keypoint associations in a single stage, making this the first real-time pose detection and tracking algorithm. We present a generic neural network architecture that uses Composite Fields to detect and construct a spatio-temporal pose which is a single, connected graph whose nodes are the semantic keypoints (e.g., a person's body joints) in multiple frames. For the temporal associations, we introduce the Temporal Composite Association Field (TCAF) which requires an extended network architecture and training method beyond previous Composite Fields. Our experiments show competitive accuracy while being an order of magnitude faster on multiple publicly available datasets such as COCO, CrowdPose and the PoseTrack 2017 and 2018 datasets. We also show that our method generalizes to any class of semantic keypoints such as car and animal parts to provide a holistic perception framework that is well suited for urban mobility such as self-driving cars and delivery robots.

READ FULL TEXT

page 1

page 3

page 5

page 6

page 8

page 9

page 10

page 13

03/15/2019

PifPaf: Composite Fields for Human Pose Estimation

We propose a new bottom-up method for multi-person 2D human pose estimat...
06/17/2020

BlazePose: On-device Real-time Body Pose tracking

We present BlazePose, a lightweight convolutional neural network archite...
11/29/2018

Efficient Online Multi-Person 2D Pose Tracking with Recurrent Spatio-Temporal Affinity Fields

We present an online approach to efficiently and simultaneously detect a...
12/05/2016

ArtTrack: Articulated Multi-person Tracking in the Wild

In this paper we propose an approach for articulated tracking of multipl...
02/24/2021

4D Panoptic LiDAR Segmentation

Temporal semantic scene understanding is critical for self-driving cars ...
04/28/2018

Remote Detection of Idling Cars Using Infrared Imaging and Deep Networks

Idling vehicles waste energy and pollute the environment through exhaust...
03/24/2020

Capturing and Explaining Trajectory Singularities using Composite Signal Neural Networks

Spatial trajectories are ubiquitous and complex signals. Their analysis ...

Code Repositories

openpifpaf

Official implementation of "OpenPifPaf: Composite Fields for Semantic Keypoint Detection and Spatio-Temporal Association" in PyTorch.


view repo

openpifpaf

Official implementation of "PifPaf: Composite Fields for Human Pose Estimation" in PyTorch.


view repo

openpifpafwebdemo

Web browser based demo of OpenPifPaf.


view repo

openpifpafwebdemo

Web browser based demo of OpenPifPaf.


view repo

openpifpaf_posetrack

OpenPifPaf plugin for Posetrack


view repo