Joint Forecasting of Features and Feature Motion for Dense Semantic Future Prediction

01/26/2021
by   Josip Šarić, et al.
11

We present a novel dense semantic forecasting approach which is applicable to a variety of architectures and tasks. The approach consists of two modules. Feature-to-motion (F2M) module forecasts a dense deformation field which warps past features into their future positions. Feature-to-feature (F2F) module regresses the future features directly and is therefore able to account for emergent scenery. The compound F2MF approach decouples effects of motion from the effects of novelty in a task-agnostic manner. We aim to apply F2MF forecasting to the most subsampled and the most abstract representation of a desired single-frame model. Our implementations take advantage of deformable convolutions and pairwise correlation coefficients across neighbouring time instants. We perform experiments on three dense prediction tasks: semantic segmentation, instance-level segmentation, and panoptic segmentation. The results reveal state-of-the-art forecasting accuracy across all three modalities on the Cityscapes dataset.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 6

page 7

page 9

page 11

page 12

07/26/2019

Single Level Feature-to-Feature Forecasting with Deformable Convolutions

Future anticipation is of vital importance in autonomous driving and oth...
03/30/2018

Predicting Future Instance Segmentations by Forecasting Convolutional Features

Anticipating future events is an important prerequisite towards intellig...
03/21/2018

Fast Semantic Segmentation on Video Using Motion Vector-Based Feature Interpolation

Models optimized for accuracy on challenging, dense prediction tasks suc...
09/21/2018

Recurrent Flow-Guided Semantic Forecasting

Understanding the world around us and making decisions about the future ...
11/28/2018

Future Segmentation Using 3D Structure

Predicting the future to anticipate the outcome of events and actions is...
05/23/2021

HOME: Heatmap Output for future Motion Estimation

In this paper, we propose HOME, a framework tackling the motion forecast...
09/16/2021

Raising context awareness in motion forecasting

Learning-based trajectory prediction models have encountered great succe...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.