Joint Forecasting of Features and Feature Motion for Dense Semantic Future Prediction

by   Josip Šarić, et al.

We present a novel dense semantic forecasting approach which is applicable to a variety of architectures and tasks. The approach consists of two modules. Feature-to-motion (F2M) module forecasts a dense deformation field which warps past features into their future positions. Feature-to-feature (F2F) module regresses the future features directly and is therefore able to account for emergent scenery. The compound F2MF approach decouples effects of motion from the effects of novelty in a task-agnostic manner. We aim to apply F2MF forecasting to the most subsampled and the most abstract representation of a desired single-frame model. Our implementations take advantage of deformable convolutions and pairwise correlation coefficients across neighbouring time instants. We perform experiments on three dense prediction tasks: semantic segmentation, instance-level segmentation, and panoptic segmentation. The results reveal state-of-the-art forecasting accuracy across all three modalities on the Cityscapes dataset.



There are no comments yet.


page 1

page 6

page 7

page 9

page 11

page 12


Single Level Feature-to-Feature Forecasting with Deformable Convolutions

Future anticipation is of vital importance in autonomous driving and oth...

Predicting Future Instance Segmentations by Forecasting Convolutional Features

Anticipating future events is an important prerequisite towards intellig...

Fast Semantic Segmentation on Video Using Motion Vector-Based Feature Interpolation

Models optimized for accuracy on challenging, dense prediction tasks suc...

Recurrent Flow-Guided Semantic Forecasting

Understanding the world around us and making decisions about the future ...

Future Segmentation Using 3D Structure

Predicting the future to anticipate the outcome of events and actions is...

HOME: Heatmap Output for future Motion Estimation

In this paper, we propose HOME, a framework tackling the motion forecast...

Raising context awareness in motion forecasting

Learning-based trajectory prediction models have encountered great succe...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.