Understanding Road Layout from Videos as a Whole

by   Buyu Liu, et al.

In this paper, we address the problem of inferring the layout of complex road scenes from video sequences. To this end, we formulate it as a top-view road attributes prediction problem and our goal is to predict these attributes for each frame both accurately and consistently. In contrast to prior work, we exploit the following three novel aspects: leveraging camera motions in videos, including context cuesand incorporating long-term video information. Specifically, we introduce a model that aims to enforce prediction consistency in videos. Our model consists of one LSTM and one Feature Transform Module (FTM). The former implicitly incorporates the consistency constraint with its hidden states, and the latter explicitly takes the camera motion into consideration when aggregating information along videos. Moreover, we propose to incorporate context information by introducing road participants, e.g. objects, into our model. When the entire video sequence is available, our model is also able to encode both local and global cues, e.g. information from both past and future frames. Experiments on two data sets show that: (1) Incorporating either globalor contextual cues improves the prediction accuracy and leveraging both gives the best performance. (2) Introducing the LSTM and FTM modules improves the prediction consistency in videos. (3) The proposed method outperforms the SOTA by a large margin.


page 1

page 3

page 5

page 6

page 8


A Parametric Top-View Representation of Complex Road Scenes

In this paper, we address the problem of inferring the layout of complex...

Temporally Consistent Video Transformer for Long-Term Video Prediction

Generating long, temporally consistent video remains an open challenge i...

Context-TAP: Tracking Any Point Demands Spatial Context Features

We tackle the problem of Tracking Any Point (TAP) in videos, which speci...

MMVP: Motion-Matrix-based Video Prediction

A central challenge of video prediction lies where the system has to rea...

Temporal View Synthesis of Dynamic Scenes through 3D Object Motion Estimation with Multi-Plane Images

The challenge of graphically rendering high frame-rate videos on low com...

Dynamic and Static Context-aware LSTM for Multi-agent Motion Prediction

Multi-agent motion prediction is challenging because it aims to foresee ...

Revealing Disocclusions in Temporal View Synthesis through Infilling Vector Prediction

We consider the problem of temporal view synthesis, where the goal is to...

Please sign up or login with your details

Forgot password? Click here to reset