Three for one and one for three: Flow, Segmentation, and Surface Normals

by   Hoang-An Le, et al.

Optical flow, semantic segmentation, and surface normals represent different information modalities, yet together they bring better cues for scene understanding problems. In this paper, we study the influence between the three modalities: how one impacts on the others and their efficiency in combination. We employ a modular approach using a convolutional refinement network which is trained supervised but isolated from RGB images to enforce joint modality features. To assist the training process, we create a large-scale synthetic outdoor dataset that supports dense annotation of semantic segmentation, optical flow, and surface normals. The experimental results show positive influence among the three modalities, especially for objects' boundaries, region consistency, and scene structures.


page 2

page 8

page 10


Joint Optical Flow and Temporally Consistent Semantic Segmentation

The importance and demands of visual scene understanding have been stead...

EDEN: Multimodal Synthetic Dataset of Enclosed GarDEN Scenes

Multimodal large-scale datasets for outdoor scenes are mostly designed f...

Optical Flow augmented Semantic Segmentation networks for Automated Driving

Motion is a dominant cue in automated driving systems. Optical flow is t...

Unsupervised Domain Adaptation by Optical Flow Augmentation in Semantic Segmentation

It is expensive to generate real-life image labels and there is a domain...

Optical Flow with Semantic Segmentation and Localized Layers

Existing optical flow methods make generic, spatially homogeneous, assum...

Learning Common and Specific Features for RGB-D Semantic Segmentation with Deconvolutional Networks

In this paper, we tackle the problem of RGB-D semantic segmentation of i...

Abstract Flow for Temporal Semantic Segmentation on the Permutohedral Lattice

Semantic segmentation is a core ability required by autonomous agents, a...