Log In Sign Up

Multi-dataset Pretraining: A Unified Model for Semantic Segmentation

by   Bowen Shi, et al.

Collecting annotated data for semantic segmentation is time-consuming and hard to scale up. In this paper, we for the first time propose a unified framework, termed as Multi-Dataset Pretraining, to take full advantage of the fragmented annotations of different datasets. The highlight is that the annotations from different domains can be efficiently reused and consistently boost performance for each specific domain. This is achieved by first pretraining the network via the proposed pixel-to-prototype contrastive loss over multiple datasets regardless of their taxonomy labels, and followed by fine-tuning the pretrained model over specific dataset as usual. In order to better model the relationship among images and classes from different datasets, we extend the pixel level embeddings via cross dataset mixing and propose a pixel-to-class sparse coding strategy that explicitly models the pixel-class similarity over the manifold embedding space. In this way, we are able to increase intra-class compactness and inter-class separability, as well as considering inter-class similarity across different datasets for better transferability. Experiments conducted on several benchmarks demonstrate its superior performance. Notably, MDP consistently outperforms the pretrained models over ImageNet by a considerable margin, while only using less than 10 samples for pretraining.


page 1

page 2

page 3

page 4


Contrastive Learning for Label-Efficient Semantic Segmentation

Collecting labeled data for the task of semantic segmentation is expensi...

SimEx: Express Prediction of Inter-dataset Similarity by a Fleet of Autoencoders

Knowing the similarity between sets of data has a number of positive imp...

Decoder Denoising Pretraining for Semantic Segmentation

Semantic segmentation labels are expensive and time consuming to acquire...

Cross-Dataset Collaborative Learning for Semantic Segmentation

Recent work attempts to improve semantic segmentation performance by exp...

Learning from 2D: Pixel-to-Point Knowledge Transfer for 3D Pretraining

Most of the 3D networks are trained from scratch owning to the lack of l...

Unbiasing Semantic Segmentation For Robot Perception using Synthetic Data Feature Transfer

Robot perception systems need to perform reliable image segmentation in ...

NullSpaceNet: Nullspace Convoluional Neural Network with Differentiable Loss Function

We propose NullSpaceNet, a novel network that maps from the pixel level ...