Multi-dataset Pretraining: A Unified Model for Semantic Segmentation

06/08/2021
by   Bowen Shi, et al.
0

Collecting annotated data for semantic segmentation is time-consuming and hard to scale up. In this paper, we for the first time propose a unified framework, termed as Multi-Dataset Pretraining, to take full advantage of the fragmented annotations of different datasets. The highlight is that the annotations from different domains can be efficiently reused and consistently boost performance for each specific domain. This is achieved by first pretraining the network via the proposed pixel-to-prototype contrastive loss over multiple datasets regardless of their taxonomy labels, and followed by fine-tuning the pretrained model over specific dataset as usual. In order to better model the relationship among images and classes from different datasets, we extend the pixel level embeddings via cross dataset mixing and propose a pixel-to-class sparse coding strategy that explicitly models the pixel-class similarity over the manifold embedding space. In this way, we are able to increase intra-class compactness and inter-class separability, as well as considering inter-class similarity across different datasets for better transferability. Experiments conducted on several benchmarks demonstrate its superior performance. Notably, MDP consistently outperforms the pretrained models over ImageNet by a considerable margin, while only using less than 10 samples for pretraining.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/13/2020

Contrastive Learning for Label-Efficient Semantic Segmentation

Collecting labeled data for the task of semantic segmentation is expensi...
research
02/27/2023

LMSeg: Language-guided Multi-dataset Segmentation

It's a meaningful and attractive topic to build a general and inclusive ...
research
01/14/2020

SimEx: Express Prediction of Inter-dataset Similarity by a Fleet of Autoencoders

Knowing the similarity between sets of data has a number of positive imp...
research
03/21/2021

Cross-Dataset Collaborative Learning for Semantic Segmentation

Recent work attempts to improve semantic segmentation performance by exp...
research
04/10/2021

Learning from 2D: Pixel-to-Point Knowledge Transfer for 3D Pretraining

Most of the 3D networks are trained from scratch owning to the lack of l...
research
09/11/2018

Unbiasing Semantic Segmentation For Robot Perception using Synthetic Data Feature Transfer

Robot perception systems need to perform reliable image segmentation in ...
research
05/25/2022

Primitive3D: 3D Object Dataset Synthesis from Randomly Assembled Primitives

Numerous advancements in deep learning can be attributed to the access t...

Please sign up or login with your details

Forgot password? Click here to reset