3D-to-2D Distillation for Indoor Scene Parsing

04/06/2021
by   Zhengzhe Liu, et al.
15

Indoor scene semantic parsing from RGB images is very challenging due to occlusions, object distortion, and viewpoint variations. Going beyond prior works that leverage geometry information, typically paired depth maps, we present a new approach, a 3D-to-2D distillation framework, that enables us to leverage 3D features extracted from large-scale 3D data repository (e.g., ScanNet-v2) to enhance 2D features extracted from RGB images. Our work has three novel contributions. First, we distill 3D knowledge from a pretrained 3D network to supervise a 2D network to learn simulated 3D features from 2D features during the training, so the 2D network can infer without requiring 3D data. Second, we design a two-stage dimension normalization scheme to calibrate the 2D and 3D features for better integration. Third, we design a semantic-aware adversarial training model to extend our framework for training with unpaired 3D data. Extensive experiments on various datasets, ScanNet-V2, S3DIS, and NYU-v2, demonstrate the superiority of our approach. Also, experimental results show that our 3D-to-2D distillation improves the model generalization.

READ FULL TEXT

page 1

page 3

page 7

page 8

page 10

page 11

page 14

page 15

research
02/13/2023

A Deep Learning-based Global and Segmentation-based Semantic Feature Fusion Approach for Indoor Scene Classification

Indoor scene classification has become an important task in perception m...
research
02/22/2020

Shallow2Deep: Indoor Scene Modeling by Single Image Understanding

Dense indoor scene modeling from 2D images has been bottlenecked due to ...
research
11/01/2019

Centroid-Based Scene Classification (CBSC): Using Deep Features and Clustering for RGB-D Indoor Scene Classification

This paper contributes a novel method for RGB-D indoor scene classificat...
research
06/17/2021

Dynamic Knowledge Distillation with A Single Stream Structure for RGB-DSalient Object Detection

RGB-D salient object detection(SOD) demonstrates its superiority on dete...
research
11/25/2022

Privileged Prior Information Distillation for Image Matting

Performance of trimap-free image matting methods is limited when trying ...
research
12/12/2021

BIPS: Bi-modal Indoor Panorama Synthesis via Residual Depth-aided Adversarial Learning

Providing omnidirectional depth along with RGB information is important ...
research
04/09/2018

Large scale distributed neural network training through online distillation

Techniques such as ensembling and distillation promise model quality imp...

Please sign up or login with your details

Forgot password? Click here to reset