NeSF: Neural Semantic Fields for Generalizable Semantic Segmentation of 3D Scenes

11/25/2021
by   Suhani Vora, et al.
0

We present NeSF, a method for producing 3D semantic fields from posed RGB images alone. In place of classical 3D representations, our method builds on recent work in implicit neural scene representations wherein 3D structure is captured by point-wise functions. We leverage this methodology to recover 3D density fields upon which we then train a 3D semantic segmentation model supervised by posed 2D semantic maps. Despite being trained on 2D signals alone, our method is able to generate 3D-consistent semantic maps from novel camera poses and can be queried at arbitrary 3D points. Notably, NeSF is compatible with any method producing a density field, and its accuracy improves as the quality of the density field improves. Our empirical analysis demonstrates comparable quality to competitive 2D and 3D semantic segmentation baselines on complex, realistically rendered synthetic scenes. Our method is the first to offer truly dense 3D scene segmentations requiring only 2D supervision for training, and does not require any semantic input for inference on novel scenes. We encourage the readers to visit the project website.

READ FULL TEXT

page 5

page 8

page 17

page 19

page 21

page 22

page 23

research
05/21/2023

VL-Fields: Towards Language-Grounded Neural Implicit Spatial Representations

We present Visual-Language Fields (VL-Fields), a neural implicit spatial...
research
03/28/2020

Inferring Semantic Information with 3D Neural Scene Representations

Biological vision infers multi-modal 3D representations that support rea...
research
05/01/2015

SynthCam3D: Semantic Understanding With Synthetic Indoor Scenes

We are interested in automatic scene understanding from geometric cues. ...
research
05/14/2013

A Bag of Words Approach for Semantic Segmentation of Monitored Scenes

This paper proposes a semantic segmentation method for outdoor scenes ca...
research
11/21/2022

SegNeRF: 3D Part Segmentation with Neural Radiance Fields

Recent advances in Neural Radiance Fields (NeRF) boast impressive perfor...
research
10/11/2022

CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory

We propose CLIP-Fields, an implicit scene model that can be trained with...
research
07/24/2023

CarPatch: A Synthetic Benchmark for Radiance Field Evaluation on Vehicle Components

Neural Radiance Fields (NeRFs) have gained widespread recognition as a h...

Please sign up or login with your details

Forgot password? Click here to reset