DeepAI AI Chat
Log In Sign Up

iLabel: Interactive Neural Scene Labelling

by   Shuaifeng Zhi, et al.
Imperial College London

Joint representation of geometry, colour and semantics using a 3D neural field enables accurate dense labelling from ultra-sparse interactions as a user reconstructs a scene in real-time using a handheld RGB-D sensor. Our iLabel system requires no training data, yet can densely label scenes more accurately than standard methods trained on large, expensively labelled image datasets. Furthermore, it works in an 'open set' manner, with semantic classes defined on the fly by the user. iLabel's underlying model is a multilayer perceptron (MLP) trained from scratch in real-time to learn a joint neural scene representation. The scene model is updated and visualised in real-time, allowing the user to focus interactions to achieve efficient labelling. A room or similar scene can be accurately labelled into 10+ semantic categories with only a few tens of clicks. Quantitative labelling accuracy scales powerfully with the number of clicks, and rapidly surpasses standard pre-trained semantic segmentation methods. We also demonstrate a hierarchical labelling variant.


page 1

page 3

page 4

page 5

page 6

page 7


Feature-Realistic Neural Fusion for Real-Time, Open Set Scene Understanding

General scene understanding for robotics requires flexible semantic repr...

In-Place Scene Labelling and Understanding with Implicit Scene Representation

Semantic labelling is highly correlated with geometry and radiance recon...

SemanticPaint: A Framework for the Interactive Segmentation of 3D Scenes

We present an open-source, real-time implementation of SemanticPaint, a ...

Realtime strategy for image data labelling using binary models and active sampling

Machine learning (ML) and Deep Learning (DL) tasks primarily depend on d...

Real-time Mapping of Physical Scene Properties with an Autonomous Robot Experimenter

Neural fields can be trained from scratch to represent the shape and app...

Comparing View-Based and Map-Based Semantic Labelling in Real-Time SLAM

Generally capable Spatial AI systems must build persistent scene represe...