Unseen Object Instance Segmentation with Fully Test-time RGB-D Embeddings Adaptation

by   Lu Zhang, et al.

Segmenting unseen objects is a crucial ability for the robot since it may encounter new environments during the operation. Recently, a popular solution is leveraging RGB-D features of large-scale synthetic data and directly applying the model to unseen real-world scenarios. However, even though depth data have fair generalization ability, the domain shift due to the Sim2Real gap is inevitable, which presents a key challenge to the unseen object instance segmentation (UOIS) model. To tackle this problem, we re-emphasize the adaptation process across Sim2Real domains in this paper. Specifically, we propose a framework to conduct the Fully Test-time RGB-D Embeddings Adaptation (FTEA) based on parameters of the BatchNorm layer. To construct the learning objective for test-time back-propagation, we propose a novel non-parametric entropy objective that can be implemented without explicit classification layers. Moreover, we design a cross-modality knowledge distillation module to encourage the information transfer during test time. The proposed method can be efficiently conducted with test-time images, without requiring annotations or revisiting the large-scale synthetic training data. Besides significant time savings, the proposed method consistently improves segmentation results on both overlap and boundary metrics, achieving state-of-the-art performances on two real-world RGB-D image datasets. We hope our work could draw attention to the test-time adaptation and reveal a promising direction for robot perception in unseen environments.


page 1

page 3

page 7

page 8


Unseen Object Instance Segmentation for Robotic Environments

In order to function in unstructured environments, robots need the abili...

The Best of Both Modes: Separately Leveraging RGB and Depth for Unseen Object Instance Segmentation

In order to function in unstructured environments, robots need the abili...

Learning RGB-D Feature Embeddings for Unseen Object Instance Segmentation

Segmenting unseen objects in cluttered scenes is an important skill that...

Segmenting unseen industrial components in a heavy clutter using rgb-d fusion and synthetic data

Segmentation of unseen industrial parts is essential for autonomous indu...

Test-time Adaptation vs. Training-time Generalization: A Case Study in Human Instance Segmentation using Keypoints Estimation

We consider the problem of improving the human instance segmentation mas...

Fourier Test-time Adaptation with Multi-level Consistency for Robust Classification

Deep classifiers may encounter significant performance degradation when ...

Please sign up or login with your details

Forgot password? Click here to reset