Prompt-guided Scene Generation for 3D Zero-Shot Learning

09/29/2022
by   Majid Nasiri, et al.
0

Zero-shot learning on 3D point cloud data is a related underexplored problem compared to its 2D image counterpart. 3D data brings new challenges for ZSL due to the unavailability of robust pre-trained feature extraction models. To address this problem, we propose a prompt-guided 3D scene generation and supervision method that augments 3D data to learn the network better, exploring the complex interplay of seen and unseen objects. First, we merge point clouds of two 3D models in certain ways described by a prompt. The prompt acts like the annotation describing each 3D scene. Later, we perform contrastive learning to train our proposed architecture in an end-to-end manner. We argue that 3D scenes can relate objects more efficiently than single objects because popular language models (like BERT) can achieve high performance when objects appear in a context. Our proposed prompt-guided scene generation method encapsulates data augmentation and prompt-based annotation/captioning to improve 3D ZSL performance. We have achieved state-of-the-art ZSL and generalized ZSL performance on synthetic (ModelNet40, ModelNet10) and real-scanned (ScanOjbectNN) 3D object datasets.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/15/2019

Mitigating the Hubness Problem for Zero-Shot Learning of 3D Objects

The development of advanced 3D sensors has enabled many objects to be ca...
research
04/11/2021

Zero-Shot Learning on 3D Point Cloud Objects and Beyond

Zero-shot learning, the task of learning to recognize new classes not se...
research
07/01/2021

Segmenting 3D Hybrid Scenes via Zero-Shot Learning

This work is to tackle the problem of point cloud semantic segmentation ...
research
03/20/2022

Towards 3D Scene Understanding by Referring Synthetic Models

Promising performance has been achieved for visual perception on the poi...
research
05/25/2022

ZeroGen^+: Self-Guided High-Quality Data Generation in Efficient Zero-Shot Learning

Nowadays, owing to the superior capacity of the large pre-trained langua...
research
03/03/2021

Energy-Based Learning for Scene Graph Generation

Traditional scene graph generation methods are trained using cross-entro...
research
06/30/2023

Topological Data Analysis Guided Segment Anything Model Prompt Optimization for Zero-Shot Segmentation in Biological Imaging

Emerging foundation models in machine learning are models trained on vas...

Please sign up or login with your details

Forgot password? Click here to reset