PQ-Transformer: Jointly Parsing 3D Objects and Layouts from Point Clouds

09/12/2021
by   Xiaoxue Chen, et al.
17

3D scene understanding from point clouds plays a vital role for various robotic applications. Unfortunately, current state-of-the-art methods use separate neural networks for different tasks like object detection or room layout estimation. Such a scheme has two limitations: 1) Storing and running several networks for different tasks are expensive for typical robotic platforms. 2) The intrinsic structure of separate outputs are ignored and potentially violated. To this end, we propose the first transformer architecture that predicts 3D objects and layouts simultaneously, using point cloud inputs. Unlike existing methods that either estimate layout keypoints or edges, we directly parameterize room layout as a set of quads. As such, the proposed architecture is termed as P(oint)Q(uad)-Transformer. Along with the novel quad representation, we propose a tailored physical constraint loss function that discourages object-layout interference. The quantitative and qualitative evaluations on the public benchmark ScanNet show that the proposed PQ-Transformer succeeds to jointly parse 3D objects and layouts, running at a quasi-real-time (8.91 FPS) rate without efficiency-oriented optimization. Moreover, the new physical constraint loss can improve strong baselines, and the F1-score of the room layout is significantly promoted from 37.9

READ FULL TEXT

page 2

page 3

page 4

page 5

page 6

page 7

page 11

page 12

research
03/03/2022

LGT-Net: Indoor Panoramic Room Layout Estimation with Geometry-Aware Transformer Network

3D room layout estimation by a single panorama using deep neural network...
research
12/17/2018

Floorplan Priors for Joint Camera Pose and Room Layout Estimation

We present a novel approach to reconstruct large or featureless scenes. ...
research
11/27/2018

Iterative Transformer Network for 3D Point Cloud

3D point cloud is an efficient and flexible representation of 3D structu...
research
04/17/2023

U2RLE: Uncertainty-Guided 2-Stage Room Layout Estimation

While the existing deep learning-based room layout estimation techniques...
research
10/04/2022

Bridged Transformer for Vision and Point Cloud 3D Object Detection

3D object detection is a crucial research topic in computer vision, whic...
research
07/03/2017

Physics Inspired Optimization on Semantic Transfer Features: An Alternative Method for Room Layout Estimation

In this paper, we propose an alternative method to estimate room layouts...
research
07/07/2022

MCTS with Refinement for Proposals Selection Games in Scene Understanding

We propose a novel method applicable in many scene understanding problem...

Please sign up or login with your details

Forgot password? Click here to reset