UrbanScene3D: A Large Scale Urban Scene Dataset and Simulator

07/09/2021
by   Yilin Liu, et al.
0

The ability to perceive the environments in different ways is essential to robotic research. This involves the analysis of both 2D and 3D data sources. We present a large scale urban scene dataset associated with a handy simulator based on Unreal Engine 4 and AirSim, which consists of both man-made and real-world reconstruction scenes in different scales, referred to as UrbanScene3D. Unlike previous works that purely based on 2D information or man-made 3D CAD models, UrbanScene3D contains both compact man-made models and detailed real-world models reconstructed by aerial images. Each building has been manually extracted from the entire scene model and then has been assigned with a unique label, forming an instance segmentation map. The provided 3D ground-truth textured models with instance segmentation labels in UrbanScene3D allow users to obtain all kinds of data they would like to have: instance segmentation map, depth map in arbitrary resolution, 3D point cloud/mesh in both visible and invisible places, etc. In addition, with the help of AirSim, users can also simulate the robots (cars/drones)to test a variety of autonomous tasks in the proposed city environment. Please refer to our paper and website(https://vcc.tech/UrbanScene3D/) for further details and applications.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 2

12/01/2016

TorontoCity: Seeing the World with a Million Eyes

In this paper we introduce the TorontoCity benchmark, which covers the f...
12/18/2021

3D Instance Segmentation of MVS Buildings

We present a novel framework for instance segmentation of 3D buildings f...
02/04/2022

StandardSim: A Synthetic Dataset For Retail Environments

Autonomous checkout systems rely on visual and sensory inputs to carry o...
08/04/2017

Augmented Reality Meets Computer Vision : Efficient Data Generation for Urban Driving Scenes

The success of deep learning in computer vision is based on availability...
03/14/2020

OccuSeg: Occupancy-aware 3D Instance Segmentation

3D instance segmentation, with a variety of applications in robotics and...
08/19/2021

CenterPoly: real-time instance segmentation using bounding polygons

We present a novel method, called CenterPoly, for real-time instance seg...
09/08/2021

Panoptic nuScenes: A Large-Scale Benchmark for LiDAR Panoptic Segmentation and Tracking

Panoptic scene understanding and tracking of dynamic agents are essentia...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

Introduction

Figure 1: Overview of UrbanScene3D.

The ability to perceive the environments in different ways is essential to computer graphics, vision, and robotic research. This involves the analysis of various data sources, such as depth map, visual image, and LIDAR data, etc. Related works in 2D/2.5D [coco, kitti] image domains have been proposed. However, a comprehensive understanding of 3D scenes needs the cooperation of 3D data (e.g., point clouds and textured polygon meshes), which is still far from sufficient in the community.

We present a large scale urban scene dataset associated with a handy simulator based on Unreal Engine 4 [unrealengine] and AirSim [airsim], which consists of both man-made and real-world reconstruction scenes in different scales, referred to as UrbanScene3D (https://vcc.tech/UrbanScene3D). The manually made scene models have compact structures, which are carefully constructed/designed by professional modelers according to the images and maps of target areas; see the first row of Figure 1 for a glance. In contrast, UrbanScene3D also offers dense, detailed scene models reconstructed by aerial images through multi-view stereo (MVS) techniques. These scenes have realistic textures and meticulous structures; see e.g., the second row of Figure 1. We have also released the originally captured aerial images that have been used to reconstruct the 3D scene models, as well as a set of 4K video sequences, all of which would facilitate designing algorithms, such SLAM and MVS; please check some samples shown in the third and fourth rows of Figure 1.

Although there are 3D instance segmentation datasets, e.g., S3DIS [Silberman2012], ScanNet [Dai2017], NYUv2 [Armeni2016], and SceneNN [Hua2016]

, they are all collected from indoor scenes and still not enough for deep learning based methods. Please noting that, there is basically no decent dataset for learning 3D instance segmentation in outdoor scenes, especially for complicated urban regions. In this context, our released UrbanScene3D provides rich, large scale 3D urban scene building annotation data for outdoor instance segmentation research. To segment and label 3D urban architectures, we manually extract all single building models from the entire scene model. Every building is then assigned with an unique label, forming an instance segmentation map, which indicates the ground-truth of the instance segmentation task. The provided 3D ground-truth textured models with instance segmentation label in UrbanScene3D allow users to obtain all kinds of data they would like to have: instance segmentation map, depth map in arbitrary resolution, 3D point cloud/mesh in both visible and invisible place, etc. In addition, with the help of AirSim 

[airsim], users can also simulate the robots (cars/drones) to test a variety of autonomous tasks in the proposed city environment; see e.g., the bottom row of Figure 1.

Features and Requirements

We show the statistics and features of our dataset below.

CAD Scene Area () Model () Texture (#) Texture () Object (#)
New York 7.4 86.4 762 122 744
Chicago 24 146 2277 227 1629
San Francisco 55 225 2865 322 2801
Shenzhen 3 50.3 199 72.5 1126
Suzhou 7 191 395 23.7 168
Shanghai 37 308 2285 220 6850
REAL Campus 1.34 1859 122 3676 178
Sci-Art 0.2 458 118 556 3
Residence 0.1 356 52 1760 34
Square 1.1 3665 799 980 156
Hospital 0.5 6266 94 744 114
Table 1: Statistics of our 3D urban dataset. We provide six virtual scenes with their CAD models that are constructed by professional artists according to images or satellite maps. In addition, we also share 5 reconstructed real world scenes, including both their corresponding aerial images and detailed mesh models. All these models have already been associated with building-level instance segmentation. Here, Area () represents the covered area of the scene; Model () represents the size of the model; Texture (#) represents the number of texture images contained in this model; Texture () represents the size of texture; Object (#) represents the number of objects in this scene.
Dataset 2D/3D Bbox Instance Mask Depth Image GT Mesh with Pose
COCO [coco] 2D
KITTI [kitti]
Cityscape [cityscapes] 2D
ApolloCar3D [apollo] 3D part
HoliCity [zhou2020holicity] 3D CAD
UrbanScene3D 3D CAD/Detailed
Table 2: Features of the data sources of different datasets. Compared to existing datasets, UrbanScene3D has both CAD and detailed models, as well as the corresponding 2D/3D original data.

The released zip file on our website contains the unreal project of the above proposed urban scenes. Users can use either pure Unreal Engine or AirSim client (both in C++ or python) to capture their desired data. The ground truth textured meshes and their relevant poses are also provided in the Unreal project.

Required packages:

- Unreal Engine 4 (4.24 is recommended)

- C++ or Python

- AirSim (Optional)

Advantages and Applications

UrbanScene3D not only has CAD models that contain sharp edges and compact primitive structures for man-made virtual environments, but also has reconstructed mesh models with detailed, realistic features for real-world urban scenes. Please refer to the videos shown at our project page https://vcc.tech/UrbanScene3D. We build the touring in different scenes to demonstrate their corresponding CAD/Detailed models.

In addition, UrbanScene3D also releases the raw acquisition data, including high resolution aerial images (6000x4000) and aerial videos (4K) together along with their poses and detailed reconstructed 3D models. Check out the video samples shown below. These data can be largely used to train and test algorithms like SLAM, MVS or single view reconstruction in the wild.

Moreover, based on the UrbanScene3D simulator, users can design autonomous driving/flying in various environments. Below we demonstrate some popular applications via our simulator.

Download (70G)

Note that this dataset is free for Research and Education Use ONLY. Please cite this document if you use any part of our DATA in any publication.

- Download via FTP

- Download via HTTP

Acknowledgment

This work was supported in parts by NSFC (U2001206), Guangdong Talent Program (2019JC05X328), Guangdong Science and Technology Program (2020A0505100064, 2015A030312015), DEGP Key Project (2018KZDXM058), and Shenzhen Science and Technology Program (RCJC20200714114435012).

References