Log In Sign Up

TorontoCity: Seeing the World with a Million Eyes

In this paper we introduce the TorontoCity benchmark, which covers the full greater Toronto area (GTA) with 712.5 km^2 of land, 8439 km of road and around 400,000 buildings. Our benchmark provides different perspectives of the world captured from airplanes, drones and cars driving around the city. Manually labeling such a large scale dataset is infeasible. Instead, we propose to utilize different sources of high-precision maps to create our ground truth. Towards this goal, we develop algorithms that allow us to align all data sources with the maps while requiring minimal human supervision. We have designed a wide variety of tasks including building height estimation (reconstruction), road centerline and curb extraction, building instance segmentation, building contour extraction (reorganization), semantic labeling and scene type classification (recognition). Our pilot study shows that most of these tasks are still difficult for modern convolutional neural networks.


page 2

page 3

page 4

page 5

page 6

page 7


UrbanScene3D: A Large Scale Urban Scene Dataset and Simulator

The ability to perceive the environments in different ways is essential ...

PP-LinkNet: Improving Semantic Segmentation of High Resolution Satellite Imagery with Multi-stage Training

Road network and building footprint extraction is essential for many app...

BuildingNet: Learning to Label 3D Buildings

We introduce BuildingNet: (a) a large-scale dataset of 3D building model...

Continental-Scale Building Detection from High Resolution Satellite Imagery

Identifying the locations and footprints of buildings is vital for many ...

Mapping Road Lanes Using Laser Remission and Deep Neural Networks

We propose the use of deep neural networks (DNN) for solving the problem...

CBHE: Corner-based Building Height Estimation for Complex Street Scene Images

Building height estimation is important in many applications such as 3D ...

Combining Maps and Street Level Images for Building Height and Facade Estimation

We propose a method that integrates two widely available data sources, b...

1 Introduction

”It is a narrow mind which cannot look at a subject from various points of view.”

George Eliot, Middlemarch

In recent times, a great deal of effort has been devoted to creating large scale benchmarks. These have been instrumental to the development of the field, and have enabled many significant break-throughs. ImageNet


made it possible to train large-scale convolutional neural networks, initiating the deep learning revolution in computer vision in 2012 with SuperVision (most commonly refer as AlexNet

[16]). Efforts such as PASCAL [11] and Microsoft COCO [18] have pushed the performance of segmentation and object detection approaches to previously inconceivable levels. Similarly, benchmarks such as KITTI [12] and Cityscapes [8] have shown that visual perception is going to be an important component of advanced driver assistance systems (ADAS) and self-driving cars in the imminent future.

Data Maps Tasks  
[width=0.196trim=0 0 .30pt .10pt,clip]fig/task/ZoomIn_OverlayBuilding.pdf [width=0.196trim=0 0 .30pt .10pt,clip]fig/task/ZoomIn_OverlayHeight.pdf [width=0.196trim=0 0 .30pt .10pt,clip]fig/task/ZoomIn_OverlayInstance.pdf [width=0.196trim=0 0 .30pt .10pt,clip]fig/task/ZoomIn_OverlayRoad.pdf [width=0.196trim=0 0 .30pt .10pt,clip]fig/task/ZoomIn_OverlayZoning.pdf    
Figure 1: Summary of the TorontoCity benchmark. Data source: aerial RGB image, streetview panorama, GoPro, stereo image, street-view LIDAR, airborne LIDAR; Maps: buildings and roads, 3D buildings, property meta-data; Tasks: semantic segmentation, building height estimation, instance segmentation, road topology, zoning segmentation and classification.

However, current large scale datasets suffer from two shortcomings. First, they have been captured by a small set of sensors with similar perspectives of the world, e.g., internet photos for ImageNet or cameras/LIDAR mounted on top of a car in the case of KITTI. Second, they do not contain rich semantics and 3D information at a large-scale. We refer the reader to Fig. 2 for an analysis of existing datasets.

In this paper, we argue that the field is in need of large scale benchmarks that allow joint reasoning about geometry, grouping and semantics. This has been commonly referred to as the three R’s of computer vision. Towards this goal, we have created the TorontoCity benchmark, covering the full greater Toronto area (GTA) with of land, of road and around buildings. According to the census, people live in the GTA, which is around of the population of Canada. We have gathered a wide range of views of the city: from the overhead perspective, we have aerial images captured during four different years as well as LIDAR from airborne. From the ground, we have HD panoramas as well as stereo, Velodyne LIDAR and Go-pro data captured from a moving vehicle driving around in the city. We are also augmenting the dataset with a 3D camera as well as imagery captured from drones.

Manually labeling such a large scale dataset is not feasible. Instead, we propose to utilize different sources of high-precision maps to create our ground truth. Compared to online map services such as OpenStreetMap [1] and Google Maps, our maps are much more accurate and contain richer meta-data, which we exploit to create a wide variety of diverse benchmarks. This includes tasks such as building height estimation (reconstruction), road centerline and curb extraction, building instance segmentation, building contour extraction (reorganization), semantic labeling and scene type classfication (recognition). Participants can exploit any subset of the data (e.g., aerial and ground images) to solve these tasks.

One of the main challenges in creating TorontoCity was aligning the maps to all data sources such that the maps can produce accurate ground truth. While the aerial data was perfectly aligned, this is not the case of the panoramas where geolocalization is fairly noisy. To alleviate this problem, we have created a set of tools which allow us to reduce the labeling task to a simple verification process, speeding up labeling, thus making TorontoCity possible.

We perform a pilot study using the aerial images captured in 2011 as well as the ground panoramas. Our experiments show that state-of-the-art methods work well on some tasks, such as semantic segmentation and scene classification. However, tasks such as instance segmentation, contour extraction and height estimation remain an open challenge. We believe our benchmark provides a great platform for developing and evaluating new ideas, particularly techniques that can leverage different viewpoints of the world. We plan to extend the current set of benchmarks in the near future with tasks such as building reconstruction, facade parsing, tree detection and tree species classification as well as traffic lights and traffic sign detection, for which our maps provide accurate ground truth. We have only scratched the surface of TorontoCity’s full potential.

2 Related Work

Figure 2: Statistics of our data and comparison of current state-of-the-art urban benchmarks and datasets.

Automatic mapping, reconstruction and semantic labeling from urban scenes have been an important topic for many decades. Several benchmarks have been proposed to tackle subsets of these tasks. KITTI [12] is composed of stereo images and LIDAR data collected from a moving vehicle, and evaluates SLAM, optical flow, stereo and road segmentation tasks. Cityscapes [8] focuses on semantic and instance annotations of images captured from a car. Aerial-KITTI [21] augments the KITTI dataset with aerial imagery of a subset of Karlsruhe to encourage reasoning of semantics from both ground and bird’s eye view.

The photometry community has developed several benchmarks towards urban scene understanding [15, 24, 30, 25, 20]. TUM-DLR [15] and ISPRS Multi-Platform [24] benchmarks contain imagery captured through multiple perspectives from UAV, satellite images and handheld cameras. Oxford RobotCar contains lidar point cloud and stereo images captured from a vehicle [20]. However, these benchmarks do not offer any semantic ground-truth for benchmarking purposes. Perhaps the most closely related dataset to ours is the ISPRS Urban classification and building reconstruction benchmark [30], where the task is to extract urban object, such as building, road and trees from both aerial images and airborne laserscanner point clouds. However, this dataset has a relatively small coverage and does not provide ground-view imagery. In contrast, TorontoCity is more than two orders of magnitude bigger. Furthermore, we offer many different perspectives through various sensors, along with diverse semantic and geometric benchmarks with accurate ground-truth. The readers may refer to Fig. 2 for a detailed comparison against previous datasets.

A popular alternative is to use synthetic data to generate large scale benchmarks [3, 5, 26, 29, 31, 13, 6, 28]. Through 3D synthetic scenes and photo-realistic renderers large-scale datasets can be easily created. To date, however, these datasets have been focused on a single view of the world. This contrasts TorontoCity. Unlike other benchmarks, our input is real-world imagery, and the large-scale 3D models are a high-fidelity modeling of the real world rather than a synthetic scene.

Maps have been proven useful for many computer vision and robotics applications [34, 23, 22, 35, 21], including vehicle detection and pose estimation [23], semantic labeling and monocular depth estimation [34] as well as HD-map extraction [21]. However, there has been a lack of literature that exploit maps as ground-truth to build benchmarks. This is mainly due to both the lack of high-fidelity maps to provide pixel-level annotation and the lack of accurately georeferenced imagery that aligned well with the maps. One exception is [35], where the streetree catalog is used to generate ground-truth for tree detection. [36] utilizes 3D building models to generate correspondences from multiple streetview images. In this paper, we use maps to create multiple benchmarks for reconstruction, recognition and reorganization from many different views of the world.

Figure 3: Road surface generation: (left) input data with curbs (yellow) and center lines (red). Extracted road surface is the union of polygons shown in blue and black. Note that a formulation ensuring connectivity is needed, otherwise the road surface would contain holes at intersections.
(a) NCC: before vs. after
(b) Overlay: before vs. after
(c) Location: before vs. after
Figure 4: Ground-aerial alignment

[width=trim=0 0 .50pt .60pt,clip]fig/final_viz/623000_4833500.pdf

[width=trim=0 .20pt 0 0,clip]fig/final_viz/626000_4834500.pdf

(a) Input

[width=trim=0 0 .50pt .60pt,clip]fig/final_viz/623000_4833500_gt.pdf

[width=trim=0 .20pt 0 0,clip]fig/final_viz/626000_4834500_gt.pdf

(b) GT

[width=trim=0 0 .50pt .60pt,clip]fig/final_viz/623000_4833500_fcn_opening_val.pdf

[width=trim=0 .20pt 0 0,clip]fig/final_viz/626000_4834500_resnet_opening_val.pdf

(c) ResNet56

[width=trim=0 .20pt 0 0,clip]fig/final_viz/623000_4833500.pdf

[width=trim=0 .20pt 0 0,clip]fig/final_viz/629500_4834500.pdf

(d) Input

[width=trim=0 .20pt 0 0,clip]fig/final_viz/623000_4833500_gt.pdf

[width=trim=0 .20pt 0 0,clip]fig/final_viz/629500_4834500_gt.pdf

(e) GT

[width=trim=0 .20pt 0 0,clip]fig/final_viz/623000_4833500_resnet_opening_val.pdf

[width=trim=0 .20pt 0 0,clip]fig/final_viz/629500_4834500_resnet_opening_val.pdf

(f) ResNet56
Figure 5: Examples of aerial semantic segmentation, road curb extraction, and road centerline estimation.

3 TorontoCity at a Glimpse

TorontoCity is an extremely large dataset enabling work on many exciting new tasks. We first describe the data in detail. In the next section we describe our efforts to simplify the labeling task, as otherwise it is infeasible to create such a large-scale dataset. We then show the challenges and metrics that will compose the benchmark. Finally, we perform a pilot study of how current algorithms perform on most tasks, and analyze the remaining challenges.

3.1 Dataset

Toronto is the largest city in Canada, and the fourth largest in North America. The TorontoCity dataset covers the greater Toronto area (GTA), which contains of land, of road and around buildings. According to the census people live in the GTA, which is around of the population of Canada.

We have gathered a wide range of views of the city: from the overhead perspective, we have aerial images captured during four different years (containing several seasons) as well as airborne LIDAR. From the ground perspective, we have HD panoramas as well as stereo, Velodyne LIDAR and Go-pro data captured from a moving vehicle driving around the city. In addition, we are augmenting the dataset with a 3D camera as well as imagery captured from drones. Fig. 1 depicts some of the data sources that compose our dataset. We now describe the data in more details and refer the reader to Fig. 2 for a comparison against existing datasets.

[width=trim= .50pt .60pt 0 0,clip]fig/instance_viz/629500_4834500.pdf

[width=trim=.250pt .20pt 0 .20pt,clip]fig/instance_viz/623000_4833500.pdf

(a) Input

[width=trim= .50pt .60pt 0 0 ,clip]fig/instance_viz/629500_4834500_gt_instance.pdf

[width=trim=.250pt .20pt 0 .20pt,clip]fig/instance_viz/623000_4833500_gt_instance.pdf

(b) GT

[width=trim= .50pt .60pt 0 0 ,clip]fig/instance_viz/629500_4834500_resnet_opening_val_instance.pdf

[width=trim=.250pt .20pt 0 .20pt,clip]fig/instance_viz/623000_4833500_resnet_opening_val_instance.pdf

(c) ResNet56

[width=trim= 0 .60pt .50pt 0,clip]fig/instance_viz/623000_4833500.pdf

[width=trim=0 0 .250pt .40pt,clip]fig/instance_viz/629500_4834500.pdf

(d) Input

[width=trim=0 .60pt .50pt 0,clip]fig/instance_viz/623000_4833500_gt_instance.pdf

[width=trim=0 0 .250pt .40pt,clip]fig/instance_viz/629500_4834500_gt_instance.pdf

(e) GT

[width=trim=0 .60pt .50pt 0,clip]fig/instance_viz/623000_4833500_resnet_opening_val_instance.pdf

[width=trim=0 0 .250pt .40pt ,clip]fig/instance_viz/629500_4834500_resnet_opening_val_instance.pdf

(f) ResNet56
Figure 6: Examples of building instance segmentation.


We downloaded Google Streetview panoramas [2] that densely populate the GTA. On average, we crawled around 520 full spherical panoramas for each . In addition, we crawled the associated metadata, including the geolocation, address and the parameters of the spherical projection, including pitch, yaw and tilt angles. We resized all panoramas to pixels.

Aerial Imagery:

We use aerial images with full coverage of the GTA taken in 2009, 2011, 2012 and 2013. They are orthorectified to /pixel resolution for 2009 and 2011, and and /pixel for 2012 and 2013 respectively. This contrasts satellite images, which are at best /pixel. Our aerial images have four channels (i.e., RGB and Near infrared), and are 16 bit resolution for 2011 and 8 bit for the rest. As is common practice in remote sensing [4], we projected each image to the Universal Transverse Mercator (UTM) 17 zone in the WGS84 geodetic datum and tiled the area to images without overlap. Note that the images are not true orthophotos and thus facades are visible.

Airborne LIDAR:

We also exploit airborne LIDAR data captured in 2008 with a Leica ALS sensor with a resolution of 6.8 points per . The total coverage is 22 . All of the points are also geo-referenced and projected to the UTM17 Zone in WGS84 geodetic datum.

Car setting:

Our recording platform includes a GoPro Hero 4 RGB camera, a Velodyne HDL-64E LIDAR and a PointGray Bumblebee3 Stereo Camera mounted on top of the vehicle. All the sensors are calibrated and synchronized with a Applanix POS LV positioning system to record real-time geo-location and orientation information. We have driven this platform for around 90km, which includes repeats of the same area. Note that we are collecting and aligning new data from ground-view vehicles, and plan to have a much larger coverage by the time of release.

3.2 Maps as Annotations

Manually labeling such a large scale dataset as TorontoCity is simply not possible. Instead, in this paper we exploit different sources of high-precision maps covering the whole GTA to create our ground truth. Compared to online map services such as OpenStreetMap [1] and Google Maps, our maps are much more accurate. Furthermore, they contain many additional sources of detailed meta data which we exploit. One of the main challenges in creating TorontoCity was the alignment of the maps to all data sources. In the following, we first describe the annotated data composing TorontoCity and postpone our discussion on the algorithms we developed to align all data sources to the next section.


The TorontoCity dataset contains 3D buildings covering the full GTA. As shown in Fig. 2, the buildings are very diverse, with the tallest being the CN Tower with 443m of elevation. Toronto contains many individual family houses, which makes tasks such as instance level segmentation particularly difficult. The mean height of each building is 4.7m, and the mean building area is . In contrast, the largest building has an area . The level of detail of the 3D models varies per building (see Fig. 1). Many of these buildings are accurate to within centimeters and contain many other semantic primitives such as roof types, windows and balconies.


Our maps contain very accurate polylines representing streets, sidewalks, rivers and railways within the GTA. Each line segment is described with a series of attributes such as name, road category and address number range. Road intersections are explicitly encoded as intersecting points between polylines. Road curbs are also available, and describe the shape of roads (see Fig. 1).

Urban Zoning:

Our maps contain government zoning information on the division of land into categories. This zoning includes categories such as residential, commercial, industrial and institutional. Note that multiple categories are allowed for one zone, e.g., commercial+residential. Understanding urban zoning is important in applications such as urban planning, real estate and law-enforcement.

Additional data:

We also have cartographic information with full coverage of the GTA. For instance, we have the location of all the poles, traffic lights, street lights and street trees with meta-information for each. The meta-information includes the height of the pole/traffic light, type of model of each street lights, trunk radius and species of each trees. We plan to exploit this in the near future.

Figure 7: Qualitative results on building structured contour prediction: ResNet vs GT

4 Maps for Creating Large Scale Benchmarks

In this section we describe our algorithms to automatically align maps with our sources of imagery. We then describe the alignment of the different road maps.

4.1 Aligning Maps with All Data Sources

The aerial images we employed are already perfectly aligned with the maps. This, however, is not the case for the panoramas. As noted in [7], the geolocalizaiton error is up to with an average of , while rotation is very accurate. As a consequence, projecting our maps will not generate good ground truth due to the large misalignments as shown in Fig. 4. To handle this issue, we design an alignment algorithm that exploits both aerial images and maps. Their information is complementary, as aerial images give us appearance, while maps give us sparse structures (e.g., road curves).

For this, we first rectify the panoramas by projecting them onto the ground-plane. We extract a m ground plane region with

cm/pixel resolution and parameterize the alignment with three degrees of freedom representing the camera’s offset. We then perform a two step alignment process. We obtain a coarse alignment by maximizing a scoring function that compromises between appearance matching and a regularizer. In particular, we use normalized cross correlation (NCC) as our appearance matching term and a Gaussian prior with mean

and diagonal covariance . We re-scale both aerial and ground images to before NCC. The solution space is a discrete search window in the range with a step of . We use exhaustive search to perform this search, and exploit the fact that NCC can be computed efficiently using FFT and the Gaussian prior score is a fixed look-up-table. As shown in Fig. 4 this procedure produces very good coarse alignments. The alignment is coarse as we reason at the aerial images’ resolution, which is relatively lower.

Our fine alignment then utilizes the road curves and aligns them to the boundary edges [10] in the panorama. We use a search area of with a step of 5cm. This is followed by a human verification process that selects the images where this alignment succeeds. Mistakes in the alignment are due to occlusions (e.g., cars in the panoramas) as well as significant non-flat terrain. Our success rate is 34.35%, and it takes less than 2s to verify an image. In contrast annotating the alignment takes 20s.

Figure 8: Examples of road segmentation. Left: panoramic view; right: top-down view. (TP: yellow, FP: red, FN: green)
Method Mean Building Road
FCN [19] 77.64% 70.44% 73.32%
ResNet [14] 78.46% 69.15% 76.44%
Table 1: Aerial image semantic segmentation IoU.
Method WeightedCov AP Re-50% Pr-50%
FCN 39.74% 8.04% 19.64% 18.38%
FCN + Open 43.19% 16.45% 24.55% 36.09%
ResNet 38.70% 10.47% 21.30% 21.93%
ResNet + Open 41.10% 22.92% 22.78% 43.78%
Table 2: Building instance segmentation IoU.

4.2 Semantic Segmentation from Polyline Data

Our maps provide two types of road structures: curbs defining the road boundaries as well as center lines defining the connectivity (adjacency) in the street network. Unfortunately, these two sources are not aligned, and occasionally center lines are outside the road area. In this section we show our procedure of exploiting a Markov random field (MRF) to align road centerlines and curves. We can then generate the polygons describing the road surfaces. Fig. 3 shows an example for the road surface generation.

Let be the assignment of the -th curb segment to one of the nearest centerline segments, where state denotes no match. We define an MRF composed of unary and pairwise terms, which connects only adjacent curbs segments, and thus naturally form a set of chains. For the unary terms , we use the weighted sum of the distance of the curve to each centerline segment (condition on the state) and the angular distance between curves and centerlines. For the pairwise terms , we employ a Potts potential that encourages smoothness along the road. This is important as otherwise there may be holes in places such as intersections, since the center of the intersection is further away from other points. Due to the chain structure of the graphical model, inference can be done exactly and efficiently in parallel for each chain using dynamic programming. Our formulation allows for multiple curbs to be matched to one road, which is needed as there are curbs on both sides of the centerline. We manually inspect the results and mark errors as “don’t care” regions. We convert each continuous curb-road center line assignment to polygons which gives us the final road surface. We refer the reader to Fig. 3 for an example.

Road centerline Road curb
Method F1 Pr Re F1 Pr Re F1 Pr Re F1 Pr Re
FCN 0.169 0.156 0.186 0.626 0.576 0.687 0.444 0.413 0.482 0.778 0.726 0.837
FCN+Close 0.173 0.164 0.183 0.639 0.604 0.678 0.444 0.427 0.462 0.781 0.752 0.812
ResNet 0.162 0.143 0.186 0.613 0.567 0.667 0.575 0.585 0.566 0.796 0.830 0.765
ResNet+Close 0.162 0.169 0.155 0.644 0.671 0.619 0.568 0.614 0.529 0.799 0.862 0.745
Table 3: Road centerline and curb results. Metric: F1, Precision, Recall with minimal distance threshold 0.5m and 2m.

5 Benchmark Tasks and Metrics

We designed a diverse set of benchmarks to push computer vision approaches to reason about geometry, semantics and grouping. To our knowledge, no previous dataset is able to do this at this scale. In the evaluation server, participants can submit results using any subset of the imagery types provided in the benchmark (e.g., aerial images, panoramas, Go-Pro, stereo). In this section, we briefly describe the tasks and metrics, and refer the reader to the supplementary material for further details. Note also that Fig. 1 shows an illustration of some of our tasks.

Building Footprint and Road segmentation:

Our first task is semantic segmentation of building footprints and roads. Following common practice in semantic segmentation, we utilize mean Intersection-Over-Union (mIOU) as our metric. This is evaluated from a top-down view.

Building Footprint Instance Segmentation:

Our second task is building instance segmentation. We adopt multiple metrics for this task, since there is no consensus in the community of what is the best metric. We thus evaluate weighted coverage (Cov), average precision (AP) as well as instance level precision and recall at


Building Structured Contours:

Most semantic and instance segmentation algorithms produce ”blob”-like results, which do not follow the geometry of the roads and/or buildings. We thus want to push the community to produce instance segmentations that follow the structure of the primitives. Towards this goal, we define a metric that merges (in a multiplicative fashion) segmentation scoring with geometric similarity. In particular, segmentation is measured in terms of IOU, and we exploit the similarity between the turning functions of the estimated and ground truth polygons as a geometric metric. We refer the reader to the supplementary material for more details.

Road Topology:

One of the remaining fundamental challenges in mapping is estimating road topology. In this task, participants are asked to extract polylines that represent road curbs and road centerlines in bird’s eye perspective. We discretize both estimated and ground truth polylines in intervals of size cm. We define precision and recall as our metrics, where an estimated segment is correct if its distance to the closest segment on the target polyline set is smaller than a threshold (i.e., 0.5m and 2.0m).

Method AlexNet [16] VGG-16 [32] GoogleNet [33] ResNet-152 [14] AlexNet [16] ResNet-32 [14] GoogleNet [33] NiN [17]
From-scratch no no no no yes yes yes yes
Top-1 accuracy 75.49% 79.12% 77.95% 79.33% 66.48% 75.65% 75.08% 79.07%
Table 4: Ground-Level Urban Zoning Classification

Ground Road Segmentation:

We use IOU as our metric.

Ground Urban Zoning Classification:

This benchmark is motivated by the human’s ability to recognize the urban function of a local region by its appearance. We use Top-1 accuracy as our metric and evaluate on the ground view.

Urban Zoning Segmentation:

Our goal is to produce a segmentation in bird’s eye view of the different urban zones including residential, commercial, open space, employment, etc. We utilize IOU as our metric.

Building Height Estimation:

This tasks consists on estimating building height. Useful cues include size of the buildings, pattern of shading and shadows as well as the imperfect rectification in aerial views. We adopt root mean square error in the log domain (log-RMSE) as our metric.

Additional Tasks:

We plan to add many tasks in the coming months. This includes detecting trees and recognizing their species. Moreover, the accurate 3D building models allow us to build a benchmark of normal estimation as well as facade parsing. We also plan to have benchmarks for detection and segmentation of traffic lights, traffic signs and poles. We are just scratching the surface of the plenthora of possibilities with this dataset.

6 Experimental Evaluation

Method WeightedCov PolySim
FCN 0.456 0.323
ResNet 0.401 0.292
Table 5: Building contour results.
Method Residential Open Space Others
FCN 60.20% 32.20% 5.57%
ResNet 51.71% 33.63% 1.49%
Table 6: Qualitative results for urban zoning segmentation.

We perform a pilot study of the difficulty of our tasks in a subset of TorontoCity, containing 125 region (50 for training, 50 for testing and for validation). The train/val/test regions do not overlap and are not adjacent. We utilize 56K streetview images around these regions (22K for training, 18K for validation and 16K for testing). Hyper-parameters are chosen based on validation performance, and all numbers reported are on the testing set.

To perform the different segmentation related tasks, we train two types of convolutional networks: a variant of FCN-8 architecture [19] as well as a ResNet [14] with 56 convolutional layers. More details are in supp. material.

Semantic Segmentation:

As shown in Tab. 1, both networks perform well. Fig. 5 illustrates qualitative results of ResNet56 output. It is worth noting that large networks such as ResNet56 can be trained from scratch given our large-scale dataset. Visually ResNet’s output tends to be more sharp, while FCN’s output is more smooth.

Instance Segmentation:

We estimate instance segmentation by taking the output of the semantic labeling and performing connected-component labeling. Each component is assigned a different label. Since convolutional nets tend to generate blob like structures, a single component might contain multiple instances connected with a small number of pixels. To alleviate this problem, we apply morphological opening operators over the semantic labeling masks (an erosion filtering followed by a dilation filtering with the same size). As shown in Tab. 2 and Fig. 6 the performance is low. There is still much for the community to do to solve this task. With more than buildings, the TorontoCity dataset provides an ideal platform for new developments.

Road Centerlines and Curbs:

We compute the medial axis of the semantic segmentation to extract the skeleton of the mask as our estimate of road centerline. In order to smooth the skeletonization, we first conduct a morphological closing operator (dilation followed by erosion) over the road masks. To estimate road curbs, we simply extract the contours of the road segmentation and exploit closing operator. As shown in Table. 1, ResNet achieves the highest score in both tasks, and morphological filtering helps for both networks. Qualitative results are shown in Fig. 5. Note that there is still much room for improvement.

Building Contours:

We compute building contours from our estimated building instances, and apply the Ramer-Douglas-Peucker algorithm [27] to simplify each polygon with a threshold of m. This results in polygons with 13 vertices on average. As shown in Tab. 5 and Fig. 7, this simple procedure offers reasonable yet not satisfactory resulst. This suggests there is still a large improvement space for generating building polygons from aerial images.

Ground Urban Zoning Classification:

We train multiple state-of-the-art convolutional networks for this task including AlexNet [16], VGG-16 [32], GoogleNet [33] and ResNet-152 [14] that are fine-tuned from the ImageNet benchmark [9]. We also train AlexNet [16], ResNet-32 [14], Network-In-Network [17] and ResNet-152 [14] from scratch over our ground-view panoramic image tiles. As shown in Table. 1 ResNet-152 with pre-trained initialization achieves the best results. Net-in-net achieves the best performance among all models that are trained from scratch. For more details, please refer to the supplementary material.

Urban Zoning Segmentation:

This is an extremely hard task from aerial views alone. To simplify it, we merged the zone-types into residential, others (including commercial, utility and employment) as well as open spaces (including natural, park, recreational etc.). Please refer to the supplementary material for detailed label merging. As shown in Tab. 6 more research is needed to solve this task.

Ground-view road segmentation:

We utilize a subset of the labeled panoramas, which includes 1000 training, 200 validation and 800 testing images. The average IOU is . The average pixel accuracy is and average top-down IOU is . This shows that a state-of-the-art neural network can nearly solve this task, suggesting that it is promising to automatically generate high-resolution maps by capturing geo-referenced street-view panoramas.

Building Height:

No network was able to estimate building height from aerial images alone. This task is either too hard, or more sophisticated methods are needed. For example, utilizing ground imagery seems a logical first step.

7 Conclusions

In this paper, we have argued that the field is in need of large scale benchmarks that will allow joint reasoning about geometry, grouping and semantics. Towards this goal, we have created the TorontoCity benchmark, covering the full Greater Toronto area (GTA) with of land, of road and around buildings. Unlike existing datasets, our benchmark provides a wide variety of views of the world captured from airplanes, drones, as well as cars driving around the city. As using human annotators is not feasible for such a large-scale dataset, we have exploited different sources of high-precision maps to create our ground truth. We have designed a wide variety of tasks including building height estimation, road centerline and curb extraction, building instance segmentation, building contour extraction (reorganization), semantic labeling and scene type classification (recognition). Our pilot study shows that most of these tasks are still difficult for modern convolutional networks. We plan to extend the current set of benchmarks with tasks such as building reconstruction, facade parsing, tree detection, tree species categorization, traffic light detection, and traffic sign detection. This is only the beginning of the exciting TorontoCity benchmark.


  • [1] Openstreetmap.
  • [2] D. Anguelov, C. Dulong, D. Filip, C. Frueh, S. Lafon, R. Lyon, A. Ogale, L. Vincent, and J. Weaver. Google street view: Capturing the world at street level. IEEE Computer, 2010.
  • [3] S. Baker, D. Scharstein, J. Lewis, S. Roth, M. J. Black, and R. Szeliski. A database and evaluation methodology for optical flow. IJCV, 2011.
  • [4] W. Brooks. The universal transverse mercator grid. In Proceedings of the Indiana Academy of Science, 1973.
  • [5] D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black. A naturalistic open source movie for optical flow evaluation. In ECCV, 2012.
  • [6] C. Chen, A. Seff, A. Kornhauser, and J. Xiao. Deepdriving: Learning affordance for direct perception in autonomous driving. In ICCV, 2015.
  • [7] H. Chu, S. Wang, R. Urtasun, and S. Fidler. Housecraft: Building houses from rental ads and street views. In ECCV, 2016.
  • [8] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
  • [9] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
  • [10] P. Dollár and C. L. Zitnick. Structured forests for fast edge detection. In ICCV, 2013.
  • [11] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. IJCV, 2010.
  • [12] A. Geiger, P. Lenz, and R. Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR, 2012.
  • [13] A. Handa, V. Patraucean, V. Badrinarayanan, S. Stent, and R. Cipolla. Scenenet: Understanding real world indoor scenes with synthetic data. arXiv, 2015.
  • [14] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. CVPR, 2016.
  • [15] T. Koch, P. d’Angelo, F. Kurz, F. Fraundorfer, P. Reinartz, and M. Korner. The tum-dlr multimodal earth observation evaluation benchmark. In CVPRW, 2016.
  • [16] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.
  • [17] M. Lin, Q. Chen, and S. Yan. Network in network. ICLR, 2014.
  • [18] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
  • [19] J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
  • [20] W. Maddern, G. Pascoe, C. Linegar, and P. Newman. 1 year, 1000km: The oxford robotcar dataset. IJRR, 2016.
  • [21] G. Máttyus, S. Wang, S. Fidler, and R. Urtasun. Enhancing road maps by parsing aerial images around the world. In ICCV, 2015.
  • [22] G. Máttyus, S. Wang, S. Fidler, and R. Urtasun. Hd maps: Fine-grained road segmentation by parsing ground and aerial images. In CVPR, 2016.
  • [23] K. Matzen and N. Snavely. Nyc3dcars: A dataset of 3d vehicles in geographic context. In ICCV, 2013.
  • [24] F. Nex, M. Gerke, F. Remondino, H. Przybilla, M. Bäumker, and A. Zurhorst. Isprs benchmark for multi-platform photogrammetry. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2015.
  • [25] J. Niemeyer, F. Rottensteiner, and U. Soergel. Contextual classification of lidar data and building object detection in urban areas. ISPRS journal of photogrammetry and remote sensing, 2014.
  • [26] N.Mayer, E.Ilg, P.Häusser, P.Fischer, D.Cremers, A.Dosovitskiy, and T.Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In CVPR, 2016.
  • [27] U. Ramer. An iterative procedure for the polygonal approximation of plane curves. Computer graphics and image processing, 1972.
  • [28] S. R. Richter, V. Vineet, S. Roth, and V. Koltun. Playing for data: Ground truth from computer games. In ECCV, 2016.
  • [29] G. Ros, L. Sellart, J. Materzynska, D. Vazquez, and A. M. Lopez. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In CVPR, 2016.
  • [30] F. Rottensteiner, G. Sohn, M. Gerke, and J. D. Wegner. Isprs test project on urban classification and 3d building reconstruction. 2013.
  • [31] A. Shafaei, J. J. Little, and M. Schmidt. Play and learn: using video games to train computer vision models. arXiv, 2016.
  • [32] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv, 2014.
  • [33] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In CVPR, 2015.
  • [34] S. Wang, S. Fidler, and R. Urtasun. Holistic 3d scene understanding from a single geo-tagged image. In CVPR, 2015.
  • [35] J. D. Wegner, S. Branson, D. Hall, K. Schindler, and P. Perona. Cataloging public objects using aerial and street-level images-urban trees. In CVPR, 2016.
  • [36] A. R. Zamir, T. Wekel, P. Agrawal, C. Wei, J. Malik, and S. Savarese. Generic 3d representation via pose estimation and matching. In ECCV, 2016.