The rise of multi-million-item dataset initiatives has enabled data-hungry machine learning algorithms to reach near-human semantic classification at tasks such as object and scene recognition. Here we describe the Places Database, a repository of 10 million scene photographs, labeled with scene semantic categories and attributes, comprising a quasi-exhaustive list of the types of environments encountered in the world. Using state of the art Convolutional Neural Networks, we provide impressive baseline performances at scene classification. With its high-coverage and high-diversity of exemplars, the Places Database offers an ecosystem to guide future progress on currently intractable visual recognition problems.READ FULL TEXT VIEW PDF
Identifying potential abuses of human rights through imagery is a novel ...
With the success of new computational architectures for visual processin...
While there has been remarkable progress in the performance of visual
Convolutional Neural Networks (CNNs) have made remarkable progress on sc...
We present a Visual Place Recognition system that follows the two-stage
In this paper, we present a study on learning visual recognition models ...
The Machine Recognition of Crystallization Outcomes (MARCO) initiative h...
What does it take to reach human-level performance with a machine-learning algorithm? In the case of supervised learning, the problem is two-fold. First, the algorithm must be suitable for the task, such as pattern classification in the case of object recognition[1, 2], pattern localization for object detection 
or the necessity of temporal connections between different memory units for natural language processing[4, 5]. Second, it must have access to a training dataset of appropriate coverage (quasi-exhaustive representation of classes and variety of examplars) and density (enough samples to cover the diversity of each class). The optimal space for these datasets is often task-dependent, but the rise of multi-million-item sets has enabled unprecedented performance in many domains of artificial intelligence.
The successes of Deep Blue in chess, Watson in “Jeopardy!”, and AlphaGo in Go against their expert human opponents may thus be seen as not just advances in algorithms, but the increasing availability of very large datasets: 700,000, 8.6 million, and 30 million items, respectively [6, 7, 8]. Convolutional Neural Networks [1, 9] have likewise achieved near human-level visual recognition, trained on 1.2 million object [10, 11, 12] and 2.5 million scene images . Expansive coverage of the space of classes and samples allows getting closer to the right ecosystem of data that a natural system, like a human, would experience.
Here we describe the Places Database, a quasi-exhaustive repository of 10 million scene photographs, labeled with 476 scene semantic categories and attributes, comprising the types of visual environments encountered in the world. Image samples are shown in Fig. 1. In the context of Places, we explain the steps to create high-quality datasets enabling the remarkable feats of machine-learning algorithms.
The primary asset of a high-quality dataset is an expansive coverage of the categorical space we want to learn. The strategy of Places is to provide an exhaustive list of the categories of environments encountered in the world, bounded by spaces where a human body would fit (e.g. closet, shower). The SUN (Scene UNderstanding) dataset  provided that initial list of semantic categories. The SUN dataset was built around a quasi-exhaustive list of scene categories with different functionalities, namely categories with unique identities in discourse. Through the use of WordNet , the SUN database team selected 70,000 words and concrete terms that described scenes, places and environments that can be used to complete the phrase “I am in a place”, or “let’s go to the/a place”. Most of the words referred to basic and entry-level names (), resulting in a corpus of 900 different scene categories after bundling together synonyms, and separating classes described by the same word but referring to different environments (e.g. inside and outside views of churches). Details about the building of that initial corpus can be found in . Places Database has inherited the same list of scene categories from the SUN dataset.
From online image search engines (Google Images, Bing Images, and Flickr), candidate images were downloaded using a query word from the list of scene classes provided by the SUN database . In order to increase the diversity of visual appearances in the Places dataset (see Fig. 2), each scene class query was combined with 696 common English adjectives (e.g., messy, spare, sunny, desolate, etc.). About 60 million images (color images of at least 200200 pixels size) with unique URLs were identified. Importantly, the Places and SUN datasets are complementary: PCA-based duplicate removal was conducted within each scene category in both databases so that they do not contain the same images.
Image ground truth label verification was done by crowdsourcing the task to Amazon Mechanical Turk (AMT). Fig.3 illustrates the experimental paradigm used: AMT workers were each given instructions relating to a particular image category at a time (e.g. cliff), with a definition and samples of true and false images. Workers then performed a go/no-go categorical task (Fig.3). The experimental interface displayed a central image, flanked by smaller version of images the worker had just responded to, on the left, and will respond to next, on the right. Information gleaned from construction of the SUN dataset suggests that the first iteration of labeling will show that more than 50% of the the downloaded images are not true exemplars of the category. As illustrated in Fig.3, the default answer is set to No (see images with bold red contours), so the worker can more easily press the space bar to move the majority of No images forward. Whenever a true category exemplar appears in the center, the worker can press a specific key to mark it as a positive exemplar (responding yes to the question: “is this a place term
”). Reaction time from the moment the image is centrally placed to the space bar or key press is recorded. The interface also allows moving backwards to revise previous annotations. Each AMT HIT (Human Intelligence Task, one assignment for one worker), consisted of 750 images for manual annotation. A control set of 30 positive samples and 30 negative samples with ground-truth category labels from the SUN database were intermixed in the HIT as well. Only worker HITs with an accuracy of 90% or higher on these control images were kept.
The positive images resulting from the first cleaning iteration were sent for a second iteration of cleaning. We used the same task interface but with the default answer set to Yes. In this second iteration, 25.4% of the images were relabeled as No. We tested a third iteration on a few exemplars but did not pursue it further as the percentage of images relabeled as No was not significant.
After the two iterations of annotation, we collected one scene label for 7,076,580 images pertaining to 476 scene categories. As expected, the number of images per scene category vary greatly (i.e. there are many more images of bedroom than cave on the web). There were 413 scene categories that ended up with at least 1000 exemplars, and 98 scene categories with more than 20,000 exemplars.
As a result of the previous round of image annotation, there were 53 million remaining downloaded images not assigned to any of the 476 scene categories (e.g. a bedroom picture could have been downloaded when querying images for living-room
category, but marked as negative by the AMT worker). Therefore, a third annotation task was designed to re-classify then re-annotate those images, using a semi-automatic bootstrapping approach.
A deep learning-based scene classifier, AlexNet , was trained to classify the remaining 53 million images: We first randomly selected 1,000 images per scene category as training set and 50 images as validation set (for the 413 categories which had more than 1000 samples). AlexNet achieved 32% scene classification accuracy on the validation set after training and was then used to classify the 53 million images. We used the predicted class score by the AlexNet to rank the images within one scene category as follow: for a given category with too few exemplars, the top ranked images with predicted class confidence higher than 0.8 were sent to AMT for a third round of manual annotation using the same interface shown in Fig.3. The default answer was set to No.
After completing the third round of AMT annotation, the distribution of the number of images per category flattened out: 401 scene categories had more than 5,000 images per category and 240 scene categories had more than 20,000 images. Totally there are about 3 million images added into the dataset.
Despite the initial effort to bundle synonyms from WordNet, the scene list from the SUN database still contained categories with very close synonyms (e.g. ‘ski lodge’ and ‘ski resort’, or ‘garbage dump’ and ‘landfill’). We identified 46 synonym pairs like these and merged their images into a single category.
Additionally, some scene categories are easily confused with blurry categorical boundaries, as illustrated in Fig. 5. This means that answering the question “Does image I belong to class A?” might be difficult. It is easier to answer the question “Does image I belong to class A or B?” In that case, the decision boundary becomes clearer for a human observer and it also gets closer to the final task that a computer system will be trained to solve.
Indeed, in the previous three steps of the AMT annotation, it became apparent that workers were confused with some pairs of scene categories, for instance, putting images of ‘canyon’ and ‘butte’ into ‘mountain’, or putting ‘jacuzzi’ into ‘swimming pool indoor’, mixing images of ‘pond’ and ’lake’, ‘volcano’ and ‘mountain’, ‘runway’ and ‘landing deck’, ‘highway and road’, ‘operating room’ and ‘hospital room’, etc. In the whole set of categories, we identified 53 such ambiguous pairs.
To further differentiate the images from the categories with shared content, we designed a new interface (Fig. 4) for a fourth step of annotation. We combined exemplar images from the two categories with shared content (such as art school and art studio), and asked the AMT workers to classify images into either of the categories or neither of them.
After the four steps of annotations, the Places database was finalized with over 10 millions labeled exemplars (10,624,928 images) from 434 place categories.
Scene-centric datasets correspond to images labeled with a scene, or place name, as opposed to an object name. Fig. 6
illustrates the differences among the number of images found in Places, ImageNet and SUN for a set of scene categories common to all three datasets. Places Database is the largest scene-centric image dataset so far.
Here we describe four subsets of Places as benchmarks. Places205 and Places88 are from . Two new benchmarks were added: from the 434 categories, we selected 365 categories with more than 4000 images each to create Places365-Standard and Places365-Challenge.
Places365-Standard has 1,803,460 training images with the image number per class varying from 3,068 to 5,000. The validation set has 50 images per class and the test set has 900 images per class. Note that the experiments in this paper are reported on Places365-Standard.
Places365-Challenge contains the same categories as Places365-Standard, but the training set is significantly larger with a total of 8 million training images. The validation set and testing set are the same as the Places365-Standard. This subset was released for the Places Challenge 2016111http://places2.csail.mit.edu/challenge.html
held in conjunction with the European Conference on Computer Vision (ECCV) 2016, as part of the ILSVRC Challenge.
Places205. Places205, described in , has 2.5 million images from 205 scene categories. The image number per class varies from 5,000 to 15,000. The training set has 2,448,873 total images, with 100 images per category for the validation set and 200 images per category for the test set.
Places88. Places88 contains the 88 common scene categories among the ImageNet , SUN  and Places205 databases. Note that Places88 contains only the images obtained in round 2 of annotations, from the first version of Places used in . We call the corresponding subsets Places88, ImageNet88 and SUN88. These subsets are used to compare performances across different scene-centric databases, as the three datasets contain different exemplars per category. Note that finding correspondences between the classes defined in ImageNet and Places brings some challenges. ImageNet follows the WordNet definitions, but some WordNet definitions are not always appropriate for describing places. For instance, the class ’elevator’ in ImageNet refers to an object. In Places, ’elevator’ takes different meanings depending on the location of the observer: elevator door, elevator interior, or elevator lobby. Many categories in ImageNet do not differentiate between indoor and outdoor (e.g., ice-skating rink) while in Places, indoor and outdoor versions are separated as they do not necessarily afford the same function.
Given the types of images found on the internet, some categories will be more biased than others in terms of viewpoints, types of objects, or even image style . However, bias can be compensated with a high diversity of images (with many appearances represented in the dataset). In the next section, we describe a measure of dataset diversity to compare how diverse images from three scene-centric datasets (Places88, SUN88 and ImageNet88) are.
Comparing datasets is an open problem. Even datasets covering the same visual classes have notable differences providing different generalization performances when used to train a classifier . Beyond the number of images and categories, there are aspects that are important but difficult to quantify, like the variability in camera poses, in decoration styles or in the type of objects that appear in the scene.
Although the quality of a database is often task dependent, it is reasonable to assume that a good database should be dense (with a high degree of data concentration), and diverse (it should include a high variability of appearances and viewpoints). Imagine, for instance, a dataset composed of 100,000 images all taken within the same bedroom. This dataset would have a very high density but a very low diversity as all the images will look very similar. An ideal dataset, expected to generalize well, should have high diversity
as well. While one can achieve high density by collecting a large number of images, diversity is not an obvious quantity to estimate in image sets, as it assumes some notion of similarity between images. One way to estimate similarity is to ask the questionare these two images similar? However, similarity in the wild is a subjective and loose concept, as two images can be viewed as similar if they contain similar objects, and/or have similar spatial configurations, and/or have similar decoration styles and so on. A way to circumvent this problem is to define relative measures of similarity for comparing datasets.
Several measures of diversity have been proposed, particularly in biology for characterizing the richness of an ecosystem (see  for a review). Here, we propose to use a measure inspired by the Simpson index of diversity 
. The Simpson index measures the probability that two random individuals from an ecosystem belong to the same species. It is a measure of how well distributed the individuals across different species are in an ecosystem, and it is related to the entropy of the distribution. Extending this measure for evaluating the diversity of images within a category is non-trivial if there are no annotations of sub-categories. For this reason, we propose to measure the relative diversity of image datasets A and B based on the following idea: if set A is more diverse than set B, then two random images from set B are more likely to be visually similar than two random samples from A. Then, the diversity of A with respect to B can be defined as, where and are randomly selected. With this definition of relative diversity we have that A is more diverse than B if, and only if, . For an arbitrary number of datasets, :
where are randomly selected.
We measured the relative diversities between SUN, ImageNet and Places using AMT. Workers were presented with different pairs of images and they had to select the pair that contained the most similar images. The pairs were randomly sampled from each database. Each trial was composed of 4 pairs from each database, giving a total of 12 pairs to choose from. We used 4 pairs per database to increase the chances of finding a similar pair and avoiding users having to skip trials. AMT workers had to select the most similar pair on each trial. We ran 40 trials per category and two observers per trial, for the 88 categories in common between ImageNet, SUN and Places databases. Fig. 7.a-b shows some examples of pairs from the diversity experiments for the scene categories playground (a) and bedroom (b). In the figure only one pair from each database is shown. We observed that different annotators were consistent in deciding whether a pair of images was more similar than another pair of images.
Fig. 7.c shows the histograms of relative diversity for all the 88 scene categories common to the three databases. If the three datasets were identical in terms of diversity, the average diversity should be 2/3 for the three datasets. Note that this measure of diversity is a relative measure between the three datasets. In the experiment, users selected pairs from the SUN database to be the closest to each other of the time, while the pairs from the Places database were judged to be the most similar only on of the trials. ImageNet pairs were selected of the time.
The results show that there is a large variation in terms of diversity among the three datasets, showing Places to be the most diverse of the three datasets. The average relative diversity on each dataset is for Places, for ImageNet and for SUN. To illustrate, the categories with the largest variation in diversity across the three datasets were playground, veranda and waiting room.
Given the impressive performance of the deep Convolutional Neural Networks (CNNs), particularly on the ImageNet benchmark [1, 12], we choose three popular CNN architectures, AlexNet , GoogLeNet , and VGG 16 convolutional-layer CNN , then train them on Places205 and Places365-Standard respectively to create baseline CNN models. The trained CNNs are named as PlacesSubset-CNN, i.e., Places205-AlexNet or Places365-VGG.
All the Places-CNNs presented here were trained using the Caffe package on Nvidia GPUs Tesla K40 and Titan X222All the Places-CNNs are available at https://github.com/metalbubble/places365. Additionally, given the recent breakthrough performances of the Residual Network (ResNet) on ImageNet classification , we further fine-tuned ResNet152 on the Places365-Standard (termed as Places365-ResNet) and compared it with the other trained-from-scratch Places-CNNs for scene classification.
After training the various Places-CNNs, we used the final output layer of each network to classify the test set images of Places205 and SUN205 (see ). The classification results for Top-1 accuracy and Top-5 accuracy are listed in Table I. As a baseline comparison, we show the results of a linear SVM trained on ImageNet-CNN features of 5000 images per category in Places205 and 50 images per category in SUN205 respectively.
Places-CNNs perform much better than the ImageNet feature+SVM baseline while, as expected, Places205-GoogLeNet and Places205-VGG outperformed Places205-AlexNet with a large margin due to their deeper structures. To date (Oct 2, 2016) the top ranked results on the test set of Places205 leaderboard333http://places.csail.mit.edu/user/leaderboard.php is 64.10% on Top-1 accuracy and 90.65% on Top-5 accuracy. Note that for the test set of SUN205, we didn’t fine-tune the Places-CNNs on the training set of SUN205, as we directly evaluated them on the test set of SUN.
|Test set of Places205||Test set of SUN205|
|Top-1 acc.||Top-5 acc.||Top-1 acc.||Top-5 acc.|
We further evaluated the baseline Places365-CNNs on the validation set and test set of Places365 shown in Fig.II. Places365-VGG and Places365-ResNet have similar top performances compared with the other two CNNs444The performance of the ResNet might result from fine-tuning or under-training, as the ResNet is not trained from scratch.. Even if Places365 has 160 more categories than Places205, the Top-5 accuracy of the Places205-CNNs (trained on the previous version of Places ) on the test set only drops by 2.5%.
Fig.8 shows the responses to examples correctly predicted by the Places365-VGG. Most of the Top-5 responses are very relevant to the scene description. Some failure or ambiguous cases are shown in Fig.9: Broadly, we can identify two kinds of misclassification given the current label attribution of Places: 1) less-typical activities happening in a scene, such as taking group photo in a construction site and camping in a junkyard; 2) images composed of multiple scene parts, which make one ground-truth scene label not sufficient to describe the whole environment. These illustrate the need to have multi-ground truth labels for describing environments.
It is important to emphasize that for many scene categories the Top-1 accuracy might be an ill-defined measure: environments are inherently multi-labels in terms of their semantic description. Different observers will use different terms to refer to the same place, or different parts of the same environment, and all the labels might fit well the description of the scene. This is obvious in the examples of Fig.9. Future development of the Places database, and the Places Challenge, will explore to assign multiple ground truth labels or free-form sentences to images to better capture the richness of visual descriptions inherent to environments.
|Validation Set of Places365||Test Set of Places365|
|Top-1 acc.||Top-5 acc.||Top-1 acc.||Top-5 acc.|
Based on the Places-CNN we trained, we created a web-demo for scene recognition555http://places.csail.mit.edu/demo.html, accessible through a computer browser or mobile phone. People can upload photos to the web-demo to predict the type of environment, with the 5 most likely semantic categories, and relevant scene attributes. Two screenshots of the prediction result on the mobile phone are shown in Fig.10. Note that people can submit feedback about the result. The top-5 recognition accuracy of our recognition web-demo in the wild is about 72% (from the 9,925 anonymous feedbacks dated from Oct.19, 2014 to May 5, 2016), which is impressive given that people uploaded all kinds of photos from real-life and not necessarily places-like photos (these results are for Places205-AlexNet as the back-end prediction model in the demo).
We further used the activation from the trained Places-CNNs as generic features for visual recognition tasks using different image classification benchmarks. Activations from the higher-level layers of a CNN, also termed deep features, have proven to be effective generic features with state-of-the-art performance on various image datasets [23, 24]. But most of the deep features are from the CNNs trained on ImageNet, which is mostly an object-centric dataset.
Here we evaluated the classification performances of the deep features from object-centric CNNs and scene-centric CNNs in a systematic way. The deep features from several Places-CNNs and ImageNet-CNNs on the following scene and object benchmarks are tested: SUN397 , MIT Indoor67 , Scene15 , SUN Attribute , Caltech101 , Caltech256 , Stanford Action40 , and UIUC Event8 .
All of the experiments follow the standards in those papers. In the SUN397 experiment , the training set size is 50 images per category. Experiments were run on 5 splits of the training set and test set given in the dataset. In the MIT Indoor67 experiment , the training set size is 100 images per category. The experiment is run on the split of the training set and test set given in the dataset. In the Scene15 experiment , the training set size is 50 images per category. Experiments are run on 10 random splits of the training set and test set. In the SUN Attribute experiment , the training set size is 150 images per attribute. The reported result is the average precision. The splits of the training set and test set are given in the paper. In Caltech101 and Caltech256 experiment [28, 29], the training set size is 30 images per category. The experiments are run on 10 random splits of the training set and test set. In the Stanford Action40 experiment , the training set size is 100 images per category. Experiments are run on 10 random splits of the training set and test set. The reported result is the classification accuracy. In the UIUC Event8 experiment , the training set size is 70 images per category and the test set size is 60 images per category. The experiments are run on 10 random splits of the training set and test set.
Places-CNNs and ImageNet-CNNs have the same network architectures for AlexNet, GoogLeNet, and VGG, but they are trained on scene-centric data and object-centric data respectively. For AlexNet and VGG, we used the 4096-dimensional feature vector from the activation of the Fully Connected Layer (fc7) of the CNN. For GoogLeNet, we used the 1024-dimensional feature vector from the response of the global average pooling layer before softmax producing the class predictions. The classifier in all of the experiments is a linear SVM with the default parameter for all of the features.
|Deep Feature||SUN397||MIT Indoor67||Scene15||SUN Attribute||Caltech101||Caltech256||Action40||Event8||Average|
Table III summarizes the classification accuracy on various datasets for the deep features of Places-CNNs and the deep features of the ImageNet-CNNs. Fig.11 plots the classification accuracy for different visual features on the SUN397 database over different numbers of training samples per category. The classifier is a linear SVM with the same default parameters for the two deep feature layers (C=1) . The Places-CNN features show impressive performance on scene-related datasets, outperforming the ImageNet-CNN features. On the other hand, the ImageNet-CNN features show better performance on object-related image datasets. Importantly, our comparison shows that Places-CNN and ImageNet-CNN have complementary strengths on scene-centric tasks and object-centric tasks, as expected from the type of the datasets used to train these networks. On the other hand, the deep features from the Places365-VGG achieve the best performance (63.24%) on the most challenging scene classification dataset SUN397, while the deep features of Places205-VGG performs the best on the MIT Indoor67 dataset. As far as we know, they are the state-of-the-art scores achieved by a single feature + linear SVM on those two datasets. Furthermore, we merge the 1000 classes from the ImageNet and the 365 classes from the Places365-Standard to train a VGG (Hybrid1365-VGG). The deep feature from the Hybrid1365-VGG achieves the best score averaged over all the eight image datasets.
Through the visualization of the units responses for various levels of network layers, we can have a better understanding of what has been learned inside CNNs and what are the differences between the object-centric CNN trained on ImageNet and the scene-centric CNN trained on Places given that they share the same architecture (here we use AlexNet). Following the methodology in ， we estimated the receptive fields of the units in the Places-CNN and ImageNet-CNN. Then we segmented the images with high unit activation using the estimated receptive fields. The image segmentation results by the receptive fields of units from different layers are shown in Fig.12. We can see that from pool1 to pool5, the units detect visual concepts from low-level edge/texture to high-level object/scene parts. Furthermore, in the object-centric ImageNet-CNN there are more units detecting object parts such as dog and people’s heads in the pool5 layer, while in the scene centric Places-CNN there are more units detecting scene parts such as bed, chair, or buildings in the pool5 layer.
Thus the specialty of the units in the object-centric CNN and scene-centric CNN yield very different performances of generic visual features on a variety of recognition benchmarks (object-centric datasets vs scene-centric datasets) in Table III.
We further synthesized preferred input images for the Places-CNN by using the image synthesis technique proposed in . This method uses a learned prior deep generator network to generate images which maximize the final class activation or the intermediate unit activation of the Places-CNN. The synthetic images for 50 scene categories are shown in Fig.13. These abstract image contents reveal the knowledge of the specific scene learned and memorized by the Places-CNN: examples include the buses within a road environment in the bus station, and the tents surrounded by forest-types of features for the campsite. Here we used Places365-AlexNet (other Places365-CNNs generated similar results). We further used the synthesis technique to generate the images preferred by the units in the pool5 layer of Places365-AlexNet. As shown in Fig.14, the synthesized images are very similar to the segmented image regions by the estimated receptive field of the units.
From the Tiny Image dataset , to ImageNet  and Places , the rise of multi-million-item dataset initiatives and other densely labeled datasets [36, 37, 38, 39] have enabled data-hungry machine learning algorithms to reach near-human semantic classification of visual patterns, like objects and scenes. With its high-coverage and high-diversity of exemplars, Places offers an ecosystem of visual context to guide progress on currently intractable visual recognition problems. Such problems could include determining the actions happening in a given environment, spotting inconsistent objects or human behaviors for a particular place, and predicting future events or the cause of events given a scene.
The authors would like to thank Santani Teng, Zoya Bylinskii, Mathew Monfort and Caitlin Mullin for comments on the paper. Over the years, the Places project was supported by the National Science Foundation under Grants No. 1016862 to A.O and No. 1524817 to A.T; ONR N000141613116 to A.O; as well as MIT Big Data Initiative at CSAIL, Toyota, Google, Xerox and Amazon Awards, and a hardware donation from NVIDIA Corporation, to A.O and A.T. B.Z is supported by a Facebook Fellowship.
S. Hochreiter and J. Schmidhuber, “Long short-term memory,”Neural computation, 1997.