A Harder ImageNet Test Set
We introduce natural adversarial examples -- real-world, unmodified, and naturally occurring examples that cause classifier accuracy to significantly degrade. We curate 7,500 natural adversarial examples and release them in an ImageNet classifier test set that we call ImageNet-A. This dataset serves as a new way to measure classifier robustness. Like l_p adversarial examples, ImageNet-A examples successfully transfer to unseen or black-box classifiers. For example, on ImageNet-A a DenseNet-121 obtains around 2 accuracy drop of approximately 90 because ImageNet-A examples exploit deep flaws in current classifiers including their over-reliance on color, texture, and background cues. We observe that popular training techniques for improving robustness have little effect, but we show that some architectural changes can enhance robustness to natural adversarial examples. Future research is required to enable robust generalization to this hard ImageNet test set.READ FULL TEXT VIEW PDF
In this paper, we develop improved techniques for defending against
This paper analyzes the problems of standard adversarial accuracy and
Over the last few years, the phenomenon of adversarial examples ---
We introduce Mischief, a simple and lightweight method to produce a clas...
We introduce a systematic framework for quantifying the robustness of
Human visual systems are robust to a wide range of image transformations...
The repeated reuse of test sets in popular benchmark problems raises dou...
A Harder ImageNet Test Set
Research on the ImageNet benchmark has led to numerous advances in classification (AlexNet), object detection (Huang2017SpeedAccuracyTF), segmentation (He2018MaskR), and perceptual metrics (zhang2018perceptual). ImageNet classification improvements have are broadly applicable and are highly predictive of improvements on other tasks (Kornblith2018DoBI). Unfortunately, ImageNet classification accuracy has plateaued, so much that the final ImageNet Challenge competition was in 2017. While this suggests image classification is nearly complete, Recht2019DoIC remind us that ImageNet test examples tend to be relatively simple close-up images, so that the current test set may be too easy and not represent harder images encountered in the real world.
Real-world images may also be constructed or chosen adversarially. goodfellowblog define adversarial examples (adversarial)
as “inputs to machine learning models that an attacker has intentionally designed to cause the model to make a mistake.” Adversarial examples enable measuring worst-case model performance. This we aim to measure under the constraint that examples are naturally occurring. Most adversarial examples research centers around artificialadversarial examples, which are examples perturbed by nearly worst-case distortions that are small in an sense. Aside from the known difficulties in evaluating robustness (bypass; evaluatingadvrobustness), Gilmer2018MotivatingTR point out that adversarial examples assume an unrealistic threat model because attackers are often free to choose any desired input. Consequently, if an attacker aims to subvert black-box classifier accuracy, they could mimic known errors (Gilmer2018MotivatingTR). We find that model errors on natural adversarial examples are highly consistent. Attackers can reliably and easily create black-box attacks by exploiting these consistent natural model errors, and thus carefully computing gradient perturbations to create an attack is unnecessary. This less restricted threat model has been discussed but not explored thoroughly until now.
We adversarially curate a hard ImageNet classifier test set which we call ImageNet-A. These images are natural, unmodified, real-world examples collected from online and are selected to cause a “model to make a mistake,” as with synthetic adversarial examples. ImageNet-A examples cause mistakes due to occlusion, weather, and other complications encountered in the long tail of scene configurations, or by exploiting consistent classifier blind spots. Some examples are depicted in Figure 1. In Section 4 we show that ImageNet-A images successfully transfer to break many unseen or black-box models. As with other black-box adversarial examples, natural adversarial examples are selected to break a fixed model, in this case ResNet-50, but to black-box models they transfer reliably. For example, these images undermine models such as ResNeXt-50 (resnext) and DenseNet-121 (densenet), both of which have an ImageNet-A accuracy rate of less than 3%.
We examine ways to improve robustness on ImageNet-A. Numerous techniques have been proposed to improve classifier robustness. Two of these, Stylized ImageNet data augmentation (geirhos2019) and adversarial example training on an ImageNet scale (kang), hardly increase robustness to natural adversarial examples. However, much greater robustness gains follow from architectural modifications such as self-attention and increasing model size. These results suggest architecture design research as a promising avenue for improving robustness. Code and the ImageNet-A dataset are available at https://github.com/hendrycks/natural-adv-examples.
Adversarial examples are a means to estimate worst-case model performance. While we aim to estimate the worst-case accuracy in natural settings, most useadversarial attacks. Several other forms of adversarial attacks have been considered in the literature, including elastic deformations (Xiao2018SpatiallyTA), adversarial coloring (Bhattad2019BigBI; Hosseini2018SemanticAE), and synthesis via generative models (Baluja2017AdversarialTN; Song2018ConstructingUA) and evolutionary search (Nguyen2015DeepNN), among others. Other work has shown how to print 2D (Kurakin2017AdversarialEI; brown2017adversarial) or 3D (sharif2016accessorize; athalye2017synthesizing) objects that fool classifiers. For all that, these existing adversarial attacks are all based on synthesized images or objects, and some have questioned whether they provide a reliable window into real-world robustness (Gilmer2018MotivatingTR). Our examples are closer in spirit to the recent thought experiment of an adversarial photographer discussed in Brown2018UnrestrictedAE, and by definition occur in the real world.
Robustness to Train-Test Mismatch. Recht2019DoIC create a new ImageNet test set resembling the original test set as closely as possible. They determined that matching the difficulty of the original ImageNet test set required selecting images deemed the easiest and most obvious by the Mechanical Turkers. ImageNet-A helps measure generalization to images of harder scenarios, not just the easiest examples. Brendel2018ApproximatingCW show that classifiers which do not know the spatial ordering of image regions can be competitive on the ImageNet test set, possibly due to the dataset’s lack of difficulty. Next, note that judging classifiers by their performance on easy examples has potentially masked many of their shortcomings. For example, geirhos2019 artificially overwrite each ImageNet image’s textures and conclude that classifiers learn to rely on textural cues and under-utilize information about object shape. Recent work proposes a benchmark for evaluating robustness to non-adversarial corruptions called ImageNet-C (hendrycks2019robustness). They corrupt images with 75 different algorithmically generated corruptions, while in this work we measure robustness to less palpable distribution shift with adversarial data. Our sources of distribution shift also tend to be more realistic, heterogeneous, and varied. Obtaining robustness to varied forms of distribution shift is difficult. For example, igor; geirhos train on various distortions and show that networks tend to memorize distortions and thereby fail to generalize to new and unseen distortions. Hence, robustly generalizing to unseen long-tail complications in ImageNet-A, such as translucent shrink wrap that envelopes a toaster, could also be difficult.
ImageNet-A is a dataset of natural adversarial examples, or real-world examples which fool current classifiers. To find natural adversarial examples, we first download numerous images related to an ImageNet class. Thereafter we delete the images that ResNet-50 (resnet) classifiers correctly classify. With the remaining incorrectly classified examples, we manually select a subset of high-quality images. This process is explicated below.
Class Restrictions. Before adversarially curating data to create ImageNet-A, we select a 200-class subset of ImageNet-1K’s 1,000 classes so that errors among these 200 classes would be considered egregious (imagenet). For instance, wrongly classifying Norwich terriers as Norfolk terriers does less to demonstrate faults in current classifiers than mistaking a Persian cat for a candle. In selecting ImageNet-A’s 200 classes, we adhere to several other design considerations. We avoid rare classes such as “snow leopard,” classes which have changed much since 2012 such as “iPod,” coarse and abstract ImageNet classes such as “spiral,” classes which are often image backdrops such as “valley,” and finally classes which tend to overlap such as “honeycomb,” “bee,” “bee house,” and “bee eater”; “eraser,” “pencil sharpener” and “pencil case”; “sink,” “medicine cabinet,” “pill bottle” and “band-aid”; and so on. The 200 ImageNet-A classes cover most broad categories spanned by ImageNet-1K. The full class list is in Appendix A.
Data Aggregation and Adversarial Selection.
Curating a large set of natural adversarial examples requires combing through an even larger set of images. Fortunately, the website iNaturalist has millions of user-labeled images of animals, and Flickr has even more user-tagged images of objects. We download a bevy of images related to each of the 200 ImageNet classes by leveraging user-provided labels and tags. This user-provided information enables us to avoid collecting images based on a search engine’s convolutional neural network image ranking algorithms, which would prevent us from finding blindspots in convolutional neural networks. After exporting or scraping data from sites including iNaturalist and Flickr, we adversarially select the images.
We now describe our adversarial selection process. At a high level, this process has parallels to Song2018ConstructingUA. They use generative models to create adversarial examples that fool target classifier, and remove examples which fail to fool said target classifier. Rather than sampling synthetic images from the range of a generative model, we sample natural images from the real world. We remove examples which fail to fool our target classifiers, which are based on the commonplace and competitive ResNet-50 architecture. Of the remaining images, we select low-confidence images and ensure each image is valid through human review. This procedure also has similarities to hard-example mining, much like how adversarial perturbations are hard perturbations selected from an ball.
More specifically, we use two ResNet-50s for filtering, one pre-trained on ImageNet-1K and fine-tuned on the 200 class subset, and one pre-trained on ImageNet-1K where 200 of its 1,000 logits are used in classification. Both classifiers have similar accuracy on the 200 clean test set classes from ImageNet-1K, but we choose two models in order to have a small ensemble. The ResNet-50s perform 10-crop classification of each image, and should any crop be classified correctly by the ResNet-50s, the image is removed. If either ResNet-50 assigns an image greater than 15% confidence in the correct class, the image is also removed; this is so done because we want natural adversarial examples which yield misclassifications with low confidence in the correct class, like in untargeted adversarial attacks. Now, some classification confusions are greatly over-represented, such as Persian cat and lynx. We would likeImageNet-A
to have great variability in its types of errors and cause classifiers to have a dense confusion matrix. Consequently, we perform a second round of filtering to create a shortlist where each confusion only appears at most 15 times. Finally, we manually select images from this shortlist in order to ensureImageNet-A images are simultaneously valid, single-class, and high-quality.
We concretely describe the process for the dragonfly class. We download 81,413 dragonfly images from iNaturalist, and after performing a basic filter we have 8,925 dragonfly images. In the algorithmically suggested shortlist, 1,452 images remain. From this shortlist, 80 dragonfly images are manually selected, but hundreds more could be chosen. The exact number of natural adversarial examples varies by class. In all, the ImageNet-A dataset has 7,500 natural adversarial examples.
The natural adversarial examples in ImageNet-A demonstrate numerous failure modes of modern convolutional neural networks. For example, Figure 2 shows that classifiers may predict a class even when the image does not contain the subparts necessary to identify the predicted class. In the leftmost image, the candle is predicted as a jack-o’-lantern with 99.94% confidence, despite the absence of a pumpkin or carved faces. A separate way to measure how well a network learns shape is to test generalization on paintings, cartoons, sketches (wang2019learning)
, sculptures, toys, origami, embroidery, and so on, but we avoid such examples and only keep natural images during adversarial filtration. While computer vision models such as the deformable parts model(dpm) more explicitly represented part configurations, current convolutional neural networks may produce confident predictions without necessary shape evidence.
Networks may also rely too heavily on texture and color cues (Figure 4), for instance misclassifying a dragonfly as a banana presumably due to a nearby yellow shovel. Since classifiers are taught to associate entire images with an object class, frequently appearing background elements may also become associated with a class, such as hummingbird feeders with hummingbirds or wood with nails (Figure 4). Other examples include leaf-covered tree branches being associated with the white-headed capuchin monkey class, snow being associated with shovels, and dumpsters with garbage trucks.
A small but sizeable share of ImageNet-A examples are naturally distorted. That these examples cause our ResNet-50 filters to misclassify is in keeping with previous observations (hendrycks2019robustness; geirhos) that classifiers often fail to generalize to distorted images. These distorted images highlight one way in which the ImageNet-A dataset is harder than the usual ImageNet test set, so methods which only decorrelate predictions from ResNet-50 are not sufficient to solve ImageNet-A.
Classifiers also demonstrate fickleness to small scene variations. Figure 7 shows an American alligator swimming. With different frames of the alligator swimming, the classifier prediction varies erratically between classes that are semantically loose and separate. For other images of the swimming alligator, classifiers predict that the alligator is a cliff, lynx, and fox squirrel.
Last, we find that classifiers also overgeneralize (hendrycks17baseline; hendrycks2019oe; kimin), either over-extrapolating a pattern within a class (shadow sundial), or incorrectly including nearby categories (car limousine). In addition to Figure 7’s examples, we find that the classifiers overgeneralize tricycles to bicycles and circles, digital clocks are overgeneralized to keyboards and calculators, and so on. In all, current convolutional networks have pervasive and diverse failure modes that can now be estimated with ImageNet-A.
Much of the force of the research community has been channeled toward endowing networks with robust representations. We examine the best-in-class robust training techniques. Unfortunately, we find that on natural adversarial examples, these techniques hardly help.
Adversarial Training. We investigate how much robustness adversarial training confers. Adversarially training the parameters
with loss functionon dataset involves the objective
Such adversarial examples are found through an iterative procedure similar to projected gradient ascent (madry),
where is with random uniform noise added to each pixel.
We try three different adversarial training schemes with adversaries of different strengths. The first is degenerate adversarial training with a zero-step adversary. In this case, training examples are simply corrupted by random uniform noise where , so varies between examples. We randomly scale epsilon so that the model learns to be robust to perturbations of various scales. The second is FGSM training against a single-step adversary. Here . Finally, we adversarially train against a 10-step PGD attacker with and .
We train a ResNeXt-50 (324d) (resnext) from scratch on the 200 ImageNet-1K classes appearing in ImageNet-A
. This network trains for 90 epochs. The first five epochs follow a linear warmup learning rate scheduleGoyal2017AccurateLM, and the learning rate drops by a factor of at epochs 30, 60, and 80. We use a batch size of 256, a maximum learning rate of , a momentum parameter of , and a weight decay strength of . We use standard random horizontal flipping and cropping where each image is of size .
Observe in Figure 9 that augmenting the training data with random uniform noise changes robustness. Adding noise from a 1-step FGSM adversary slightly increases robustness further. A stronger 10-step adversary imparts slightly greater ImageNet-A robustness. However, the model trained on clean data has 89.22% accuracy on the 200 class subset of ImageNet-1K’s test set, while uniform noise data augmentation corresponds to an accuracy of 88.93%, FGSM to 83.95%, and PGD to 81.88%. Thus adversarial training’s miniscule robustness gains are hardly worth the cost.
Stylized ImageNet Augmentation. In Figure 4, we observe that classifiers may rely too heavily on color and textural features. geirhos2019 propose making networks rely less on texture by training classifiers on images where textures are transferred from art pieces. They accomplish this by applying style transfer to ImageNet training images to create a dataset they call Stylized ImageNet or SIN for short. We test whether training with SIN images can improve ImageNet-A robustness.
Reducing a ResNeXt-50’s texture bias by training with SIN images does little to help ImageNet-A robustness. For reference, the ResNeXt-50 trained on ImageNet images obtains 89.22% top-1 accuracy on the 200 class subset of ImageNet-1K’s test set. If we train a ResNeXt-50 entirely on Stylized ImageNet images, the top-1 accuracy on ImageNet-1K’s 200 class test set set is a meager 65.87%. Accuracy increases from 1.31% to 2.09% by switching from normal ImageNet training to adding Stylized ImageNet data augmentation. Notice that the accuracies are still small. This is a consequence of natural adversarial examples successfully transferring to unseen models and architectures. As shown in Figure 9, data augmentation with Stylized ImageNet results in minor ImageNet-A accuracy changes.
Uncertainty Metrics. The RMS Calibration Error is how we measure miscalibration. We would like classifiers that can reliably forecast their accuracy. Concretely, we want classifiers which give examples 60% confidence to be correct 60% of the time. Say that a classifier produces a confidence estimate that its prediction is correct. Then, we can judge a classifier’s miscalibration with the Root Mean Square Calibration Error. The RMS Calibration Error is
or the squared discrepancy between the classifier’s accuracy at a confidence level and the confidence level. We estimate the RMS Calibration Error using adaptive binning (oconnor; hendrycks2019oe; kilian)
. Here we take the confidence to be the maximum softmax probability(hendrycks17baseline).
Our second uncertainty estimation metric is the Area Under the Response Rate Accuracy Curve (AURRA). In a multitude of scenarios, responding only when confident is preferable to predicting falsely. In these experiments, we allow classifiers to respond to a subset of the test set and abstain from predicting the rest. Classifiers with quality uncertainty estimates should be capable identifying examples it is likely to predict falsely and abstain. If a classifier is required to abstain from predicting on 90% of the test set, or equivalently respond to the remaining 10% of the test set, then we should like the classifier’s uncertainty estimates to separate correctly and falsely classified examples and have high accuracy on the selected 10%. At a fixed response rate, we should like the accuracy to be as high as possible. At a 100% response rate, the classifier accuracy is the usual test set accuracy. We vary the response rates and compute the corresponding accuracies to obtain the Response Rate Accuracy (RRA) curve. The area under the Response Rate Accuracy curve is the AURRA. To compute the AURRA in this paper, we use the maximum softmax probability to determine which examples receive a response. If the response rate is 10%, we select the top 10% of examples with the highest confidence and compute the accuracy on these examples. An example RRA curve is in Figure 12.
Size and Self-Attention. Simply increasing the width and number of layers of a network is sufficient to automatically impart more robustness, calibration, and error detection quality on natural adversarial examples. Increasing network capacity has been shown to improve robustness to adversarial examples (kurakin), common corruptions (hendrycks2019robustness), and now also ImageNet-A as demonstrated in Figure 11. Networks are trained on ImageNet-1K without special training techniques.
Convolutional neural networks with self-attention (Hu2018GatherExciteE) are designed to better capture long-range dependencies and interactions across an image. Self-attention helps GANs learn how to generate images with plausible shape (Zhang2018SelfAttentionGA), and in classification, self-attention is utilized in state-of-the-art ImageNet-1K models. We consider the self-attention technique called Squeeze-and-Excitation (SE) (Hu2018SqueezeandExcitationN), which won the final ImageNet competition. While integrating Squeeze-and-Excitation into a ResNeXt-101 (324d) improves top-1 accuracy on the 200 class subset of ImageNet-1K by less than 1%, SE improves ImageNet-A accuracy by approximately 10%. An implication is that judging models with ImageNet-1K’s now simple test set can obscure a technique’s true benefits. Moreover, this form of self-attention greatly improves model calibration and error detection on ImageNet-A. Full results are shown in Figures 12 and 11. We also found that self-attention improves robustness to common corruptions; a ResNet-50’s mean corruption error (mCE) is 76.7% and an SE-ResNet-50 has a 68.2% mCE. Architectural modifications that improve ImageNet-C and ImageNet-1K accuracies can help tremendously on ImageNet-A.
In this paper, we introduced the first expansive dataset of natural adversarial examples. This dataset is challenging for both the computer vision and adversarial example research communities. That is because these naturally occurring images expose common blindspots of current convolutional networks, and closing the gap between ImageNet-1K’s test set and ImageNet-A will require addressing long-standing but under-explored failure modes of current models such as texture over-reliance, over-generalization, corruption robustness, and more. We found that these failure modes are only slightly less pronounced when models train with stylized ImageNet data augmentation or with adversarial training, while the most improvements in robustness came from modifying the classifier architecture itself. Even with both model scale and self-attention, classifier accuracies are still below that of adversarially trained classifiers on typical adversarial examples (xie2018feature), so that natural adversarial examples are in some sense currently more challenging. In this work, we identified several methods that increase natural adversarial example robustness, demonstrated that models have similar blindspots and make correlated errors, and created a hard new ImageNet test set to measure model robustness—an important research aim as computer vision systems are deployed in increasingly precarious environments.
We should like to thank Ludwig Schmidt for his extensive and continued suggestions throughout this paper’s development, and especially for his initial suggestion to collect naturally instances of common corruptions. We should also like to thank Adam Dziedzic and Saurav Kadavath for their help. This material is in part based upon work supported by the National Science Foundation Frontier Grant. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
The 200 ImageNet classes that we selected for ImageNet-A are as follows.
‘Stingray;’ ‘goldfinch, Carduelis carduelis;’ ‘junco, snowbird;’ ‘robin, American robin, Turdus migratorius;’ ‘jay;’ ‘bald eagle, American eagle, Haliaeetus leucocephalus;’ ‘vulture;’ ‘eft;’ ‘bullfrog, Rana catesbeiana;’ ‘box turtle, box tortoise;’ ‘common iguana, iguana, Iguana iguana;’ ‘agama;’ ‘African chameleon, Chamaeleo chamaeleon;’ ‘American alligator, Alligator mississipiensis;’ ‘garter snake, grass snake;’ ‘harvestman, daddy longlegs, Phalangium opilio;’ ‘scorpion;’ ‘tarantula;’ ‘centipede;’ ‘sulphur-crested cockatoo, Kakatoe galerita, Cacatua galerita;’ ‘lorikeet;’ ‘hummingbird;’ ‘toucan;’ ‘drake;’ ‘goose;’ ‘koala, koala bear, kangaroo bear, native bear, Phascolarctos cinereus;’ ‘jellyfish;’ ‘sea anemone, anemone;’ ‘flatworm, platyhelminth;’ ‘snail;’ ‘crayfish, crawfish, crawdad, crawdaddy;’ ‘hermit crab;’ ‘flamingo;’ ‘American egret, great white heron, Egretta albus;’ ‘oystercatcher, oyster catcher;’ ‘pelican;’ ‘sea lion;’ ‘Chihuahua;’ ‘golden retriever;’ ‘Rottweiler;’ ‘German shepherd, German shepherd dog, German police dog, alsatian;’ ‘pug, pug-dog;’ ‘red fox, Vulpes vulpes;’ ‘Persian cat;’ ‘lynx, catamount;’ ‘lion, king of beasts, Panthera leo;’ ‘American black bear, black bear, Ursus americanus, Euarctos americanus;’ ‘mongoose;’ ‘ladybug, ladybeetle, lady beetle, ladybird, ladybird beetle;’ ‘rhinoceros beetle;’ ‘weevil;’ ‘fly;’ ‘bee;’ ‘ant, emmet, pismire;’ ‘grasshopper, hopper;’ ‘walking stick, walkingstick, stick insect;’ ‘cockroach, roach;’ ‘mantis, mantid;’ ‘leafhopper;’ ‘dragonfly, darning needle, devil’s darning needle, sewing needle, snake feeder, snake doctor, mosquito hawk, skeeter hawk;’ ‘monarch, monarch butterfly, milkweed butterfly, Danaus plexippus;’ ‘cabbage butterfly;’ ‘lycaenid, lycaenid butterfly;’ ‘starfish, sea star;’ ‘wood rabbit, cottontail, cottontail rabbit;’ ‘porcupine, hedgehog;’ ‘fox squirrel, eastern fox squirrel, Sciurus niger;’ ‘marmot;’ ‘bison;’ ‘skunk, polecat, wood pussy;’ ‘armadillo;’ ‘baboon;’ ‘capuchin, ringtail, Cebus capucinus;’ ‘African elephant, Loxodonta africana;’ ‘puffer, pufferfish, blowfish, globefish;’ ‘academic gown, academic robe, judge’s robe;’ ‘accordion, piano accordion, squeeze box;’ ‘acoustic guitar;’ ‘airliner;’ ‘ambulance;’ ‘apron;’ ‘balance beam, beam;’ ‘balloon;’ ‘banjo;’ ‘barn;’ ‘barrow, garden cart, lawn cart, wheelbarrow;’ ‘basketball;’ ‘beacon, lighthouse, beacon light, pharos;’ ‘beaker;’ ‘bikini, two-piece;’ ‘bow;’ ‘bow tie, bow-tie, bowtie;’ ‘breastplate, aegis, egis;’ ‘broom;’ ‘candle, taper, wax light;’ ‘canoe;’ ‘castle;’ ‘cello, violoncello;’ ‘chain;’ ‘chest;’ ‘Christmas stocking;’ ‘cowboy boot;’ ‘cradle;’ ‘dial telephone, dial phone;’ ‘digital clock;’ ‘doormat, welcome mat;’ ‘drumstick;’ ‘dumbbell;’ ‘envelope;’ ‘feather boa, boa;’ ‘flagpole, flagstaff;’ ‘forklift;’ ‘fountain;’ ‘garbage truck, dustcart;’ ‘goblet;’ ‘go-kart;’ ‘golfcart, golf cart;’ ‘grand piano, grand;’ ‘hand blower, blow dryer, blow drier, hair dryer, hair drier;’ ‘iron, smoothing iron;’ ‘jack-o’-lantern;’ ‘jeep, landrover;’ ‘kimono;’ ‘lighter, light, igniter, ignitor;’ ‘limousine, limo;’ ‘manhole cover;’ ‘maraca;’ ‘marimba, xylophone;’ ‘mask;’ ‘mitten;’ ‘mosque;’ ‘nail;’ ‘obelisk;’ ‘ocarina, sweet potato;’ ‘organ, pipe organ;’ ‘parachute, chute;’ ‘parking meter;’ ‘piggy bank, penny bank;’ ‘pool table, billiard table, snooker table;’ ‘puck, hockey puck;’ ‘quill, quill pen;’ ‘racket, racquet;’ ‘reel;’ ‘revolver, six-gun, six-shooter;’ ‘rocking chair, rocker;’ ‘rugby ball;’ ‘saltshaker, salt shaker;’ ‘sandal;’ ‘sax, saxophone;’ ‘school bus;’ ‘schooner;’ ‘sewing machine;’ ‘shovel;’ ‘sleeping bag;’ ‘snowmobile;’ ‘snowplow, snowplough;’ ‘soap dispenser;’ ‘spatula;’ ‘spider web, spider’s web;’ ‘steam locomotive;’ ‘stethoscope;’ ‘studio couch, day bed;’ ‘submarine, pigboat, sub, U-boat;’ ‘sundial;’ ‘suspension bridge;’ ‘syringe;’ ‘tank, army tank, armored combat vehicle, armoured combat vehicle;’ ‘teddy, teddy bear;’ ‘toaster;’ ‘torch;’ ‘tricycle, trike, velocipede;’ ‘umbrella;’ ‘unicycle, monocycle;’ ‘viaduct;’ ‘volleyball;’ ‘washer, automatic washer, washing machine;’ ‘water tower;’ ‘wine bottle;’ ‘wreck;’ ‘guacamole;’ ‘pretzel;’ ‘cheeseburger;’ ‘hotdog, hot dog, red hot;’ ‘broccoli;’ ‘cucumber, cuke;’ ‘bell pepper;’ ‘mushroom;’ ‘lemon;’ ‘banana;’ ‘custard apple;’ ‘pomegranate;’ ‘carbonara;’ ‘bubble;’ ‘cliff, drop, drop-off;’ ‘volcano;’ ‘ballplayer, baseball player;’ ‘rapeseed;’ ‘yellow lady’s slipper, yellow lady-slipper, Cypripedium calceolus, Cypripedium parviflorum;’ ‘corn;’ ‘acorn.’
Their WordNet IDs are as follows.
n01498041, n01531178, n01534433, n01558993, n01580077, n01614925, n01616318, n01631663, n01641577, n01669191, n01677366, n01687978, n01694178, n01698640, n01735189, n01770081, n01770393, n01774750, n01784675, n01819313, n01820546, n01833805, n01843383, n01847000, n01855672, n01882714, n01910747, n01914609, n01924916, n01944390, n01985128, n01986214, n02007558, n02009912, n02037110, n02051845, n02077923, n02085620, n02099601, n02106550, n02106662, n02110958, n02119022, n02123394, n02127052, n02129165, n02133161, n02137549, n02165456, n02174001, n02177972, n02190166, n02206856, n02219486, n02226429, n02231487, n02233338, n02236044, n02259212, n02268443, n02279972, n02280649, n02281787, n02317335, n02325366, n02346627, n02356798, n02361337, n02410509, n02445715, n02454379, n02486410, n02492035, n02504458, n02655020, n02669723, n02672831, n02676566, n02690373, n02701002, n02730930, n02777292, n02782093, n02787622, n02793495, n02797295, n02802426, n02814860, n02815834, n02837789, n02879718, n02883205, n02895154, n02906734, n02948072, n02951358, n02980441, n02992211, n02999410, n03014705, n03026506, n03124043, n03125729, n03187595, n03196217, n03223299, n03250847, n03255030, n03291819, n03325584, n03355925, n03384352, n03388043, n03417042, n03443371, n03444034, n03445924, n03452741, n03483316, n03584829, n03590841, n03594945, n03617480, n03666591, n03670208, n03717622, n03720891, n03721384, n03724870, n03775071, n03788195, n03804744, n03837869, n03840681, n03854065, n03888257, n03891332, n03935335, n03982430, n04019541, n04033901, n04039381, n04067472, n04086273, n04099969, n04118538, n04131690, n04133789, n04141076, n04146614, n04147183, n04179913, n04208210, n04235860, n04252077, n04252225, n04254120, n04270147, n04275548, n04310018, n04317175, n04344873, n04347754, n04355338, n04366367, n04376876, n04389033, n04399382, n04442312, n04456115, n04482393, n04507155, n04509417, n04532670, n04540053, n04554684, n04562935, n04591713, n04606251, n07583066, n07695742, n07697313, n07697537, n07714990, n07718472, n07720875, n07734744, n07749582, n07753592, n07760859, n07768694, n07831146, n09229709, n09246464, n09472597, n09835506, n11879895, n12057211, n12144580, n12267677.