Faster RER-CNN: application to the detection of vehicles in aerial images

by   Jean Ogier Du Terrail, et al.

Detecting small vehicles in aerial images is a difficult job that can be challenging even for humans. Rotating objects, low resolution, small inter-class variability and very large images comprising complicated backgrounds render the work of photo-interpreters tedious and wearisome. Unfortunately even the best classical detection pipelines like Faster R-CNN cannot be used off-the-shelf with good results because they were built to process object centric images from day-to-day life with multi-scale vertical objects. In this work we build on the Faster R-CNN approach to turn it into a detection framework that deals appropriately with the rotation equivariance inherent to any aerial image task. This new pipeline (Faster Rotation Equivariant Regions CNN) gives, without any bells and whistles, state-of-the-art results on one of the most challenging aerial imagery datasets: VeDAI and give good results w.r.t. the baseline Faster R-CNN on two others: Munich and GoogleEarth .



There are no comments yet.


page 12

page 13


Fast Vehicle Detection in Aerial Imagery

In recent years, several real-time or near real-time object detectors ha...

LR-CNN: Local-aware Region CNN for Vehicle Detection in Aerial Imagery

State-of-the-art object detection approaches such as Fast/Faster R-CNN, ...

Aerial Images Processing for Car Detection using Convolutional Neural Networks: Comparison between Faster R-CNN and YoloV3

In this paper, we address the problem of car detection from aerial image...

Convolutional Neural Networks for Aerial Multi-Label Pedestrian Detection

The low resolution of objects of interest in aerial images makes pedestr...

An Accurate and Real-time Self-blast Glass Insulator Location Method Based On Faster R-CNN and U-net with Aerial Images

The location of broken insulators in aerial images is a challenging task...

AMRNet: Chips Augmentation in Areial Images Object Detection

Detecting object in aerial image is challenging task due to 1) objects a...

Small or Far Away? Exploiting Deep Super-Resolution and Altitude Data for Aerial Animal Surveillance

Visuals captured by high-flying aerial drones are increasingly used to a...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

In just a few years the detection capabilities of modern methods have rocketed. From the Viola-Jones detector to the brand new installment of the R-CNN family He et al. [2017]

the computer vision community in detection has progressively let go of old paradigms (sliding windows, feature extraction

training, traditional NMS) to embrace shinier new ones (Fully convolutional methods, end-to-end, soft/learnable NMS), which has led to drastic improvements in speed, performances and flexibility in modern detection pipelines.

However it seems that one of them, the use of PASCAL VOC vertical (axis-aligned) bounding boxes for every single detection problem, is still resisting the course of time even in situations in which it makes little sense.

There are two main reasons why it is the case; First of all, natural objects on images can have arbitrary shapes therefore we either find a simple approximation that kind of work in every case (the vertical bounding boxes) or we move to semantic segmentation (or even do both like in He et al. [2017]). Second, the need for a standard to be able to compare different methods on an equal footing with each other. Thanks to PASCAL VOC IoU criteria (see Everingham et al. [2015]) we can for instance compare the recall of very different object proposals.

We argue that in the specific domain of vehicle detection in aerial imagery these two reasons are lapsed. First, all the vehicles to detect seen from the sky are perfect rectangles or at least fairly close. That is why doing semantic segmentation like in Audebert et al. [2017] is arguably a bit ’overkill’ and is not possible in most of the available aerial imagery datasets. Therefore one cannot use Mask-RCNN He et al. [2017]-like structures. Moreover these rectangles can have any orientation (see Figure 1).

Figure 1: The two frameworks side by side: straight and oriented annotations

It is therefore completely natural to propose a new paradigm: oriented rectangle-shaped bounding boxes for annotations (which already exist or have been provided in all aerial imagery datasets), for region proposals and for evaluation also making it a new unified coherent framework.

This very simple idea has been, to the best of our knowledge, mostly ignored in aerial imagery until the very recent introduction of DOTAXia et al. [2018] although it has received some attention by concurrent work very recently in the text detection community (Liu and Jin [2017], Shi et al. [2017], Busta et al. [2017], Ma et al. [2018]) specifically for the ICDAR challenges. This led to authors either completely ignoring orientations by using vertical bounding boxes e.g., Tang et al. [2017]

and thus inventing their own evaluation metric in the process (which prevents any fair comparison), studying the orientations completely separately from the detections

e.g., Gonzalez et al. [2017], or at best studying the two problems simultaneously but performing detection first on one side and only then angle regression on another like in Liu and Máttyus [2015], a mindset which is visible in the different metrics used to evaluate the final result. Yet we prove experimentally that performing both simultaneously even improves the classification accuracy.

This contribution is therefore three-folded: (i) We adapt the Faster R-CNN framework to simultaneously do detection and orientation inference. (ii) We suggest a suitable metric to compare rotated and non rotated detection methods (iii) We get state-of-the-art results on a well-known aerial imagery dataset using the previous framework and get competitive performance w.r.t. the baseline on two others.

2 Related work

General detection on images is a broad subject. Countless work has been done on the topic. However without going back all the way to Viola and Jones Viola and Jones [2001] we feel compelled to refer to the emblematic series of the R-CNN articles Girshick [2015], Girshick et al. [2014], He et al. [2017], Ren et al. [2015], the recent extensions like Dai et al. [2016b] and also all modern single shot methods like Liu et al. [2016], Redmon et al. [2016], Redmon and Farhadi [2017].

All those articles have inspired to various extents this paper the closest one being Faster R-CNN Ren et al. [2015], which we extend in this work.

Vehicle detection on aerial imagery was also heavily investigated. Previous works like Ali et al. [2012], Eikvil et al. [2009], Kembhavi et al. [2011]are worth mentioning. More recent studies involve handcrafted descriptors Gleason et al. [2011], Razakarivony and Jurie [2014]

and/or convolutional neural networks (CNNs)

Chen et al. [2013], Mundhenk et al. [2016].

Lately in 2017, several articles in the domain stood out. Among them one can cite the article by Tang et al. Tang et al. [2017], which demonstrates the need to extend the capabilities of Faster R-CNN in order to tackle aerial imagery detection. They chose to focus on augmenting the resolution of the feature maps while doing hard example mining via a boosting strategy. Our approach is orthogonal to their work so they could be combined with the benefits of both frameworks.

Audebert et al. Audebert et al. [2017]

has also shown how important could segmenting vehicles correctly before classifying them be. But one needs a huge amount of pixel level data to even start talking about semantic segmentation, data which is not always available specifically in this domain. But these results are in concert with ours: once you have the orientation mask of the vehicle, classification is considerably simplified.

Ogier du Terrail and Jurie Ogier Du Terrail and Jurie [2017] got good results on VeDAI using a LeNet-5 LeCun et al. [1998] fully convolutional network with hard examples mining but agree on the interest of a cascade for this problem.

Our work is meant as a tribute to Liu and Mattius Liu and Máttyus [2015], who pioneered our direction of studies in aerial imagery they were the first to observe that the classification of cars is made easier if one lowers the intra-class variability by learning an angle regressor first and only then a classifier on the now vertical cars.

We feel that although many great recent articles worked on rotation invariant classifiers (possible first step in making a detector equivariant) like Gonzalez et al. [2017], Tsaftaris and Giuffrida [2016] this line of work has been insufficiently explored especially in the context of detection. We note that Marco et al. Gonzalez et al. [2017] and Joao and Vedaldi Henriques and Vedaldi [2017]

learn to estimate the angles of vehicles from aerial imagery without considering the detection problem.

Interestingly, this is in the domain of text detection that we can find the most relevant literature although it seems still very recent. Both Liu and Jin [2017], Shi et al. [2017] use oriented proposals and interior point methods like us. For text recognition it was abundantly clear from the start that one needs a detection aligned with the text and therefore that vertical bounding boxes are insufficient. However oriented rectangles are also too coarse an approximation. Liu and Jin Liu and Jin [2017] have to use complicated quadrangles and Shi et alShi et al. [2017] have to resort to segment detection but even then both methods fail to capture text with too much curvature. In aerial imagery we do not have this problem nor the requirement to use quadrangles/segments instead of simple rectangles. That makes our method faster and more accurate as our prior can be stronger.

We also built on the two very recent works of Busta et al. [2017], Ma et al. [2018]. However, even if the structure of their detector bears resemblance to ours they made very different design choices:

  • Both of them use a simple regression cost, which did not converge in our case. It forced us to put more thoughts on engineering a more stable cost (see 3.3.3).

  • Ma et al. Ma et al. [2018] parametrization is impractical because they need to recompute all corners for each bounding box during the IoU calculations whereas our allow implementation tricks (see 3.2.2).

  • Busta et al. Busta et al. [2017] do not mention the NMS step nor the IoU computation it is unclear how they did it.

  • Both the bilinear interpolation from

    Busta et al. [2017] and the RRoI from Ma et al. [2018] are more precise but slower than our coarse pooling implementation that check pixel membership using dot products (see 3.3.2)

  • They have different goals than us: Ma et al. Ma et al. [2018] is doing oriented text detection and Busta et al. Busta et al. [2017] is more focused on text recognition.

Outside of the text detection domain, we have found very few articles that approach the detection problem with rotated proposals except the work from Chen and Dong Chen et al. [2016]

on faces that can detect only vertical bounding boxes even if, thanks to the spatial transformer network

Jaderberg et al. [2015], they can warp to a certain extent what is inside the vertical regions to ease classification, which is close in spirit to this work.

All works on 3D oriented bounding boxes predictions from point clouds or voxels Zhou and Tuzel [2018] are also distantly related to our article.

3 Adaptation of Faster R-CNN to orientation inference

As aforementioned, the Faster R-CNN detector has been designed for datasets such as MS-COCO for which bounding boxes are not oriented. This section proposes an efficient way to extend it to the prediction of oriented detection boxes. We first provide some minimum information about Faster R-CNN, give a parametrization of the bounding boxes allowing them to rotate, explain how to compute bounding boxes intersection in this context and finally how to extend the Faster R-CNN to allow it to handle rotating bounding boxes.

3.1 Faster R-CNN, a reminder

Faster R-CNN Ren et al. [2015] is a modern cascaded detection framework whose simplicity and robustness have seduced a wide range of authors since its coming of age in 2015. This pipeline is made of two parts. The first one, so-called RPN (Region Proposal Network) is used to propose (vertical) regions. In order to do so the image under study is divided into a grid of NxN cells. Each cell is responsible for a certain number of reference bounding boxes called anchors. Each anchor receives an objectness score and is shifted and distorted to better match the ground truth (this step is called bounding box regression). After scoring, a certain number of these new bounding boxes is then passed on (after Non-Max Suppression) to a second network.

This second network called Fast R-CNN uses the regions to zoom in on interesting parts of the image feature maps, by a warping pooling process known as Region Of Interest Pooling. After RoI pooling all the regions have become fixed size features that can be further processed by fully connected layers to do fine grained classification (and can also be shifted and distorted a second and final time by a second layer of bounding box regression).

This work gives more flexibility rotation wise to this framework by extending several key components. The first step in building this enhanced Faster R-CNN is rethinking how to parametrize rotated boxes in order to have a coherent framework and be able to efficiently compute IoU between those boxes, which is essential in Faster R-CNN. We do so in the next section.

3.2 Introducing the foundations of Faster RER-CNN

3.2.1 Parametrizing rotating bounding boxes

The well know parameterization used for a vertical bounding box is:


which corresponds to the top left corner and the bottom right-hand corner of the box. Obviously allowing the boxes to rotate add degrees of liberty and therefore new parameters.

We wanted to find the simplest parameterization and the closest to the former so that the transition from one system to the other is as simple as possible. We were also motivated by the need to unify the annotations that can be found in aerial imagery datasets in surprisingly complex forms : the annotations provided by Joao and Vedaldi in Henriques and Vedaldi [2017] for the GoogleEarth dataset Heitz and Koller [2008], for instance, is a perfect example of how authors in aerial imagery frequently did not really consider the detection and angle regression to be related, the angle annotations of VeDAI can also be confusing at first with some vehicle having orientations in and others in .

We chose the following:


where A is one of the corners of the rectangle (A is chosen to be on the back of the vehicle if it has a back so that AB is the largest side, but it can be any corner as long as this choice remain coherent throughout the annotations). ABCD are the corners of the rectangle in clockwise order. We will see that it is crucial to get good performances. is the angle between the horizontal and AB counted clockwise

This parametrization encompasses all use cases that can be found in aerial imagery. Furthermore this system has the nice property of being exactly the same as VOC if and only if . This is therefore different from Shi et al. [2017], which uses and instead of and and very different from the very complicated ten points coordinates of Liu and Jin [2017] that certainly allows higher flexibility for text but is useless for aerial imagery.

3.2.2 Pascal VOC IoU with oriented rectangles

The Intersection Over Union is not only a way of asserting the compatibility between a predicted bounding box and the ground truth but also a crucial step in almost all modern pipelines (assessing if several boxes can be merged during NMS for instance). We wanted to design an algorithm capable of keeping this simplicity without being too computationally intensive. To do that we first compute the width and height of the two considered rectangles. Simple maths gives:


with: and , where AB and BC are always positive with this parameterization (no need to take absolute value).

Therefore, we can have access to the area of the individual rectangles. For the intersection, we need some more complex calculations. As all the rectangles are labeled in clockwise order, we can use the Sutherland-Hogman algorithm E. Sutherland and W. Hodgman [1974] to find the corners of the intersection by consecutively/successively clipping the first rectangle with each edge of the second one.

Once we have the intersection – which is therefore also in clockwise order – we need only apply the Green-Riemann theorem Green [1852] to have a simple and efficient way of computing the area of the intersection.

Thus we can calculate the IoU of two oriented regions in a non-painful way. This is summed up in the Figure 2, which shows the GR formula’s result from the corners coordinates to .

Figure 2: Calculating the intersection of two oriented rectangles (note that in image coordinates all points are in anticlockwise order)

Two ultra-efficient multithreaded Cython implementations of the IOU-calculation AND the new rotated NMS have been written. The code needs to be multithreaded and very well thought-out otherwise the computation cost is too high, that is probably why the authors from

Liu and Jin [2017] favored Monte-Carlo simulations. Our algorithm has the advantage that it is exact.

Now the stage is set, we have what we need to move forward and are now ready to grapple with the actual implementation.

3.3 From Faster R-CNN to Faster RER-CNN

We will list in the following sections what are extensions of Faster R-CNN required to carry it forward to an aerial imagery setting with rotated objects.

3.3.1 The RPN

The first part of the R-CNN cascade is the region proposal part. To adapt the Faster R-CNN to our framework we simply duplicate the anchors of Faster R-CNN a fixed number of times in all directions between so that they span the entire spectrum of orientations (in order not to capture any dataset bias if an orientation is dominant like in VeDAI). We note that it could give performance improvements to specialize in the most widespread orientations but would hurt generality. Omitting the regression part, which will be addressed in Section 3.3.3, with the IoU computations and the NMS being taken care of, the rest of the implementation is straightforward.

3.3.2 Oriented RoI pooling

Augmenting RoI pooling to gain newer abilities has been already tackled by ROIalign He et al. [2017] and several RoI warping pooling have already been proposed Jaderberg et al. [2015], Dai et al. [2016a], Ma et al. [2018]. We favor simplicity and rapidity over precision by using almost the same formulation as the original RoI pooling Girshick [2015]. Meaning we use the same harsh quantization to define the pooling cells corners; Then we go through every pixel of the feature map inside the vertical bounding box of the current rotated pooling cell (which identifies to the original non rotated pooling cell when angle is 0) to find the maximum.

The added complexity lies in the fact that we now have to check for each of these pixels if it belongs to the associated rotated pooling cell. We do that by using simple dot products.

Obviously due to the coarse quantization of the pixels grid it can happen that a rotated cell is left empty, when it is the case we fill the cell with the nearest neighbor pixel. We provide a visual sanity check of our implementation Figure 3 for a 20x20 region: the quantization is very coarse and we can observe misalignments contrary to other sampler using bilinear interpolation but ours does not need to compute the values of the 4 neighboring pixels.

Figure 3: From left to right: a Region of Interest and its parametrization on the original image, the locations of the pooled pixels in the original image (argmaxes)(backward), the pooled pixel reordered after Rotated RoI pooling (using the original image for visualization purposes)(forward)

3.3.3 Regressing oriented boxes

The vital part of the implementation is the way one regresses the boxes from the anchors (or from the RPN proposals). For instance, if we regress directly the offset () between our prediction angle and the target angle in radians, we run into several problems. Indeed by using the simple parameterization described in section 3.2.1 we have introduced a lot of ambiguity since there are now 4 valid ways of placing A even on a perfect prediction (matching the ground truth exactly) and therefore 4 possible values associated. We have to go through each 4 cases. We follow the same convention of using the subscript t to indicate the ground truth target and no subscript for the original prediction

  • A is near/at , ,

  • A is near/at , ,

  • A is near/at , ,

  • A is near/at , ,

We therefore want gradient in the second and fourth case, otherwise we will go away from the ground truth by regressing to , etc. We want no gradient flowing in the other cases as we do not want to force A onto a specific side as it leads to instability and bad generalizations. Of course in addition there is also the problem of the periodicity.

One can see that using the cost below solves both issues.


Then we can just plug the angle regression term in the regression cost:


Where is the traditional smooth L1 cost with parameter . refers to a target and to an anchor; the index

marks the center;W is AB and H is BC (see previous sections). And the other unreferenced Greek letters are just hyperparameters (cross validated). We integrate this new cost inside the Faster R-CNN framework, we train the RPN and Fast R-CNN parts simultaneously as is done in the literature. The Figure 

4 presents a graphic summary of the created framework.

Figure 4: Overview of Faster RER-CNN

4 Experiments

4.1 Datasets

In these experiments, we mainly use the VeDAI dataset Razakarivony and Jurie [2015], which consists of 1200 images of SaltLake City that comes in two different sizes 512x512 and 1024x1024. Every image is available in two versions either colored (RGB) or infrared. The dataset is provided with 10 folds and around 10 classes of vehicles (cars, pickups, vans, boats, etc.). The definition of what is a positive detection is different from the standard Intersection Over Union (IOU) criterion adopted by the Pascal VOC Everingham et al. [2015] or MS COCO Lin et al. [2014]. A detection is considered correct if its center lies within the ellipse centered on the ground truth and lying inside the edges of the target car. We are interested in mainly two metrics namely the AP (Average Precision) which is taken on 11 points as in Pascal VOC 2007 and the recall at low FPPI (recall for a given rate of False Positive Per Image). All the experiments are performed on SII (Small Infrared Images 512x512) but the images are first up sampled by a factor 2 using bilinear interpolation and are therefore 1024x1024.

We also use the Munich dataset Leitloff et al. [2014] to verify that our framework is adapted to various aerial imagery settings. This dataset contains 20 large colored images. 10 for training and 10 for testing. The images have to be rasterized for efficient training. We thus cut the images in a grid of 512x512 patches that we up-sample to 1024x1024 using bilinear interpolation. Doing so, we have missed some of the vehicles. We sample additional 512x512 windows so that each ground truth can be found in at least one image. For test we perform sliding window with a step of 50 pixels, as in Tang et al. [2017], and we resize each window. By resizing we augment the computational cost but we avoid having to use higher resolution feature maps.

Finally, we use the Google Earth dataset provided by Heitz and Koller Heitz and Koller [2008] augmented by the annotations of Joao and Vedaldi Henriques and Vedaldi [2017]. There are 20 colored images taken over the city of Bruxelles. We train on 10 of them and test on 5 other as in Henriques and Vedaldi [2017]. As for Munich we use Pascal VOC IOU and VeDAI metric to ascertain the quality of the detection. We also have to do some up sampling on GoogleEarth otherwise neither Faster nor our framework converges.

4.2 Training details

We use stochastic gradient descent with a learning rate of 1e-3 and momentum for our framework; for Faster R-CNN it is actually too big of a learning rate and does not converge on several folds therefore we used 0.0001 instead. We validated hyperparameters for Faster R-CNN and our framework on fold01 validation set (VeDAI). Our framework has a grid with 36 anchors in each cell 9 angles two aspect ratios 0.5 and 1. and two sizes. We also experimented with anchors obtained by applying k-means++

Arthur and Vassilvitskii [2007] to angle normalized and centered anchors of the ground truth with distance like in Yolo v2 Redmon and Farhadi [2017] but found that it only gave a small boost w.r.t. a reasonable choice of base anchors. This is not entirely surprising considering the lack of variability in the aspect ratio in aerial imagery evoked in the introduction. For Faster R-CNN we also used as many as 36 anchors111We use a dense sampling of aspect ratio and sizes for Faster R-CNN anchors to get 36 anchors (6 and 6 respectively) to verify that we were not simply outperforming Faster R-CNN because we had more anchors per pixel. We use for both framework the same vgg16 Simonyan and Zisserman [2015]

backbone pretrained on IMAGENET.

4.3 Experimental validation of the Faster RER-CNN

All subsequent experiments in this subsection are conducted on fold1 test of VeDAI, with the objective of validating that everything works as expected and producing a comprehensive analysis of our algorithm. During comparisons with state-of-the-art methods (next section) we will use the entire dataset.

4.3.1 Comparison of the Region Proposal Networks with and without rotated anchors

In these experiments, we want to demonstrate how important it is to have rotated anchors in the region proposal step. In order to do this, we measure, on the test set, the mean recall across classes on the VeDAI dataset w.r.t. the VeDAI metric (see Section 4.4.1) as a function of the number of proposals generated. To do this we extract from the trained network the top N proposals from the RPN (after NMS) and check how many objects are hit. We do this before regression therefore the only possible locations of the proposals lay on the original grid and after regression when proposals are allowed to move and warp. Our framework, of course, it was built for that purpose, offers more flexibility as the boxes can rotate. It is clear when looking at Figure 5 the advantages that the rotated anchors bring. What is surprising is that even for few proposals (400) the gaps between the different RPNs are well marked: from 65.66 to 92.31 (a 41% increase!) after regression. Even more surprisingly, even before regression the recall is higher (by 48% !) meaning that since the metric is solely based on the position of the centers of the proposals and that all centers lie on the same grid for both frameworks it must be that training has been made easier/more general by our formulation and that better anchors are chosen. On the right-hand side of Figure 5 we can see the details per class for our framework.

Figure 5: Left: comparison of the mean Recall of the RPN across classes between both frameworks, Right: Per class recall of the RPN for our framework

4.3.2 Angle regression in the classification head

Now that we have demonstrated our RPN is working, we want to check that the regression of angle is functioning properly in both the RPN, which seems to be the case according to the previous section, and the classification-regression branch on top of it, which we have yet to prove. We display a histogram of the repartition of the absolute error between all final detections (after rescoring and regression of the proposals by the Fast R-CNN branch) labeled as true positives (wrt the VeDAI metric) and their corresponding ground truth on Figure 7.

Only one patch is detected with more than 25 degrees of error and most of the other errors are negligible ( degrees). We note that Liu and Mattyus Liu and Máttyus [2015] also got good results of angle estimation on Munich as the orientation of a rectangle is easy enough to find our framework is cleaner as it does detection and angle regression simultaneously. The only big error ( degrees) comes from a more than questionable ground truth (visible on Figure 7 too). We draw the reader’s attention on the fact that an error of is not an error in our framework.

Figure 6: Histogram of the error of the angle prediction
Figure 7:

Confusion vector of the ”car” class detector

4.3.3 Rotation equivariance of the proposals

We do additional experiments on Munich test set by checking the performances of the detector on series of rotated images and see how the recall and precision evolve. It turns out there is little change in the performance when the images are rotated. A subset of the detections is presented Figure 8. The experiments show that our detector is, to a certain extent, provided that the RPN outputs correct proposals, rotation-equivariant: when one rotates the image the detections undergo the same transformation. One downside is that hard backgrounds remain hard backgrounds whatever the orientation.

Figure 8: Detections given by our Faster RER-CNN w.r.t. rotation

4.3.4 Confusion analysis of the ’car’ detector on VeDAI

We now focus on the car class detector extracted from our framework trained on VeDAI and display the results of a thorough miss and false positive analysis. The results of this study can be seen first on Figure 7

, where we show the repartition of all the false positive whose score are superior to 0.5 (it is hence a column vector part of a confusion matrix), Figure

10 shows the associated false positive patches (non-scaled and non-rotated). It is obvious that even most of the background mistakes (in pink) are just localization errors on other similar classes. With this kind of resolution, it is often hard even for a human to distinguish between say a pick-up and a car. We also display every car ground truth missed by our detector no matter the threshold applied in Figure 10.

Figure 9: False positives for the car detector with scores The color code used is vans: turquoise, trucks: dark blue, pickups: purple, others: yellow, backgrounds: pink
Figure 10: The only 7 ground truths labeled as cars never found by our car detector (out of 121 in fold01), numbers 4 and 5 do not resemble cars and must be annotator mistakes and numbers 2,6 and 7 are heavily occluded

It is now time to test it more thoroughly on several benchmarks and to compare it against different baselines and the state-of-the-art.

4.4 General detection results

4.4.1 A note on the evaluation protocol

As stated in the introduction comparing detection methods when their results and ground truth are not the same is not self-evident. The problem of intercepting a rectangle with another is obviously more difficult if you allow both rectangles to rotate. One solution would be to use PASCAL VOC vertical boxes as ground truth for both methods but it would be unfair towards our framework, which was trained on rotated boxes only (it would also be unfair towards Faster R-CNN to use the rotated boxes as ground truth even if they are evidently closer to the reality). Therefore in addition to VOC criteria for each framework on their respective ground truth we provide an absolute measurement, which is the VeDAI metric (defined in Razakarivony and Jurie [2015]), which only takes into account the position of the center of the predictions and thus treats each solution equitably.

4.4.2 VeDAI results

We have detection results on all folds and all classes of VeDAI. We first compared with related works on the car class (the only one present in Ogier Du Terrail and Jurie [2017]). Then we move to a detailed analysis per class for both frameworks. The only metric presented is the one described in Section 4.4.1. Some samples detection from VeDAI zoomed in can be seen in the center of Figure 11.

Figure 11: Example detections on all 3 datasets from left to right: Munich3k, VeDAI SII, GoogleEarth
method AP Recall@0.01FPPI
DPM Razakarivony and Jurie [2015]
SVM+HOG31 Razakarivony and Jurie [2015]
SVM+LBP Razakarivony and Jurie [2015]
SVM+LTP Razakarivony and Jurie [2015]
SVM+HOG31+LBP Razakarivony and Jurie [2015]
SVM Fusion AED (HOG) Razakarivony and Jurie [2014]
FCN Ogier Du Terrail and Jurie [2017] 77.80 3.3 31.04 11
Faster R-CNN (our implementation) 77.69 3.4 20.85 14
Ours 80.2 3.3 33.2 14
Table 1: Comparisons with related works on VeDAI

One can see in Table 1 that our work outperforms all others including the recent state of the art of Ogier Du Terrail and Jurie [2017] that uses hard example mining. The associated precision-recall curves for the car class are displayed Figure 12.

Table 2

shows that indeed a big enhancement is observed on all the vehicles (except boats, camping cars and planes where the two reported numbers are about equal) proving our point that our framework is beneficial for all vehicles. There is a huge variance on the plane class as there is very few of them. The variance is not negligible in all the other classes due to the very unequal difficulty of the different folds. For more explanation on confusion between classes, see section


class F. R-CNN FCNOgier Du Terrail and Jurie [2017] Ours cars 77.69 3.4 77.80 3.3 80.2 3.3 camp. cars 72.72 5.6 72.8 4 pickups 74.84 2.77 77.0 4.2 tractors 54.43 7.00 67.8 12.1 trucks 66.73 9.97 72.3 7.6 vans 69.93 9.13 74.1 11.0 boats 66.22 7.99 65.3 8.3 others 43.39 11.45 51.0 8.8 planes 77.85 34.45 77.4 32.7 mean 67.09 13.6 70.88 13.3
Table 2: Comparisons baselines VS our framework on VeDAI (10 folds)
Figure 12: PR curbs for the car class on VeDAI 10 folds

4.4.3 Munich 3K results

On Munich Leitloff et al. [2014] several tricks need to be applied to match the state of the arts such as using better networks, hard example mining or increasing the resolution of the feature map. We did not go that far as it was not the purpose of this article. Results presented in Table 3 speak for themselves.

method AP VOC@0.3 AP VEDAI
Faster R-CNN Ren et al. [2015] 85.59 85.68
Ours 87.14 87.32

Table 3: Comparison Faster RCNN vs framework on Munich3k

We observe that contrary to VeDAI, PASCALVOC@0.3 and VeDAI are in accordance because in both datasets most errors are not due to precision of localization of the boxes but to hard backgrounds and hard to find vehicles

222It is worth going through the supplementary to find detection examples of all datasets in color and magnified.


4.4.4 GoogleEarth results

We tested on this very small dataset only to show the generality of our solution for aerial imagery problems. However the number of images and vehicles is too small thus we have probably overfitted the data with either Faster R-CNN or Faster RER-CNN. Nonetheless again even if the disparities are small our framework is still leading in performances as can be seen on Table 4. Figure 13 shows one of the 5 test images and the errors w.r.t. a 0.5 detection threshold of both frameworks side by side.

Figure 13:

Vehicles missed by the detector are indicated with yellow circles and false positives in red. Left: Faster R-CNN. Right: Faster RER-CNN. This figure shows that although the problem remains difficult our framework has higher precision and recall and gives a lot more information about the vehicle’s templates.

method AP VOC@0.5 AP VEDAI
Faster R-CNN Ren et al. [2015] 84.81 87.37
Ours 88.39 88.53
Table 4: Comparison Faster RCNN vs framework on GoogleEarth

4.4.5 Running Times

We also add in table 5 speed comparisons of our framework w.r.t. the baseline. The bottleneck of our implementation being the IoU computations and NMS done with the CPUs (nms is on GPU for Faster) we expect to see a slowdown. It takes approximately twice as long to train and test, which is reasonable given that we get the angle and template.

method training step (s) inference(s)
Faster R-CNN Ren et al. [2015] 0.362 0.158
Ours 0.602 0.365
Table 5: Running times comparisons for one image 1024x1024 on an Intel i7-7700K CPU@8*4.2GHz with a GTX1080 (averaged on 100 runs)

5 Conclusions

We have presented a simple but strong alternative for the detection of small vehicles in aerial imagery built from Faster R-CNN architecture and based on rotation invariance of the classification. We saw how using rotated boxes in the RPN increases recall while maintaining precision allowing us to beat other alternatives on several benchmarks. It is our hope that this framework will serve as the new reference baseline on all the standard benchmarks from aerial imagery. Furthermore by simultaneously predicting the box and the angle we get for free a completely tight box matching the vehicle exact dimensions. It could therefore be used in a third stage to compare with a database of known vehicle dimensions (the ground to camera distance is known and the views are vertical) to get a more precise discrimination between vehicles, which is probably the hardest part. As a side note we also implemented a rotated version of RoI Align He et al. [2017] – it is completely straightforward to integrate inside our framework – and we are currently exploring it as a way to further increase the accuracy of the method at the expense of speed. 333This work was supported by ANRT through the CIFRE sponsorship No 2015/1008 and by SAFRAN Electronics and Defense.


  • Ali et al. [2012] Karim Ali, Francois Fleuret, David Hasler, and Pascal Fua. A real-time deformable detector. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(2):225–239, 2012.
  • Arthur and Vassilvitskii [2007] David Arthur and Sergei Vassilvitskii. K-means++: The advantages of careful seeding. In Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA ’07, pages 1027–1035, Philadelphia, PA, USA, 2007. Society for Industrial and Applied Mathematics. ISBN 978-0-898716-24-5. URL
  • Audebert et al. [2017] Nicolas Audebert, Bertrand Le Saux, and Sébastien Lefèvre. Segment-before-detect: Vehicle detection and classification through semantic segmentation of aerial images. Remote Sensing, 9(4), 2017. ISSN 2072-4292. doi: 10.3390/rs9040368. URL
  • Busta et al. [2017] Michal Busta, Lukas Neumann, and Jiri Matas. Deep textspotter: An end-to-end trainable scene text localization and recognition framework. In The IEEE International Conference on Computer Vision (ICCV), Oct 2017.
  • Chen et al. [2016] Dong Chen, Gang Hua, Fang Wen, and Jian Sun.

    Supervised Transformer Network for Efficient Face Detection

    , pages 122–138.
    Springer International Publishing, Cham, 2016. ISBN 978-3-319-46454-1. doi: 10.1007/978-3-319-46454-1˙8. URL
  • Chen et al. [2013] Xueyun Chen, Shiming Xiang, Cheng-Lin Liu, and Chun-Hong Pan. Vehicle detection in satellite images by parallel deep convolutional neural networks. In Pattern Recognition (ACPR), 2013 2nd IAPR Asian Conference on, pages 181–185. IEEE, 2013.
  • Dai et al. [2016a] Jifeng Dai, Kaiming He, and Jian Sun. Instance-aware semantic segmentation via multi-task network cascades. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016a.
  • Dai et al. [2016b] Jifeng Dai, Yi Li, Kaiming He, and Jian Sun. R-FCN: object detection via region-based fully convolutional networks. In Advances in Neural Information Processing Systems (NIPS), 2016b.
  • E. Sutherland and W. Hodgman [1974] Ivan E. Sutherland and Gary W. Hodgman. Reentrant polygon clipping. Commun. ACM, 17:32–42, Jan. 1974.
  • Eikvil et al. [2009] Line Eikvil, Lars Aurdal, and Hans Koren. Classification-based vehicle detection in high-resolution satellite images. ISPRS Journal of Photogrammetry and Remote Sensing, 64(1):65–72, 2009.
  • Everingham et al. [2015] Mark Everingham, S. M. Eslami, Luc Gool, Christopher K. Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. Int. J. Comput. Vision, 111(1):98–136, January 2015. ISSN 0920-5691. doi: 10.1007/s11263-014-0733-5. URL
  • Girshick [2015] Ross Girshick. Fast r-cnn. In International Conference on Computer Vision (ICCV), 2015.
  • Girshick et al. [2014] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Computer Vision and Pattern Recognition, 2014.
  • Gleason et al. [2011] Joshua Gleason, Ara V Nefian, Xavier Bouyssounousse, Terry Fong, and George Bebis. Vehicle detection from aerial imagery. In Robotics and Automation (ICRA), 2011 IEEE International Conference on, pages 2065–2070. IEEE, 2011.
  • Gonzalez et al. [2017] Diego Marcos Gonzalez, Michele Volpi, Nikos Komodakis, and Devis Tuia. Rotation equivariant vector field networks. In The IEEE International Conference on Computer Vision (ICCV), Oct 2017.
  • Green [1852] George Green. An essay on the application of mathematical analysis to the theories of electricity and magnetism. Journal für die reine und angewandte Mathematik, 44:356–374, 1852. URL
  • He et al. [2017] K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn. In The IEEE International Conference on Computer Vision (ICCV), Oct 2017.
  • Heitz and Koller [2008] Geremy Heitz and Daphne Koller. Learning spatial context: Using stuff to find things. In ECCV. European Conference on Computer Vision, October 2008.
  • Henriques and Vedaldi [2017] João F. Henriques and Andrea Vedaldi. Warped convolutions: Efficient invariance to spatial transformations. In ICML, 2017.
  • Jaderberg et al. [2015] Max Jaderberg, Karen Simonyan, Andrew Zisserman, and koray kavukcuoglu. Spatial transformer networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 2017–2025. Curran Associates, Inc., 2015. URL
  • Kembhavi et al. [2011] Aniruddha Kembhavi, David Harwood, and Larry S Davis. Vehicle detection using partial least squares. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(6):1250–1265, 2011.
  • LeCun et al. [1998] Y LeCun, L Bottou, Y Bengio, and P Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, November 1998.
  • Leitloff et al. [2014] Jens Leitloff, Dominik Rosenbaum, Franz Kurz, Oliver Meynberg, and Peter Reinartz. An operational system for estimating road traffic information from aerial images. Remote Sensing, 6(11):11315–11341, 2014. ISSN 2072-4292. doi: 10.3390/rs61111315. URL
  • Lin et al. [2014] T.-Y. Lin, M. Maire, S. Belongie, L. Bourdev, R. Girshick, J. Hays, P. Perona, D. Ramanan, C. L. Zitnick, and P. Dollár. Microsoft COCO: Common Objects in Context. European Conference on Computer Vision (ECCV, May 2014.
  • Liu and Máttyus [2015] Kang Liu and Gellért Máttyus. Fast multiclass vehicle detection on aerial images. IEEE Geosci. Remote Sensing Lett., 12:1938–1942, 2015.
  • Liu et al. [2016] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C. Berg. SSD: Single shot multibox detector. In ECCV, 2016.
  • Liu and Jin [2017] Yuliang Liu and Lianwen Jin. Deep matching prior network: Toward tighter multi-oriented text detection. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  • Ma et al. [2018] Jianqi Ma, Weiyuan Shao, Hao Ye, Li Wang, Hong Wang, Yingbin Zheng, and Xiangyang Xue. Arbitrary-oriented scene text detection via rotation proposals. IEEE Transactions on Multimedia, 2018.
  • Mundhenk et al. [2016] T. Nathan Mundhenk, Goran Konjevod, Wesam A. Sakla, and Kofi Boakye.

    A large contextual dataset for classification, detection and counting of cars with deep learning.

    In ECCV. European Conference on Computer Vision, October 2016.
  • Ogier Du Terrail and Jurie [2017] Jean Ogier Du Terrail and Frédéric Jurie. On the use of deep neural networks for the detection of small vehicles in ortho-images. In IEEE International Conference on Image Processing, Beijing, China, September 2017. URL
  • Razakarivony and Jurie [2015] Sébastien Razakarivony and Frédéric Jurie. Vehicle Detection in Aerial Imagery: A small target detection benchmark. Journal of Visual Communication and Image Representation, December 2015.
  • Razakarivony and Jurie [2014] Sébastien Razakarivony and Frédéric Jurie.

    Discriminative autoencoders for small targets detection.

    In 22nd International Conference on Pattern Recognition, ICPR 2014, Stockholm, Sweden, August 24-28, 2014, pages 3528–3533, 2014. doi: 10.1109/ICPR.2014.607. URL
  • Redmon and Farhadi [2017] Joseph Redmon and Ali Farhadi. Yolo9000: Better, faster, stronger. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  • Redmon et al. [2016] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  • Ren et al. [2015] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems (NIPS), 2015.
  • Shi et al. [2017] Baoguang Shi, Xiang Bai, and Serge Belongie. Detecting oriented text in natural images by linking segments. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  • Simonyan and Zisserman [2015] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
  • Tang et al. [2017] Tianyu Tang, Shilin Zhou, Zhipeng Deng, Huanxin Zou, and Lin Lei. Vehicle detection in aerial images based on region convolutional neural networks and hard negative example mining. Sensors, 17(2), 2017. ISSN 1424-8220. doi: 10.3390/s17020336. URL
  • Tsaftaris and Giuffrida [2016] Sotirios Tsaftaris and Mario Valerio Giuffrida.

    Rotation-Invariant Restricted Boltzmann Machine Using Shared Gradient Filters

    Theoretical Computer Science and General Issues. Springer International Publishing, 9 2016. ISBN 978-3-319-44780-3.
  • Viola and Jones [2001] Paul A. Viola and Michael J. Jones. Rapid object detection using a boosted cascade of simple features. In 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2001), with CD-ROM, 8-14 December 2001, Kauai, HI, USA, pages 511–518, 2001. doi: 10.1109/CVPR.2001.990517. URL
  • Xia et al. [2018] Gui-Song Xia, Xiang Bai, Jian Ding, Zhen Zhu, Serge Belongie, Jiebo Luo, Mihai Datcu, Marcello Pelillo, and Liangpei Zhang. Dota: A large-scale dataset for object detection in aerial images. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  • Zhou and Tuzel [2018] Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.