Various studies have established the relationships between bloom intensity, fruit load and fruit quality [14, 33]. Together with factors such as climate, bloom intensity is especially important to guide thinning, which consists of removing some flowers and fruitlets in the early growing season. Proper thinning directly impacts fruit market value, since it affects fruit size, coloration, taste and firmness.
Despite its importance, there has been relatively limited progress so far in automating bloom intensity estimation. Currently, this activity is typically carried out manually with the assistance of rudimentary tools. More specifically, it is generally done by inspecting a random sample of trees within the orchard and then extrapolating the estimates obtained from individual trees to the remainder of the orchard . As the example in Figure 1 illustrates, obstacles that hamper this process are: 1) manual tree inspection is time-consuming and labor-intensive, which contributes to making labor responsible for more than of apple production costs ; 2) estimation by visual inspection is characterized by large uncertainties and is prone to errors; 3) extrapolation of the results from the level of the inspected trees to the row or parcel level relies heavily on the grower’s experience; and 4) inspection of a small number of trees does not provide information about the spatial variability which exists in the orchard, making it difficult to develop and adopt site-specific crop load management strategies that could lead to optimal fruit quality and yield.
With the goal of introducing more accurate and less labor intensive techniques for the estimation of bloom intensity, machine vision systems using different types of sensors and image processing techniques have been proposed . Most existing methods, which are mainly based on simple color thresholding, have their applicability hindered especially by variable lighting conditions and occlusion by leaves, stems or other flowers .
Inspired by successful works using convolutional neural networks (CNNs) in multiple computer vision tasks, we propose a novel method for apple flower detection based on features extracted using a CNN. In our approach, an existing CNN trained for saliency detection is fine-tuned to become particularly sensitive to flowers. This network is then used to extract features from portraits generated by means of superpixel segmentation. After dimensionality reduction, these features are fed into a pre-trained classifier that ultimately determines whether each image region contains flowers or not. The proposed method significantly outperformed state-of-the-art approaches on four datasets composed of images acquired under different conditions.
Our main contributions are:
a novel CNN-based flower detection algorithm;
an extensive evaluation on a challenging dataset acquired under realistic and uncontrolled conditions;
an analysis of the generalization capability of the proposed approach on additional datasets previously unseen by the evaluated models.
The remainder of paper is organized as follows. Section 2 discusses the most relevant existing approaches for automated flower and fruit detection. Our proposed approach is described in Section 3, which also includes a description of three baseline comparison methods as well as implementation details. Experiments performed to evaluate the impact of specific design choices are described in Section 4, followed by an extensive comparison of our optimal model against the baseline methods on four different datasets. Our concluding remarks are presented in Section 5.
2 Related Work
While existing techniques employed for flower detection are based only on color information, methods available for fruit quantification exploit more modern computer vision techniques. For this reason, in this section we first review the most relevant works on automated flower detection, followed by a discussion of the relevant literature on fruit quantification. Moreover, to make this article self-contained and therefore accessible to a wider audience, we also provide a brief introduction to the fundamentals of CNNs.
2.1 Flower and fruits quantification
Aggelopoulou and colleagues presented in  one of the first works using computer vision techniques to detect flowers. That method is based on color thresholding and requires image acquisition at specific daylight times, with the presence of a black cloth screen behind the trees. Thus, although its reported error in predicted yield is relatively low (), such approach is applicable only for that controlled scenario.
Similar to the work of Thorp and Dierig  for identification of Lesquerella flowers, the technique described by Hočevar et al. in  does not require a background screen, but it is still not robust to changes in the environment. The image analysis procedure is based on hard thresholding according to color (in the HSL color space) and size features, such that parameters have to be adjusted whenever changes in illumination (daylight/night), in flowering density (high/low concentration) or in camera position (far/near trees) occur.
Horton and his team described in  a system for peach bloom intensity estimation that uses a different imaging approach. Based on the premise that the photosynthetic activity of this species increases during bloom period, the system relies on multispectral aerial images of the orchard, yielding an average detection rate of for test images. Similarly to the aforementioned methods, the applicability of this method also has the intrinsic limitation of considering only color/spectral information (thresholding near-infrared and blue bands), such that its performance is sensitive to changes in illumination conditions.
, classifying image pixels into leaves, almonds, trunk, ground and sky. Their method combines sparse autoencoders for feature extraction, logistic regression for label associations and conditional random fields to model correlations between pixels. Some other methods are based on support vector machine (SVM) classifiers that use information obtained from different shape descriptors and color spaces as input[11, 28]. Compared to existing methods for flower detection, these methods are more robust since morphological characteristics are taken into account. As many other shape-based and spectral-based approaches [34, 51, 52, 12], these techniques are, however, still limited by background clutter and variable lighting conditions in orchards .
Recent works on fruit quantification include the use of metadata information. Bargoti and colleagues in  built on  to propose an approach that considers pixel positions, orchard row numbers and the position of the sun relative to the camera. Similarly, Cheng et al.  proposed the use of information such as fruit number, fruit area, area of apple clusters and foliage area to improve accuracy of early yield prediction, especially in scenarios with significant occlusion. However, the inclusion of metadata is highly prone to overfitting, particularly when limited training data is available and the variability of the training set is hence low .
2.2 Deep learning
Following the success of Krizhevsky’s model 
in the ImageNet 2012 Challenge, deep learning methods based on CNNs became the dominant approach in many computer vision tasks. The architecture of traditional CNNs consists of a fixed-size input, multiple convolutional layers, pooling (downsampling) layers and fully connected layers. Winner of the ImageNet 2013 Classification task, the Clarifai model is one such network . Illustrated on the right side of Figure 2, it takes input image portraits of size pixels, which traverse a composition of convolutional layers (C1-C5) and fully connected layers (FC6-FC7 and the softmax FC8). Each type of layer plays a different role within the CNN architecture: while convolutional layers allow feature extraction, the latter fully connected layers act on this information to perform classification.
In computer vision and image processing, a feature corresponds to information that is meaningful for describing an image and its regions of interest for further processing. Feature extraction is therefore crucial in image analysis, since it represents the transition from pictorial (qualitative) to nonpictorial (quantitative) data representation . Rather than relying on hand-engineered features (e.g. HOG ), deep CNNs combine multiple convolutional layers and downsampling techniques to learn hierarchical features, which are a key factor for the success of these models . As described in , the convolutional layers C1-C2 learn to identify low-level features such as corners and other edge/color combinations. The following layers C3-C5 combine this low-level information into more complex structures, such as motifs, object parts and finally entire objects.
Traditional deep CNNs are composed of millions of learned parameters (over million in AlexNet ), such that large amounts of labeled data are required for network training. Deep learning models became feasible relatively recently, after the introduction of large publicly available datasets, of graphics processing units (GPUs), and of training algorithms that exploit GPUs to efficiently handle large amounts of data 
. Nevertheless, gathering domain specific training data is an expensive task. One alternative to reduce the required amount of labeled data is data augmentation, a technique proven to benefit the training of multiple machine learning models. It is typically performed by applying transformations such as translation, rotation and color space shifts to pre-labeled data.
In addition, transfer learning approaches such as fine-tuning have been investigated. Earlier layers of a deep network tend to contain more generic information (low-level features), which are then combined by the latter layers into task specific objects of interest. Thus, a network that can recognize different objects present in a large dataset must contain a set of low-level descriptors robust enough to characterize a wide range of patterns. Under this premise, fine-tuning procedures typically aim at adjusting the higher-level part of a network pre-trained on a large generic dataset, rather than training the full network from scratch. This greatly reduces the need for task-specific data, since only a smaller set of parameters has to be refined for the particular application.
At the classification side, most CNN architectures employ fully connected layers for final categorization. They determine which features are mostly correlated to each specific class employing a logistic regression classifier. For scenarios in which the output is binary, consistent albeit small improvements on popular datasets have been demonstrated by replacing the final CNN layer by a SVM classifier . SVM models tend to generalize better than logistic regression, since they target a solution that not only minimizes the training error, but also maximizes the margin distance between classes.
Following the success of CNNs on image classification tasks, the work of Girschick et al.  introduced the concept of region-based CNNs (R-CNN), outperforming by a large margin previous hand-engineered methods for object detection. In that work, a CNN is first pre-trained on a large auxiliary dataset (ImageNet) and then fine-tuned using a smaller but more specific dataset (PASCAL dataset for object detection). The Faster R-CNN proposed in  improved this model by replacing selective search  with the concept of Region Proposal Network (RPN), which shares convolutional layers with the classification network. Both modules compose a single, unified network for object detection.
Recent works adapt the Faster R-CNN for fruit detection. Bargoti and Underwood in  present a Faster R-CNN trained for detection of mangoes, almonds and apples fruits on trees. Stein et al. in  extended this model for tracking and localization of mangoes, combining it with a monocular multi-view tracking module that relies on a GPS system. Sa et al. in  applied the Faster R-CNN to RGB and near-infrared multi-modal images. Each modality was fine-tuned independently, with optimal results obtained using a late fusion approach. Still in the context of agricultural applications, CNNs have been also successfully used for plant identification from leaf vein patterns .
In summary, existing methods for flower identification are based on hand-engineered image processing techniques that work only under specific conditions. Color and size thresholding parameters composing these algorithms have to be readjusted in case of variations of lightning conditions, camera position with respect to the orchard (distance and angle), or expected bloom intensity. Recent techniques employed for fruit quantification exploit additional features and machine learning strategies, providing insights to further develop strategies for flower detection. Aiming at a technique for flower identification that is robust to clutter, changes in illumination and applicable for different flower species, we therefore propose a novel method in which an existing CNN trained for saliency detection is fine-tuned to become particularly sensitive to flowers.
3 Proposed Approach
In this section, we first describe the prediction steps performed by our method, i.e., the sequence of operations applied to an image in order to detect the presence of flowers. Subsequently, we describe the fine-tuning procedure carried out to obtain the core component of our model: a CNN highly sensitive to flowers. We conclude with a discussion of alternative flower detection approaches against which we evaluate our proposed method and a brief mention of relevant details regarding the implementation of our methods.
In the discussion that follows, we will refer to our proposed approach for flower detection as the CNN+SVM method. As illustrated in Figure 2, our CNN+SVM method consists of three main steps: i) computation of region proposals; ii) feature extraction using our fine-tuned CNN, which follows the Clarifai architecture ; and iii) final classification of each region according to the presence of flowers. The operations that comprise these steps are described in detail below. In our description, we make reference to Algorithm 1, which lists the operations performed by our method on each input image. The sensitivity of the method to specific design choices is detailed in Section 4.1.
1) Step 1 - Region proposals: The first step in the proposed method consists of generating region proposals by grouping similar nearby pixels into superpixels, which are perceptually meaningful clusters of variable size and shape (Line 3 of Algorithm 1). To this end, we use the simple linear iterative clustering (SLIC) superpixel algorithm. Currently one of the most widely-used algorithms for superpixel segmentation, it adapts k-means clustering to group pixels according to a weighted distance measure that considers both color and spatial proximity . For additional information on superpixel approaches, we refer the reader to the review provided in . The second leftmost image in Figure 2 illustrates the superpixels generated by the SLIC algorithm when applied to a typical image obtained in an orchard.
Although other approaches such as the Faster R-CNN  provide a unified architecture in which both region proposal and classification modules can be fine-tuned for a specific task, they have more parameters that need to be learned in a supervised manner. Since in most cases flowers are salient with respect to its surrounding background, an unsupervised, local-context based approach such as superpixel segmentation should be sufficient to obtain region proposals suitable for flower detection.
Once the image is segmented into superpixels, as Algorithm 1 indicates, we iterate over each superpixel in the image. Since the input size required by the Clarifai CNN model is , we first extract the smallest square portrait enclosing the superpixel under analysis (Line 5), which we denote . The output of this step is illustrated in the third leftmost image of Figure 2 for one superpixel. The background surrounding the superpixel of interest within a portrait is then padded with the training set mean, i.e. the average RGB color of all images composing the dataset (greenish color). Finally, the portrait is resized to (Line 6). The resulting region proposal, , is illustrated in the fourth image of Figure 2.
2) Step 2 - Feature extraction: In the feature extraction step (Line 7
), each of the portraits generated above is mean-centered and then evaluated individually by our CNN. The mean-centering step consists of subtracting from the portrait the same average training set RGB mean used for padding its background. This procedure is commonly employed to facilitate training convergence of deep learning models, since it ensures similarly ranged features within the network. For each input portrait, we collect as features the output of the rectified linear unit (ReLU) associated with the first fully connected layer (FC6). With a dimensionality of, the feature vector collected at this stage of the network encapsulates the hierarchical features extracted by layers , which contain the key information required for accurate classification.
3) Step 3 - Classification:
To classify each proposed region as containing a flower or not, we first perform principal component analysis (PCA) to reduce the feature dimensionality to a valuesuch that the new feature vector (Line 8). As demonstrated in our experimental evaluation in Section 4.1.1 a value of , which corresponds to approximately
of the original variance of the data, provides performance levels virtually identical to those of the original features. Finally, based on these features a pre-trained SVM model binary classifies superpixels according to the presence of flowers (Line9). Details on SVM training are provided in the next section.
3.1 Network fine-tuning and SVM training
Based on the techniques introduced by Girshick et al. in  and Zhao et al. in  for object and saliency detection, in our model an existing CNN architecture is made particularly sensitive to flowers by means of fine-tuning. In the work of Zhao et al. , the Clarifai model  was adopted as the starting point and fine-tuned for saliency detection. We further tuned Zhao et al.’s model for flower identification using labeled portraits from our training set, which we describe below.
The generation of training samples for network tuning takes place similarly to prediction. For each labeled image composing the training set, we computed region proposals according to Step 1 described above. Using these training examples, backpropagation training iterations were performed in order to minimize the network classification error. After fine-tuning, we computed the CNN features of the training examples, reduced their dimensionality to , and used them to train the SVM classifier.
Images of apple trees were collected using a camera model Canon EOS 60D under natural daylight illumination (i.e. uncontrolled environment). This dataset, which we refer to as AppleA, is composed of a total of images with resolution of pixels acquired under multiple angles and distances of capture. Figure 3 shows some images that comprise this dataset. For performance evaluation and learning purposes, the entire dataset was labeled using a MATLAB GUI in which the user selected only superpixels that contain parts of flowers in approximately half of its total area. As summarized in Table 1, the labeled images were randomly split into training and validation sets composed of and images, respectively. This corresponds to a total of training portraits (i.e. superpixels) and validation ones. The training examples were used to fine-tune the network and train the SVM as described above. The validation examples were used in the performance evaluation discussed in Sections 4.1 and 4.2.
|Portraits (i.e. superpixels)|
According to our labeling, only of the samples contain flowers (positives). Imbalanced datasets represent a problem for supervised machine learning approaches, since overall accuracy measures become biased towards recognizing mostly the majority class . In our case, that means the learner would present a bias towards classifying the portraits as negatives. To overcome this situation and increase the amount of training data, we quadrupled the number of positive samples using data augmentation. As illustrated in Figure 4, this was accomplished by mirroring each positive sample with respect to: (i) the vertical axis, (ii) the horizontal axis, and (iii) both axes.
Support vector machines are supervised learning models that search for a hyperplane that maximizes the margin distance to each class. This characteristic allows SVM models to generalize better than classifiers such as the ones based on logistic regression. For non-linearly separable data, kernel functions such as the popular radial basis function (RBF, orGaussian) are used. We refer to [5, 26] for further details on the formulation of SVMs.
Two main parameters control the performance of SVMs with a Gaussian kernel function, the regularization cost and the width of the Gaussian kernel . By regulating the penalty applied to misclassifications, the parameter controls the trade-off between maximizing the margin with which two classes are separated and the complexity of the separating hyperplane. The parameter regulates the flexibility of the classifier’s hyperplane. For both parameters, excessively large values can lead to overfitting.
3.2 Comparison Approaches
As has been noted in Section 2
, current algorithms for automated identification of flowers are mostly based on binarization by thresholding information from different color-spaces (typically RGB or HSV)[2, 48], occasionally combined with size filtering . We implement three alternative baseline approaches which reflect the state of the art in fruit/flower detection. The first implementation, which we call the HSV method, replicates the algorithm described by Hocevar and his team in . Images are binarized at pixel-level based on HSV color information, followed by filtering according to minimum and maximum cluster sizes.
We refer to the second baseline implementation as HSV+Bh. Similar to our proposed approach, the starting point for this method is the generation of superpixels using the SLIC algorithm. We then compute a
-bin histogram of each superpixel in the HSV color space, which has the advantage of dissociating brightness from chromaticity and saturation. Studies on human vision and color-based image retrieval have demonstrated that most of the color information is contained in the hue channel, with the saturation playing a significant role in applications where identifying white (or black) objects is important[18, 45]. In our experiments, we construct a single 1-D histogram consisting of bins, which corresponds to the concatenation of a 50-bin hue channel histogram, a 40-bin saturation histogram and a 10-bin value histogram. Afterwards, we use the Bhattacharyya distance  to compare each superpixel histogram against the histograms of all positive samples composing the training set. We compute the Bhattacharyya distance using a Gaussian kernel function, as formulated in [22, 25]. The average Bhattacharyya distance is taken as the likelihood that the superpixel includes a flower, and superpixels with distance lower than an optimal threshold are classified as flowers.
Since the technique described above is based on the average Bhattacharyya distance in the HSV color space, it makes no distinction between poorly and highly informative training sample features. Its ability to make accurate classification decisions is therefore limited in such complex feature spaces. Inspired by works on fruit quantification [11, 28], we extend this method by combining the same HSV histograms with an SVM classifier for apple flower detection. That is, rather than determining whether a superpixel contains a flower based on the Bhattacharyya distance, we train an SVM classifier that uses the HSV histograms as inputs. We call this approach the HSV+SVM method.
3.3 Implementation Details
4 Experiments and Results
Experiments were performed with three main goals. Our optimal CNN+SVM model extracts features from the CNN’s first fully connected layer (), reduces feature dimensionality to , and performs final classification using SVM. Thus, we first evaluated the impact of these specific design choices on the final performance of our method. We then compared it against the three baseline methods (HSV, HSV+Bh and HSV+SVM). Finally, we evaluated the performance of the proposed approach on previously unseen datasets to determine its generalization capability.
As described in Section 3.1, our datasets are severely imbalanced. In such scenarios, evaluations of performance using only accuracy measurements and ROC curves may be misleading, since they are insensitive to changes in the rate of class distribution. We therefore perform our analysis in terms of precision-recall curves (PR) and the corresponding score . Precision is normalized by the number of positives rather than the number of true negatives, so that false positive detections have the same relative weight as true positives. While the maximum score indicates the optimal performance of a classifier, the area under the respective PR curve (AUC-PR) corresponds to its expected performance across a range of decision thresholds, such that a model with higher AUC-PR is more likely to generalize better.
4.1 Analysis of design choices
In order to validate our design choices, we performed experiments to evaluate how the final performance of the classifier is affected by: (i) the dimensionality of the feature space; (ii) the point where features are collected from the CNN; and (iii) the type of input portrait.
4.1.1 Dimensionality analysis
PCA is one of the most widespread techniques for dimensionality reduction. It consists of projecting -dimensional input data onto a -dimensional subspace in such a way that this projection minimizes the reconstruction error (i.e. norm between original and projected data) 43].
In this application, the original dimensionality corresponds to the number of elements in the CNN vectors extracted from a fully connected layer, i.e., as represented for the two last layers in Figure 2. The first two columns of Table 2 show the reduced dimensionality of the feature vector and the corresponding ratio of the total variance of the -dimensional dataset that is retained at that dimensionality for layer . As the table indicates, the first most significant dimension alone already covers almost half of the total variance, and dimensions are sufficient to cover nearly of it.
|No. of dimensions||Variance ratio||Recall||Precision||AUC-PR|
We investigated then how samples are mapped into the lower dimensional feature space. Figure 5 shows the projections in dimensions and as well as dimensions and . These plots illustrate how the convolutional network maps the samples into a space where it is possible to differentiate between multiple clusters. With dim. as an abbreviation for dimension, let denote low dimensionality values and high values, respectively. The following clusters can be identified: flowers ( dim.1, dim.2); grass/floor ( dim.1, dim.2); branches/leaves ( dim.2); sky ( dim.3). This indicates that positive and negative samples are almost linearly separable even for projections of the original feature space.
To quantitatively assess how the classification performance is affected by the dimensionality of the feature space, we trained SVM classifiers for different numbers of dimensions. For each dimensionality, Table 2 presents the optimal performance metrics and corresponding AUC-PR. As expected, these results demonstrate that the impact of dimensionality on the optimal performance of our method is rather low. A very good performance is already obtained using a 2D feature space, with both score and AUC-PR only around and lower than the highest obtained values, respectively. In terms of optimal recall and precision, this is equivalent to missing extra positive samples out of , while including more false-positives out of . Moreover, the table shows that a dimensionality of is nearly optimal: the performance in terms of optimal score is only lower than the highest obtained value ( dimensions) and it is optimal in terms of AUC-PR.
Although in the discussion above we present results obtained using SVMs, such a high separability even for low dimensionalities indicates that the final prediction accuracy of our model is almost independent of the type of classifier employed. This conjecture is validated in the next subsection, where we demonstrate that the performance of our system does not change significantly by either including an additional fully connected layer to our CNN or by carrying out classification using the using network’s softmax layer directly.
4.1.2 Feature analysis
As explained in Section 3, after fine-tuning the model, we use it to extract features that allow the classification of superpixels according to the presence of flowers within them. Three combinations of features and classification mechanisms were investigated: (A) predict using solely the neural network, by means of its softmax output layer; (B) train a SVM classifier on features collected after the last fully connected layer (FC7); (C) train a SVM classifier on features collected after the first fully connected layer (FC6). Figure 6 shows the points where features are collected and how classification scores are computed using these features. Following the notation used in Figure 2, C1-C5 correspond to the convolutional layers of the fine-tuned Clarifai network, FC6-FC7 are the fully connected layers, and FC8 is the softmax layer.
For approaches B and C, features are collected from the output of the rectified linear units (ReLUs) located right after the respective fully connected layers. The same sequence of operations is performed for both methods B and C, i.e., the framework is the same regardless of whether the features are collected from the last (FC7) or first fully connected layer (FC6). Based on the results obtained in the previous section, for both cases dimensions are kept after PCA analysis.
Results obtained for classification on the validation set are summarized in Table 3 and Figure 7. As Figure 7 indicates, all three approaches show very similar performance. A closer inspection of Table 3 reveals that the SVM-based approaches slightly outperform the direct use of the neural network softmax layer both in terms of optimal score and AUC-PR. The performances obtained with methods B and C are very similar for both metrics. We therefore opted for method C, which uses features extracted from the earlier layer FC6 and provides slight increases in both optimal score and AUC-PR.
4.1.3 Different types of portraits
Using superpixels for region proposal computation and subsequent generation of portraits implies that our goal is to evaluate whether the superpixel itself is composed of flowers or not. In order to assess the influence of the local context surrounding the superpixel on the classification results, in addition to the approach based on replacing the region around the superpixel with the mean RGB value, two alternative approaches for portrait generation were considered. The first consists of retaining the unmodified image area surrounding the superpixel, whereas the second corresponds to blurring the background surrounding the superpixel with a low-pass filter. For all three cases, the portrait is mean-centered before being fed into the neural network. The three types of evaluated portraits are illustrated in Figure 8.
Figure 9 shows the PR curves obtained for each portrait type. The best performance is obtained with mean-padded portraits, a behavior explained by the existence of cases such as the ones illustrated in Figure 10. The superpixels highlighted in the images on the top row do not contain flowers in more than of their area and should therefore not be classified as flowers. However, these superpixels are surrounded by flowers, as depicted in the corresponding figures in the bottom row, and hence the approach of simply cropping a square region around the superpixel leads to cases in which the portrait contains a well-defined flower. As a consequence, features extracted from the CNN for the entire portrait will indicate the presence of flowers and therefore lead to high confidence false positives, which explain the non-maximal precision ratios in the upper-left part of the respective PR curve. This problem is eliminated by mean-padding the background.
4.2 Comparison against baseline methods
The analysis in Section 4.1 above validates the design choices of our optimal CNN+SVM model described in Section 3. That is, our optimal model uses mean-padded portraits and -dimensional features obtained from the FC6 layer of the CNN. In this section, we compare this optimal CNN+SVM model against the three baseline methods described in Section 3.2.
The parameters of all four methods were optimized using a grid search, as described in Section 3.3
. Optimization of the SVM hyperparameters based onscore resulted in the following values for regularization factor () and RBF kernel bandwidth (): HSV+SVM (); CNN+SVM (;
). For the HSV+Bh method, we performed an analogous grid search to optimize the standard deviation associated with the Gaussian kernel function, obtaining an optimal parameter of. For the HSV method, we performed an extensive grid search on our training dataset to selected an optimal set of threshold values. This procedure indicated that pixels composing flowers are distributed over the entire H range of , with optimal ranges of S within , V within , minimum size of pixels and maximum size of pixels.
Once the optimal parameters for all the classification models were determined, we evaluated the overall performance of each method using 10-fold cross-validation. All the samples composing the full AppleA dataset were combined and divided into 10 folds containing samples each. A total of 10 iterations was performed, in which each subsample was used exactly once as validation data.
The HSV method, which closely replicates existing approaches for flower detection, performs poorly in terms of both recall and precision. Such low performance is expected for methods that rely solely on color information. Since these techniques do not consider morphology or higher-level context to characterize flowers, they are very sensitive to changes in illumination and to clutter. Small performance improvements are obtained using the HSV+Bh method, which replaces pixel-wise hard thresholding by HSV histogram analysis at superpixel level, thereby incorporating a limited amount of context information into its classification decisions.
As illustrated by the results obtained with the HSV+SVM method, the use of an SVM classifier on the same HSV color features leads to dramatic improvements in both and AUC-PR ratios (around and , respectively). Rather than giving the same importance to all histogram regions, the SVM classifier is capable of distinguishing between poorly and highly informative features. However, as depicted in Figure 12, the precision of this method is still compromised by gross errors such as classifying parts of tree branches as flowers, since it does not take into account any morphological information.
The proposed approach (CNN+SVM) outperforms both baseline methods by extracting features using a convolutional neural network. Differently from the previous methods, the hierarchical features evaluated within the CNN take into account not only color but also morphological/spatial characteristics from each superpixel. Our results demonstrate the effectiveness of this approach, with significant improvements in both recall and precision ratios that culminate in an optimal score higher than and AUC-PR above for the evaluated dataset. Figure 3 shows examples of the final classification yielded by this method.
4.3 Performance on additional datasets
To evaluate the generalization capability of our method, we assessed its performance on three additional datasets, composed of images each and illustrated in Figure 13. We compare the results of our method with the performance of the best performing baseline approach (HSV+SVM). No dataset-specific adjustment of parameters is performed for our method nor for the baseline (HSV+SVM), i.e. both methods are assessed with the same optimal configuration obtained for the AppleA dataset.
Two of the additional datasets also correspond to apple trees, but with a blue background panel positioned behind the trees to visually separate them from other rows of the orchard, a common practice in agricultural vision systems. We denote the first dataset AppleB, which is composed of images with resolution acquired using a camera model GoPro HERO5. In this dataset there is a substantial number of occlusions between branches, leaves and flowers.
The second dataset, which we call AppleC, is composed of images with resolution acquired with a camera model JAI BB-500GE. In this dataset occlusions are less frequent but the saturation color component of the images is concentrated in a much narrower range of the spectrum than in the original AppleA dataset. The contrast between objects such as flowers and leaves is therefore significantly lower.
The third additional dataset contains images of peach flowers (we therefore call it Peach) with resolution acquired using a camera model GoPro HERO5. Peach blossoms show a noticeable pink hue in comparison to the mostly white apple flowers composing the training dataset. Additionally, images were acquired during an overcast day, such that in comparison to the training set (AppleA) the illumination is lower and the sky composing the background is gray instead of blue. Although the main scope of this work is on apple flower detection, we ultimately aim at a highly generalizable system that can be applied by fruit growers of different crops without the need for species-specific adjustments. In fruit orchards, each species of tree is typically constrained to specific areas. Hence, rather than differentiating between flower species, it is preferable to have a system that can distinguish between flowers and non-flower elements (e.g. leaves, branches, sky) regardless of species. Thus, this dataset represents a good evaluation of detection robustness.
Transfer learning steps
For all three additional datasets, both feature extraction and final classification were performed using the same parameters obtained by training with the AppleA dataset, without any dataset specific fine-tuning. Our transfer learning strategy relies solely on generic pre-processing operations that approximate the characteristics of the previously unseen images to those of the training samples.
Our first pre-processing step consists of removing the different backgrounds of the additional datasets. Whether the background is composed of a blue panel (AppleB and AppleC) or a gray sky (Peach), background identification for subsequent subtraction can be performed by means of texture analysis. For each image we compute the corresponding local entropy, which is then binarized using Otsu’s threshold  to identify low texture clusters. We then apply morphological size filtering to the binarized image and model the background as a multimodal distribution.
To model the background, we compute the RGB-mean of the largest (in terms of number of pixels) low texture clusters to build a -modal reference set. The likelihood that remaining low texture clusters belong to the background is estimated as the Euclidean distance between their means and the nearest reference in the RGB space. This metric allows differentiating between low texture components composing the background from the ones composing flowers, without any dataset specific color thresholding. For the AppleB and Peach datasets we adopted a bimodal distribution, where the modes correspond to the blue panel/gray sky and trunk/branches. Since the blue panel in the background of images composing the AppleC dataset is reflective, shadows are visible and therefore we included a third mode to automatically filter these undesired elements out. Automatically determining the number of background components is part of our future work.
Afterwards, histogram equalization and histogram matching are performed on the saturation channel of each image. While equalization aims at spreading the histogram components, histogram matching consists in approximating its distribution to the characteristic form of the training set channel distribution . Finally, to mitigate the effects of illumination discrepancies, we subtract the difference between the mean of the value channel components in the input image and in the training set.
Figure 14 shows the PR curves summarizing the performance on these datasets of our method (CNN+SVM) in comparison with the best performing baseline approach (HSV+SVM). The proposed method provides AUC-PR above for all datasets, significantly outperforming the baseline method. Since the HSV+SVM method relies solely on color information, its results are acceptable only for the AppleB dataset, the one that most closely resembles the training dataset. Its performance is notably poor for the Peach set, as this species differs to a great extent from apple flowers in terms of color. A large performance difference is also evident for the AppleC dataset, in which flowers and leaves share more similar color components than in the training set. Table 5 shows that the proposed approach also outperforms the baseline by a large margin in terms of optimal
score and the corresponding precision and recall values.
Additionally, it is noteworthy that a large number of superpixels classified as false positives by our proposed approach (CNN+SVM) correspond to regions where flowers are indeed present, but compose less than of the corresponding superpixel total area. This is illustrated in Figure 15, which contains examples for the three additional datasets. In other words, the sensitivity of the feature extractor to the presence of flowers is very high and the final performance would be improved if the region proposals were more accurate.
In this work, we introduced a novel approach for apple flower detection, which is based on deep learning techniques that represent the state of the art for computer vision applications. In comparison with existing methods, which are mainly based solely on color analysis and have limited applicability in scenarios involving changes in illumination or occlusion levels, the hierarchical features extracted by our CNN effectively combine both color and morphological information, leading to significantly better performances for all the cases under consideration. Experiments performed on four different datasets demonstrated that the proposed CNN-based model allows accurate flower identification even in scenarios of different flower species and illumination conditions, with optimal recall and precision rates near even for datasets significantly dissimilar from the training sequences.
As part of our future work, we intend to explore existing datasets and state-of-the-art models for semantic image segmentation. Particularly successful strategies consist of end-to-end architectures that, without external computation of region proposals, generate pixel dense prediction maps for inputs with arbitrary size [41, 8, 32].
Moreover, similar to the approach proposed in  for fruits, we intend to extend our module for flower tracking and localization based on probabilistic approaches that use the estimated motion between frames (e.g. particle filtering ) to predict the location of flowers. To extend the applicability of our model to the detection of fruitlets as well as other flower species, we will consider additional transfer learning approaches such as data augmentation by affine transformations and the use of external datasets.
-  R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, and S. Süsstrunk. SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(11):2274–2281, 2012.
-  A. D. Aggelopoulou, D. Bochtis, S. Fountas, K. C. Swain, T. A. Gemtos, and G. D. Nanos. Yield prediction in apple orchards based on image processing. Precision Agriculture, 12(3):448–456, 2011.
-  S. Bargoti and J. Underwood. Deep Fruit Detection in Orchards. Australian Centre for Field Robotics, pages 1–8, 2016.
-  S. Bargoti and J. Underwood. Image Segmentation for Fruit Detection and Yield Estimation in Apple Orchards. Journal of Field Robotics, 00(0):1–22, 2016.
-  A. Ben-Hur and J. Weston. A user’s guide to support vector machines. Methods in molecular biology (Clifton, N.J.), 609:223–239, 2010.
On a measure of divergence between two statistical populations defined by their probability distributions.Bulletin of the Calcutta Mathematical Society, 35:99–109, 1943.
-  A. Carpenter. Cusvm: a cuda implementation of support vector classification and regression. Update, 15(10):1–9, 2009.
-  L. C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence, PP(99):1–1, 2017.
-  H. Cheng, L. Damerow, Y. Sun, and M. Blanke. Early Yield Prediction Using Image Analysis of Apple Fruit and Tree Canopy Features with Neural Networks. Journal of Imaging, 3(1):6, 2017.
N. Dalal and B. Triggs.
Histograms of oriented gradients for human detection.
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), volume 1, pages 886–893 vol. 1, June 2005.
-  J. Das, G. Cross, C. Qu, A. Makineni, P. Tokekar, Y. Mulgaonkar, and V. Kumar. Devices, systems, and methods for automated monitoring enabling precision agriculture. In IEEE International Conference on Automation Science and Engineering (CASE), pages 462–469. IEEE, 2015.
-  U.-O. Dorj, M. Lee, and S.-s. Yun. An yield estimation in citrus orchards via fruit detection and counting using image processing. Computers and Electronics in Agriculture, 140:103–112, 2017.
-  T. Fawcett. An introduction to ROC analysis. Pattern Recognition Letters, 27(8):861–874, 2006.
-  C. G. Forshey. Chemical Fruit Thinning of Apples, volume 116. New York State Agricultural Experiment Station, Geneva, N.Y., 1986.
-  R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 580–587, 2014.
-  A. Gongal, S. Amatya, M. Karkee, Q. Zhang, and K. Lewis. Sensors and systems for fruit detection and localization: A review. Computers and Electronics in Agriculture, 116:8–19, 2015.
-  A. Gongal, A. Silwal, S. Amatya, M. Karkee, Q. Zhang, and K. Lewis. Apple crop-load estimation with over-the-row machine vision system. Computers and Electronics in Agriculture, 120:26–35, 2016.
-  R. C. Gonzalez and R. E. Woods. Digital Image Processing (3rd Edition). Prentice-Hall, Inc., Upper Saddle River, NJ, USA, 2006.
-  G. L. Grinblat, L. C. Uzal, M. G. Larese, and P. M. Granitto. Deep learning for plant identification using vein morphological patterns. Computers and Electronics in Agriculture, 127:418–424, 2016.
-  Y. Guo, Y. Liu, A. Oerlemans, S. Lao, S. Wu, and M. S. Lew. Deep learning for visual understanding: A review. Neurocomputing, 187:27–48, 2016.
-  G. E. Hinton. Learning multiple layers of representation. Trends in Cognitive Sciences, 11(10):428–434, 2007.
-  A. Hoak, H. Medeiros, and R. J. Povinelli. Image-based multi-target tracking through multi-bernoulli filtering with interactive likelihoods. Sensors, 17(3):501, 2017.
-  M. Hočevar, B. Širok, T. Godeša, and M. Stopar. Flowering estimation in apple orchards by image analysis. Precision Agriculture, 15(4):466–478, 2014.
-  R. Horton, E. Cano, D. Bulanon, and E. Fallahi. Peach Flower Monitoring Using Aerial Multispectral Imaging. 2016 ASABE International Meeting, 2016.
-  R. Hoseinnezhad, B. N. Vo, B. T. Vo, and D. Suter. Visual tracking of numerous targets via multi-Bernoulli filtering of image data. Pattern Recognition, 45(10):3625–3635, 2012.
-  C.-W. Hsu, C.-C. Chang, and C.-J. Lin. A Practical Guide to Support Vector Classification. BJU international, 101(1):1396–400, 2008.
-  C. Hung, J. Nieto, Z. Taylor, J. Underwood, and S. Sukkarieh. Orchard fruit segmentation using multi-spectral feature learning. IEEE International Conference on Intelligent Robots and Systems, pages 5314–5320, 2013.
-  W. Ji, D. Zhao, F. Cheng, B. Xu, Y. Zhang, and J. Wang. Automatic recognition vision system guided for apple harvesting robot. Computers and Electrical Engineering, 38(5):1186–1195, 2012.
-  Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093, 2014.
-  K. Kapach, E. Barnea, R. Mairon, Y. Edan, and O. B. Shahar. Computer vision for fruit harvesting robots state of the art and challenges ahead. International Journal of Computational Vision and Robotics, 3(1/2):4, 2012.
-  Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. Nature, 521(7553):436–444, 2015.
-  G. Lin, A. Milan, C. Shen, and I. Reid. RefineNet: Multi-path refinement networks for high-resolution semantic segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
-  H. Link. Significance of flower and fruit thinning on fruit quality. Plant growth regulation, 31(1):17–26, 2000.
-  R. Linker, O. Cohen, and A. Naor. Determination of the number of green apples in RGB images recorded in orchards. Computers and Electronics in Agriculture, 81:45–57, 2012.
-  O. Marques. Practical image and video processing using MATLAB. John Wiley & Sons, 2011.
-  M. Mohri. Foundations of Machine Learning - Book. The MIT Press, 2012.
-  R. J. Mozhdehi and H. Medeiros. Deep convolutional particle filter for visual tracking. In International Conference on Image Processing (ICIP), 2017.
-  N. Otsu. A threshold selection method from gray-level histograms. IEEE Transactions on Systems, Man, and Cybernetics, 9(1):62–66, Jan 1979.
-  S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. Nips, pages 1–10, 2015.
-  I. Sa, Z. Ge, F. Dayoub, B. Upcroft, T. Perez, and C. McCool. DeepFruits: A Fruit Detection System Using Deep Neural Networks. Sensors, 16(8):1222, 2016.
-  E. Shelhamer, J. Long, and T. Darrell. Fully convolutional networks for semantic segmentation. CoRR, abs/1605.06211, 2016.
-  S. Singh, M. Bergerman, J. Cannons, B. Grocholsky, B. Hamner, G. Holguin, L. Hull, V. Jones, G. Kantor, H. Koselka, G. Li, J. Owen, J. Park, W. Shi, and J. Teza. Comprehensive Automation for Specialty Crops: Year 1 results and lessons learned. Intelligent Service Robotics, 3(4):245–262, 2010.
-  L. I. Smith. A tutorial on Principal Components Analysis Introduction. Statistics, 51:52, 2002.
-  M. Stein, S. Bargoti, and J. Underwood. Image Based Mango Fruit Detection, Localisation and Yield Estimation Using Multiple View Geometry. Sensors, 16(11):1915, 2016.
-  M. A. Stricker and M. Orengo. Similarity of color images. In Storage and Retrieval for Image and Video Databases III, volume 2420, pages 381–393. International Society for Optics and Photonics, 1995.
-  D. Stutz, A. Hermans, and B. Leibe. Superpixels: an evaluation of the state-of-the-art. Computer Vision and Image Understanding, 166:1–27, 2018.
-  Y. Tang. Deep learning using linear support vector machines. In International Conference on Machine Learning: Challenges in Representation Learning Workshop, 2013.
-  K. R. Thorp and D. A. Dierig. Color image segmentation approach to monitor flowering in lesquerella. Industrial Crops and Products, 34(1):1150–1159, 2011.
-  J. R. R. Uijlings, K. E. A. Van De Sande, T. Gevers, and A. W. M. Smeulders. Selective search for object recognition. International Journal of Computer Vision, 104(2):154–171, 2013.
Issues in mining imbalanced data sets - a review paper.
in Proceedings of the Sixteen Midwest Artificial Intelligence and Cognitive Science Conference, 2005, pages 67–73, 2005.
-  J. P. Wachs, H. I. Stern, T. Burks, and V. Alchanatis. Low and high-level visual feature-based apple detection from multi-modal images. Precision Agriculture, 11(6):717–735, 2010.
-  Q. Wang, S. Nuske, M. Bergerman, and S. Singh. Automated Crop Yield Estimation for Apple Orchards. Proceedings of International Symposium of Experimental Robotics, (Iser), 2012.
-  S. C. Wong, A. Gatt, V. Stamatescu, and M. D. McDonnell. Understanding data augmentation for classification: when to warp? In Digital Image Computing: Techniques and Applications (DICTA), 2016 International Conference on, pages 1–6. IEEE, 2016.
-  J. Yosinski, J. Clune, Y. Bengio, and H. Lipson. How transferable are features in deep neural networks? In Advances in neural information processing systems, pages 3320–3328, 2014.
-  M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision (ECCV), pages 818–833. Springer, 2014.
-  R. Zhao, W. Ouyang, H. Li, and X. Wang. Saliency detection by multi-context deep learning. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 07-12-June:1265–1274, 2015.