A Novel Technique Combining Image Processing, Plant Development Properties, and the Hungarian Algorithm, to Improve Leaf Detection in Maize

Manual determination of plant phenotypic properties such as plant architecture, growth, and health is very time consuming and sometimes destructive. Automatic image analysis has become a popular approach. This research aims to identify the position (and number) of leaves from a temporal sequence of high-quality indoor images consisting of multiple views, focussing in particular of images of maize. The procedure used a segmentation on the images, using the convex hull to pick the best view at each time step, followed by a skeletonization of the corresponding image. To remove skeleton spurs, a discrete skeleton evolution pruning process was applied. Pre-existing statistics regarding maize development was incorporated to help differentiate between true leaves and false leaves. Furthermore, for each time step, leaves were matched to those of the previous and next three days using the graph-theoretic Hungarian algorithm. This matching algorithm can be used to both remove false positives, and also to predict true leaves, even if they were completely occluded from the image itself. The algorithm was evaluated using an open dataset consisting of 13 maize plants across 27 days from two different views. The total number of true leaves from the dataset was 1843, and our proposed techniques detect a total of 1690 leaves including 1674 true leaves, and only 16 false leaves, giving a recall of 90.8

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 3

page 6

11/21/2016

The subset-matched Jaccard index for evaluation of Segmentation for Plant Images

We describe a new measure for the evaluation of region level segmentatio...
04/05/2019

Leaf segmentation through the classification of edges

We present an approach to leaf level segmentation of images of Arabidops...
03/25/2018

Synthesizing Skeletons for Reactive Systems

We present an analysis technique for temporal specifications of reactive...
03/23/2021

Applying graph matching techniques to enhance reuse of plant design information

This article investigates how graph matching can be applied to process p...
06/26/2018

Multi-Task Deep Convolutional Neural Network for the Segmentation of Type B Aortic Dissection

Type B aortic dissection (TBAD) is a rare but life threatening disease. ...
06/29/2021

Machine learning for plant microRNA prediction: A systematic review

MicroRNAs (miRNAs) are endogenous small non-coding RNAs that play an imp...
05/07/2018

Near-drowning Early Prediction Technique Using Novel Equations (NEPTUNE) for Swimming Pools

Safety is a critical aspect in all swimming pools. This paper describes ...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Agriculture is the backbone of the world economy, and a significant number of countries’ economies are highly dependent on it. Plant diseases, undesirable growth, nutritional deficiency, and disorder in plants not only affect the quality and quantity of agricultural profits, but also play a vital role in food crises. Thus, monitoring the condition of plants is a fundamental step in successful cultivation of crops and plant breeding. Indeed, plant breeding, with the assistance of high-throughput phenotyping, is helping to cultivate crops under extreme climate, and to create novel plant varieties [30, 8]. This can ultimately contribute towards a greater quantity and quality of food for feeding the ever-growing population. Until recently, the observation and analysis of plant growth, disease detection, and phenotypic properties, were done entirely manually by experts, in a time intensive, and largely intuitive fashion. Thus, the potential of using image processing in plant research to automate phenotypic inspection has long been recognised as an important step forward [32]. Now, the food industry ranks among the top industries using image processing [16] to help evaluate food quality and consistency while eliminating the subjectivity of manual inspections [13].

Computer vision can be used to extract useful information from plant images [31], and to identify phenotypic traits throughout a plant’s life [12]. Various types of digital cameras are used to acquire richer information about plants of interest [2, 35, 26]

. Extracting meaningful phenotypes from plant image sequences is broadly classified into two categories: holistic and component-based

[11]. Holistic plant phenotyping considers the whole plant as a single object and gives metrics that quantify the basic geometric properties of the plant (e.g. height, width, plant aspect ratio, etc). Component-based analysis tries to identify the specific distinguishing components of a plant (leaves, stem, flower etc), their positions, and sizes [10].

Problem overview: Our goal is to reconstruct and predict maize plant growth properties, topology, numbers and positions of leaves, and their emergence, from indoor time sequence plant images. Maize is a globally-grown annual cereal crop, and one of the top three most important cereal crops in the world [27, 17, 14]. Therefore, maize has a vital role to play in our agricultural economy, and automated prediction of maize plant growth, topology, components, disease, and architecture is important.

Automatic determination of plant topology and architecture is highly dependent on accurate plant skeletons. Skeletons are a thin, sometimes one-pixel-wide, representation of any object that represents an object’s topology; it is also often useful for feature extraction. After fifty years of research, there is still no perfect skeletonization algorithm for each individual area of application

[21]. Obtaining accurate plant skeletons from images is a difficult problem, as they are sensitive to small changes leading to extraneous branches, and incorrectly joined segments (errors in topology) [15]. Extra branches, also called spurs, are especially common in plant skeletons and form due to noise in images [6]. Spurs are often incorrectly interpreted as leaves.

The complex geometry of plants, their thin structures, and missing information due to self-occlusion, make skeleton extraction and pruning extremely challenging tasks [7]. Occlusion can occur frequently in 2D images, both partially, and totally. Partial occlusion occurs when a part of a component is occluded from an image, e.g. part of a leaf hiding its branching point. Total occlusion occurs when a leaf is totally obscured by other components of the plant. For total occlusion, there is no obvious way to tell from the image itself that a component is present.

Contribution: This study proposes a novel technique to improve detection of leaves and topology in maize. The proposed method initially obtains the plant skeleton with image processing algorithms, and then it applies statistics regarding maize development available in literature to the skeleton to improve the predicted topology. Lastly, the Hungarian algorithm is applied to match the leaves in each day’s image with those in the previous and next day’s images to match skeleton components between days. The Hungarian algorithm, also known as the Munkres algorithm, is an algorithm on weighted, undirected graphs that determines the one-to-one mapping between two given sets of vertices where the matched edges have the mathematically smallest combined weight [20]

. Despite there being an exponential number of such mappings, the mathematically optimal solution can be found in polynomial time. This can be used to find the best matching between leaves in one image of a plant with those of the same plant on another day, ideally matching the same leaves together

[19]. This can both discard leaves detected from erroneous spurs, and also properly predict components even if they are completely occluded. In this way, the analyses are not completely dependent on the skeletonization techniques and pruning strategies. This contributes not only to leaf counting, but also to inference of plant topology.

While the analysis was carried out using images of maize, certain aspects of the analysis would be generalizable to time sequence images from other plant species. For example, the use of the Hungarian algorithm to match different components of the same plant between days is an interesting approach generally. Furthermore, the use of apriori knowledge regarding plant development in a given species can be used to override the classification of components identified by the computer vision algorithms.

2 Dataset

An open dataset was used from the University of Nebraska-Lincoln [24]. This dataset, called UNL-CPPD-I, has images of 13 different maize plants (with different genotypes). Plants were imaged once per day for 27 days using the visible light camera of the UNL Lemnatec Scanalyzer 3D high-throughput phenotyping facility [10]. Images were taken from two different orthogonal side views at 0 degrees and 90 degrees; denoted by view-0 image, and view-90 image, respectively. The 0-degree orientation is not always fixed across days, thus the best view for segmenting leaves differs from day to day even for the same plant.

Maize has multiple stages of development; vegetative, transitional, reproductive, and seed [5]. All images in the dataset are only from the vegetative stage. During this stage, the tip of the main stem is short, leaves are arranged in an alternate phyllotaxy (each leaf develops on the opposite side of the previous leaf, forming a left-right alternating pattern), and leaves arise at a certain distance from the top of the stem. A limited number of axillary buds can develop, but ears do not develop until further stages. Hence, at this stage, the topology is dominated by the alternating leaf pattern.

The dataset also contains ground-truth annotated images with the visible leaves marked. Note that if a leaf is not visible in a given image, then it is not annotated in the ground-truth. This is immediately evident because the number of annotated leaves from the two views can differ substantially. While this is advantageous from the perspective of identifying leaves on an individual image, it does hinder the evaluation of leaf identification procedures that try to identify leaves even if they are occluded, which is our desired goal.

The imaging started on October 10, 2015, 2 days after seed planting. The dataset contains 700 images. A detailed description about the imaging setup, dataset organization, and their genotypes is given in [10].

3 Methodology

This section discusses the methods, and algorithm implementations. Each phase is described in a subsection, and are image segmentation, view selection, plant skeletonization, a threshold-based pruning method, spur removal based on statistics from literature on maize, and the use of the Hungarian matching algorithm to improve leaf counting. Certain thresholds calculated within are appropriate for indoor time-sequence images of maize, and would likely need to be adjusted for other species and setups. However, the process used to derive the thresholds can be applied elsewhere, along with the aforementioned generalizable elements.

3.1 Segmentation

The first step is obtaining the plant area from the available images with image segmentation techniques. Background subtraction was used to extract the foreground, which, in this case, is the plant itself. Background subtraction involves removing the background of the image, which consists of the imaging chambers of the Lemnatec Scanalyzer 3D high-throughput plant phenotyping system. This has a fixed background that remains static over the period of interest for the image sequence [10] (Figure (a)a). Then, the Otsu thresholding algorithm [25] was used on the grayscale image of the foreground image to obtain the segmented image. Figure 1 shows an example of Plant_001-9 at day 15 from view-90 ((b)b), its foreground after background subtraction ((c)c), and the resulting segmented plant image ((d)d).

(a)
(b)
(c)
(d)
Figure 1: (a) Background. (b) Plant_001-9 at day 15 from view-90. (c) Plant foreground. (d) Segmented image.

Preliminary inspection of foreground image histograms showed that any threshold smaller than would label background pixels as foreground (Figure (b)b). However, for some images, the detected threshold was smaller than due to the light affecting the background. Therefore, the threshold used was the larger of and that detected by the Otsu algorithm. At this stage, there were some images where these thresholds were capturing some pixels from the plant tub (Figure (e)e). Hence, another level of thresholding was performed by calculating the excess green () of the foreground image. The initially-thresholded pixels of the excess green image was thresholded again with a threshold value of the maximum value between , and the minimum value between Otsu returned threshold, and ; which is , where is the Otsu threshold, and is the final threshold value. Figure (f)f shows how the second level thresholding removed the tub pixels.

(a)
(b)
(c)
(d)
(e)
(f)
Figure 2: (a) A plant image at day 11 [10]. (b) Segmentation with Otsu returned threshold smaller than . (c) Segmentation with threshold value . (d) A plant image at day 25 [10]. (e) The first thresholding with the tub pixels (the yellow line near the root is from the tub area). (f) After second thresholding, the tub pixels are removed.

3.2 View selection

For each plant and day, a view selection process was applied to select the view (either 0 degrees or 90 degrees) where the leaves, stem, and buds are most clearly visible. It is best to analyze the plant captured from the viewpoint at which as many leaves as possible are visible. Hence, we compute the area of the convex hulls of the binarized plant images of both views (a similar process was also used in

[10]). The view with the largest convex hull was selected. For example, Figures (a)a and (b)b show the binary images of a maize plant on day 24, from both views. It is apparent that the area of the convex hull at side view 90 is higher.

(a)
(b)
Figure 3: (a) A segmented plant image at day 24 from view-0, and (b) view-90. The convex hull is outlined in red.

3.3 Skeletonization

Skeletons are typically computed by either morphological thinning, computing the medial axis, geometric methods, or the fast marching distance transform. Morphological thinning takes a region, and gradually reduces the boundaries of that region until they are only separated by one pixel. The results of morphological thinning are similar to those of the medial axis transformation, which finds medial points by determining the set of points that are local maxima in terms of distance from the edge of the shape. Although these methods are straightforward, they require intensive heuristics to ensure connectivity of the skeleton in the case of complex dynamic structures such as plants

[10].

After extensive preliminary testing, it was observed that different skeletonization algorithms work better in specific ranges of days since emergence. This preliminary testing was measured based on the leaf count, spur count, and visually how accurately the skeleton branching points and tips are positioned. Branching points are the starting point of the leaf from the plant stem, and end-points are the leaf tips.

Two different skeletonization methods were applied on different time intervals from emergence. The first skeletonization method is the fast parallel thinning algorithm [36]. This approach works by making successive passes of the image and removing pixels on object borders; this continues until no more pixels can be removed. The image is correlated with a mask that assigns each pixel a number in the range corresponding to each possible pattern of its 8 neighbouring pixels. A lookup table is then used to assign each pixel a value of 0, 1, 2, or 3, which are selectively removed during the iterations [36]. This approach has the advantages of contour noise immunity and a good effect in thinning crossing lines [9]. Some of the earlier days’ images have branches where lines representing leaves cross the stem. From a 1-pixel wide skeleton, a branching point was determined as a pixel with 3 or more neighbours, and a leaf end-point was the pixel with one neighbour. In earlier days’ images, there are frequently overlapping lines in the skeleton, and the prediction needs to be able to properly classify the portions of the crossed lines [28]. In our testing, the fast parallel algorithm performs better at classifying these crossed lines. Hence, for skeletons from days 1 through 10 from emergence, this approach was applied.

However, this process causes numerous branching points in the skeleton near skeleton points that have more than three neighbours [22], which occurs often in later days. Most of the images at later days have occlusions and curvatures in some leaves. Thus, an algorithm that is better in terms of noise sensitivity, and also preserves topologic and geometric connectivity would be better for these images. Preliminary testing showed that the 3D medial surface/axis thinning algorithm performed well at resolving leaf occlusion and leaf curvature. Therefore, images from day 11 onwards were skeletonized by the 3D medial surface axis thinning algorithm [23]. This method uses an octree data structure to examine a neighbourhood of a pixel. The algorithm proceeds by iteratively sweeping, and removing pixels at each iteration until the image stops changing. Each iteration consists of two steps: first, a list of candidates for removal is assembled. Then pixels from this list are rechecked sequentially, to better preserve connectivity of the image [23]. The medial axis of an object is the set of all points that have more than one closest point on the object’s boundary. It ultimately produces a 1-pixel wide skeleton preserving the connectedness as the original object.

3.4 Skeleton pruning

The process of eliminating spurs to overcome skeleton instability is known as pruning [29]. A fixed-threshold-based pruning method could be used on all maize images in an attempt to remove skeleton spurs [6]. However, this resulted in many false negatives. Therefore, a pruning method called discrete skeleton evolution [3] was applied on all of the plant skeletons. The fundamental theory of this process is to remove skeleton end-branches that have the smallest relevance for shape reconstruction. It calculates the relevance of branches as their contribution to shape reconstruction by calculating a weight for every edge between an end-point and a branching point iteratively, and any such edge having a weight less than a threshold is deleted. The weight is calculated with the following formula, ; where is the current area of the skeleton, and is the area of the edge; the threshold of was used from [3] (area is the number of pixels in that object). This is appropriate because a small weight indicates that the edge has a negligible influence on the skeleton reconstruction, and the skeleton can be reconstructed without this branch in nearly the same fashion as the reconstruction with it [3].

3.5 Eliminating skeleton spurs with heuristics and statistics regarding maize development

By attempting to detect and remove spurs using the thresholding technique from Section 3.4, there is a risk that it will incorrectly identify some components as a spur and remove it. Therefore, the following are used to decide between a true leaf, and a spur.

3.5.1 Removing one pixel long spurs

A general threshold-based edge pruning was performed on the upper area. In the images, the root of the plants is always within pixels from the bottom of the image. A pixel edge pruning was performed to remove any skeleton spurs that were pixel long above pixels above the bottom. This was done because upper leaves would have emerged later on, and tended to be large (leaves produced early in development were more likely to be small). This threshold also helped remove spurs caused by leaf curvature. Figure 4 shows such an example.

(a)
(b)
Figure 4: (a) A plant skeleton image at day 26 having a 1 pixel long spur is circled in red, and (b) The spur is shown zoomed in circled in red (upper right), and after removing the spur (lower right).

3.5.2 Root area pruning with maize statistics

There was a large number of spurs created adjacent to the tub edge and soil, resulting in uneven segmentation near the root of the plant. Thus, the next few pruning steps were performed to resolve this issue. This was based on existing information regarding the collar (a spot on the stem from where leaves emerge); the third leaf collar of maize plants usually becomes visible approximately between 10 to 14 days after emergence [4]. Hence, we calculated the number of branching points, starting from the topmost (maize has an apical structure [18], which means the new leaves should only emerge towards the top) for each plant image up to day 10, and everything below the fourth branching point was removed. However, even though the rule suggests there would be at most three branching points, four was chosen to be safe. Figure (b)b shows how this pruning step removed a number of spurs around the root of Figure (c)c.

(a)
(b)
(c)
(d)
Figure 5: (a) Segmentation of plant at day 5. (b) Skeleton of the same plant, and the spurs are shown within red rectangle (c) After root area cleaning with maize statistics (3.5.2) (d) After root area cleaning by comparing last two consecutive branching point’s position (3.5.3).

3.5.3 Root area pruning by comparing last two consecutive branching point’s position

However, there were still some spurs near the root (Figure (c)c). After the above pruning steps, up to four branching points were still possible up until day 10. To remove putative spurs that remain, if their distance between the lowest two branching points is small (smaller than 10 pixels), then it is unlikely for them to be real leaves following alternate phyllotaxy. Hence, the lowest branching point was removed (Figure (d)d). This was also applied on plants up to day 10.

3.5.4 Removing tub edge

Some images had spurs near the root even after applying the above pruning steps, for example, Figure (c)c (and on images after day 10). Hence, for all of the images, the -distance, and -distance111In images, the upper leftmost pixel is , the -axis is the height, and -axis is the length. between the lowest branching point, and the lowest end-point of the skeleton were calculated, which could either be an end-point of a leaf, or an end-point of an unwanted edge or spur, or possibly the root. If the -distance is less than or equal to the -distance, then possibly it is the root, or a leaf end-point; otherwise it might be a spur, and was deleted. This is because the lowest end-point should be on the stem, and a larger -distance would be representative of moving horizontally from the stem without branching, a likely sign that it was created by the soil, or the tub edge. Figure (d)d shows how this pruning strategy removed the spur nearest the root.

(a)
(b)
(c)
(d)
Figure 6: (a) Skeleton of a plant at day 5. (b) Skeleton of the same plant, with spurs shown within the red rectangle (c) After root area cleaning with maize statistics and comparing last two consecutive branching point’s position (3.5.3). (d) After applying conditions to remove the tub edge.

3.5.5 Remove root branches created due to non-smooth segmentation boundary

Lastly, while analyzing false leaves, it was noticed that uneven boundaries near the root of the segmented plant image causes false positives. Figure (a)a shows an example of such a scenario. The main challenge here was to decide between a bent dying leaf, and a spur. To identify and remove spurs such as this, the boundary of the segmented plant is compared to the lowest end-point of the first detected leaf to assess whether it is in the middle of the boundary or not (Figure (b)b). However, there might be cases where the lowest edge is a bent leaf. To ensure that any true leaf is not deleted, the angles between the potential leaf and stem was calculated, and an angle threshold was used to make the decision. Figure (c)c, and (d)d show how the angle between the potential leaf and stem play a role in deciding between a leaf and a spur. This pruning step was applied on plants at day 15 and later, where this scenario was more prominent.

(a)
(b)
(c)
(d)
Figure 7: (a) Segmentation of a plant at day 25. (b) Skeleton of the same plant overlaid inside plant boundary, and the spurs near the root area are shown within red rectangle. (c) Plant skeleton without the plant boundary, and the spurs are within the red rectangle. (d) After removing root branches.

3.5.6 Removal of root spurs by comparing the lowest branching point with the lowest skeleton point

There were some skeletons that had a true leaf and a spur connected with the same branching point; this was common at the lowest branching point. In such a case, there were three branching points (Figure (a)a). We classify one as a continuation of the stem, one a leaf, and one a spur. A heuristic was used that looked at differences in both and coordinates of each end-point with the branching point. The stem has the smallest difference in -coordinate. Between the two remaining segments, a decision is made based on the length of the segment with small and low segments being preferred as the spur. In Figure (a)a, after stem identification, the spur is associated with the lower of the two segments. In Figure (c)c, the lowest end-point is that of the leaf, but the spur is chosen to be a short segment.

(a)
(b)
(c)
(d)
Figure 8: (a) A plant skeleton at day 17, and the spurs are shown within red rectangle (the lowest branching point and the lowest pixel point of the skeleton are shown with arrow). (b) After pruning (c) A plant skeleton at day 26, and the spurs are shown within red rectangle. d) After pruning.

3.6 Growth properties, and leaf matching with the Hungarian matching algorithm

Despite the fact that the pruning steps of Section 3.5 are largely helpful, there are some real leaves that are being improperly disregarded by them. Also, there were still some false positive skeleton spurs present in some plant skeleton images. This section describes techniques to remove some more skeleton spurs, and also detect some true leaves that were not detectable with the image processing techniques.

It is helpful to understand some statistics regarding maize development. It has been found that until the tenth-leaf stage (meaning ten leaves with a collar are visible) the rate of leaf development is approximately 2 to 3 days per additional leaf [1]. Thus, for each plant, we compared the number of detected leaves on each image between days. It would be better to not compare the number of leaves, but to match leaves between days. But, the view selected is not the same for a plant across all days, hence matching in these cases is not straightforward. Hence, as a first pass, the number of leaves between days was compared, and then if the numbers differed as described below, and the views were the same, then a matching algorithm was used.

Specifically, whenever there was a mismatch between the number of detected leaves and the range in the number of leaves expected on that day, the number of detected leaves of that day was compared with the number of detected leaves of the three previous, and three next days’ images. If an image of any specific day has missing leaves, or has spurs, it can possibly be identified through this comparison. For example, Figure (a)a shows a skeleton at day 10 with a missing leaf, and Figure (b)b shows the skeleton of the same plant at day 11 showing the missing leaf from the day 10 image. The original day 10 image has that missing leaf in it (Figure (d)d), but it went undetected via the skeletonization procedures. If we compare the leaf count between days 7 and 13, it is clear that day 10 has a missing leaf. Similarly, Figure 10 shows another example of the plant skeleton at day 17 that has a spur in it. However, the number of detected leaves between days 14 and 20 is one fewer.

(a)
(b)
(c)
(d)
Figure 9: (a) A plant skeleton at day 10 missing a leaf. (b) Skeleton of the same plant at day 11 having the leaf that is missed in day 10 skeleton. (c) The Hungarian matching algorithm successfully detects the missing leaf. (d) The plant image at day 10, and the leaf that was missing shown in red.
(a)
(b)
(c)
(d)
Figure 10: (a) A plant skeleton at day 17 with a skeleton spur as a false leaf. Skeletons of the same plant at previous day (b), and (c) next day. (d) The Hungarian matching algorithm successfully detects the false leaf.

When a difference occurred, we applied the Hungarian matching algorithm to detected leaves from one day to the next day. This algorithm operates on undirected, weighted bipartite graphs. If the two bipartite vertex sets are and where is smaller in size than , then a matching is any injective function from to . The image of an element in is its match. Given any such matching , the score of is the sum of the edge weights on edges connecting each vertex with . Of all of the (exponentially many) matchings, the Hungarian algorithm can find the matching which produces the smallest possible score, in polynomial time.

In the context of this problem, each leaf detected in the day image was represented as a vertex in , and each leaf in the day image was represented as a vertex in . The edge weights between them was the sum of the Euclidian distance of the two leaf end-points, with the Euclidian distance of the two leaf branching points. If two leaves between two days has a small weight, then they are likely the same leaf. In this way, we obtained the best matching (of leaves between days) with the Hungarian algorithm. Finally, a threshold was applied to the resulting matched leafs to only keep a match if the edge weight was small enough. After this, any leaf that is unmatched in some day vs. adjacent days could be either a spur, or had the leaf occluded, and which of those is resolved by considering the number of leaves in neighbouring days as described previously.

4 Evaluation methodologies

The verification of leaf detection was done visually, which means manually checking correspondence between skeleton segments and leaves of the predicted elements to ground-truth images. A skeleton segment is a leaf if it starts in a branching point, and has an end-point. The ground-truth number of leaves used to evaluate our method was calculated by taking the maximum number of leaves between the two views. As previously mentioned, leaves were only annotated on an image if they were visible, but a better method of evaluation would be to use the number of leaves even if they are not visible. However, this information is not available but there are at least as many leaves as the maximum of those annotated across the two views. We calculated precision and recall of detected leaves, to evaluate our proposed technique.

A comparative evaluation was done with the Deep Plant Phenomics (DPP) platform which is an open-source

[33]

programming interface for training models to perform regression and classification tasks. A convolutional neural network (CNN) was created with the DPP framework and was trained according to

[34]. Among general trials, the best results were obtained by a model that had two convolution layers, four

convolution layers with stride 2, and an output layer. A

, stride 2 max pooling layer was used after each convolution layer. The model parameters, and training hyper-parameters were: batch size 10, image dimensions

, learning rate 0.0001, number of epochs 500, 65% of data for training, 15% of data for validation, and 20% testing data. The batch size denotes the number of examples to be considered for each iteration of training. The total number of images was 630, as images acquired prior to plant emergence were excluded. We employed data augmentations consisting of cropping, flipping, and brightness/contrast adjustment. This was only used to estimate leaf count and not to detect leaf positions. However, it is possible to compare results to ours by interpreting our results as a leaf count. The model was evaluated by calculating the mean absolute loss and absolute loss standard deviation, where the absolute loss is the relative difference in count between predicted and ground-truth.

5 Results

After skeletonization
Traditional threshold
based pruning
Removing spurs with
maize literature
After comparing time series
leaf count and leaf matching
with Hungarian algorithm
Plant Ground-truth True leaf False leaf True leaf False leaf True leaf False leaf True leaf False leaf
001-9 127 125 6 125 3 125 2 126 1
006-25 151 145 14 144 2 142 0 144 1
008-19 153 137 7 134 4 134 1 136 1
016-20 110 77 0 77 0 77 0 81 0
023-1 131 122 24 119 19 121 5 122 4
045-1 131 127 37 127 31 127 7 128 4
047-25 156 151 17 150 11 148 0 151 0
063-32 156 148 22 144 8 142 1 145 0
070-11 135 128 17 127 14 122 3 124 2
071-8 156 138 7 136 2 136 1 137 1
076-24 142 130 20 127 15 126 2 127 1
104-24 155 134 7 134 6 130 1 131 1
191-28 140 122 2 121 2 121 0 122 0
Total 1843 1684 180 1665 117 1651 23 1674 16
Table 1: Column contains the Plant identifier, column contains ground-truth number of leaves (maximum across views). The remaining columns contains the number of detected true leaves, and false leaves after each technique.
After skeletonization with
traditional threshold
based pruning
After detecting and
removing spurs with
maize literature
After comparing time series leaf
count and leaf matching with
Hungarian algorithm
TP (real leaves predicted as leaves) 1665 1651 1674
FP (not real leaves predicted as leaves) 117 23 16
FN (real leaves not predicted as leaves) 178 192 169
Positive, TP+FN 1843 1843 1843
Recall, TP/(TP+FN) 0.903 0.895 0.908
Precision, TP/(TP+FP) 0.934 0.986 0.990
Table 2: Recall, and Precision calculation with, and without maize literature and Hungarian matching algorithm.

Table 1 describes the total number of true leaves and false leaves detected in different phases of the proposed technique when they were executed in order. Note that there was a significant reduction in false leaves (23 from 117) after employing the maize plant growth knowledge and statistics. When the time series leaf count comparison, and the Hungarian algorithm strategies were then applied, the number of true leaves increased, and 9 fewer false leaves were detected. However, these processes added an additional two false leaves making the final detection of 1674 true leaves, and 16 false leaves. Across all procedures, the precision, and recall of our proposed method was , and respectively (Table 2). While we used the maximum of the leaves across the two views as the ground-truth (total of true leaves), if we instead used the leaves visible in the views selected (total of true leaves), the recall and precision would be , and respectively.

We also calculated the mean absolute loss, and the absolute loss standard deviation to compare our method with the existing deep learning techniques. The best DPP leaf counter model after a few trials resulted in a mean absolute loss of

, and an absolute loss standard deviation was . In comparison, our method had a mean absolute loss, and an absolute loss standard deviation of , and respectively without the time series leaf count comparison and Hungarian matching algorithm respectively, and , and with the time series leaf count comparison and Hungarian matching algorithm respectively. Therefore, our method was achieving better results.

6 Discussion and Conclusions

The maize dataset was released in [10], where they also performed leaf counting as a part of their component-based phenotyping studies. The view selection, and the segmentation methods of [10] was similar to our method. They have evaluated their method by calculating the average plant-level accuracy, and defined the plant-level accuracy by subtracting the number of false leaves from the number of detected true leaves, and then dividing by the number of leaves present in the plant image selected. Their average plant-level accuracy of leaf detection with this dataset was [10]

, where the average plant-level accuracy is the average of the thirteen plant-level accuracy of the 13 maize plants. However, their evaluation metric was calculated with the number of leaves present in the plant image selected, but the results of their view selection was not available. Thus, it is not possible to directly compare our results with theirs (as our views were not identical to theirs).

The calculated mean absolute loss, and the absolute loss standard deviation of our technique and Deep Plant Phenomics (DPP) leaf counter model indicates that, the novel technique combining image processing and knowledge regarding maize development improved leaf counting. Moreover, this proposed approach allows us to predict the positions of the leaves, whereas the deep learning leaf counting models only output the predicted total number of leaves.

The accuracy of component-based plant phenotyping highly depends on the obtained plant skeletons. The task of determining ideal image processing techniques to figure out maize topology is challenging. Therefore this proposed novel method aims to reduce the constraints on determining the ideal image processing algorithms, and attempts to improve the results obtained from computer vision with plant-specific growth statistics, and knowledge. Moreover, the Hungarian algorithm adds additional information by matching topologies from one day to others, which refines leaf identification. This work contributes towards component-based plant phenotyping studies of maize. Altogether our method achieves a recall of , and a precision of , also indicating that our technique can play an important role in component-based plant phenotyping studies of not only maize plants but also similarly structured plants.

7 Acknowledgements

This research was undertaken thanks in part to funding from the Canada First Research Excellence Fund.

References

  • [1] L. Abendroth, R. Elmore, M. Boyer, and S. Marlay (2011-03) Corn growth and development. Iowa State University Extension Service PMR 1009, pp. 75. Cited by: §3.6.
  • [2] J. L. Araus, S. C. Kefauver, M. Zaman-Allah, M. S. Olsen, and J. E. Cairns (2018) Translating high-throughput phenotyping into genetic gain. Trends in Plant Science 23 (5), pp. 451–466. Cited by: §1.
  • [3] X. Bai and L. J. Latecki (2007) Discrete skeleton evolution. In

    International Workshop on Energy Minimization Methods in Computer Vision and Pattern Recognition

    ,
    pp. 362–374. Cited by: §3.4.
  • [4] J. Bell (2018-06) Corn growth stages and post emergent herbicide timing. Note: Texas Row Crops Newsletter Cited by: §3.5.2.
  • [5] O. T. Bonnett (1954) The inflorescences of maize. Science 120 (3107), pp. 77–87. Cited by: §2.
  • [6] J. Cai (2012) Robust filtering-based thinning algorithm for pattern recognition. The Computer Journal 55 (7), pp. 887–896. Cited by: §1, §3.4.
  • [7] A. Chaudhury, M. Brophy, and J. L. Barron (2016) Junction-based correspondence estimation of plant point cloud data using subgraph matching. IEEE Geoscience and Remote Sensing Letters 13 (8), pp. 1119–1123. Cited by: §1.
  • [8] A. Chawade, J. van Ham, H. Blomquist, O. Bagge, E. Alexandersson, and R. Ortiz (2019) High-throughput field-phenotyping tools for plant breeding and precision agriculture. Agronomy 9 (5), pp. 258. Cited by: §1.
  • [9] Y. Chen and W. Hsu (1988) A modified fast parallel algorithm for thinning digital patterns. Pattern Recognition Letters 7 (2), pp. 99–106. Cited by: §3.3.
  • [10] S. D. Choudhury, S. Bashyam, Y. Qiu, A. Samal, and T. Awada (2018) Holistic and component plant phenotyping using temporal image sequence. Plant Methods 14 (1), pp. 35. Cited by: §1, §2, §2, Figure 2, §3.1, §3.2, §3.3, §6.
  • [11] S. D. Choudhury, V. Stoerger, A. Samal, J. C. Schnable, Z. Liang, and J. Yu (2016-08) Automated vegetative stage phenotyping analysis of maize plants using visible light images. In

    KDD Workshop on Data Science for Food, Energy and Water

    ,
    San Francisco, California, USA. Cited by: §1.
  • [12] P. A. Crisp, D. Ganguly, S. R. Eichten, J. O. Borevitz, and B. J. Pogson (2016) Reconsidering plant memory: intersections between stress recovery, RNA turnover, and epigenetics. Science Advances 2 (2), pp. e1501340. Cited by: §1.
  • [13] C. Du and D. Sun (2004) Recent developments in the applications of image processing techniques for food quality evaluation. Trends in Food Science & Technology 15 (5), pp. 230–249. Cited by: §1.
  • [14] D. D. Fromme, T. A. Spivey, and W. J. Grichar (2019) Agronomic response of corn (Zea mays l.) hybrids to plant populations. International Journal of Agronomy. Cited by: §1.
  • [15] W. Gittoes, T. Botterill, and R. Green (2011) Quantitative analysis of skeletonisation algorithms for agricultural modelling of branches. In Proceedings of the 26th Conference on Image and Vision Computing New Zealand, IVCNZ, Vol. 11, pp. 1–6. Cited by: §1.
  • [16] S. Gunasekaran (1996) Computer vision technology for food quality assurance. Trends in Food Science & Technology 7 (8), pp. 245–256. Cited by: §1.
  • [17] P. J. Hodgen (2007) Individual corn plant nitrogen management. The University of Nebraska-Lincoln. Cited by: §1.
  • [18] E. E. Irish and T. M. Nelson (1991) Identification of multiple stages in the conversion of vegetative to floral development. Development 112 (3), pp. 891–898. Cited by: §3.5.2.
  • [19] R. Jonker and A. Volgenant (1987) A shortest augmenting path algorithm for dense and sparse linear assignment problems. Computing 38 (4), pp. 325–340. Cited by: §1.
  • [20] H. W. Kuhn (1955) The Hungarian method for the assignment problem. Naval Research Logistics Quarterly 2 (1-2), pp. 83–97. Cited by: §1.
  • [21] L. Lam, S. Lee, and C. Y. Suen (1992) Thinning methodologies-a comprehensive survey. IEEE Transactions on pattern analysis and machine intelligence 14 (9), pp. 869–885. Cited by: §1.
  • [22] L. Lam and C. Y. Suen (1995) An evaluation of parallel thinning algorithms for character recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 17 (9), pp. 914–919. Cited by: §3.3.
  • [23] T. Lee, R. L. Kashyap, and C. Chu (1994) Building skeleton models via 3-d medial surface axis thinning algorithms. CVGIP: Graphical Models and Image Processing 56 (6), pp. 462–478. Cited by: §3.3.
  • [24] U. of Nebraska–Lincoln Plant Vision Initiative (2018-11)(Website) External Links: Link Cited by: §2.
  • [25] N. Otsu (1979) A threshold selection method from gray-level histograms. IEEE Transactions on Systems, Man, and Cybernetics 9 (1), pp. 62–66. Cited by: §3.1.
  • [26] F. Perez-Sanz, P. J. Navarro, and M. Egea-Cortines (2017) Plant phenomics: an overview of image acquisition technologies and image data analysis algorithms. GigaScience 6 (11), pp. gix092. Cited by: §1.
  • [27] T. Rouf Shah, K. Prasad, and P. Kumar (2016) Maize—a potential source of human nutrition and health: a review. Cogent Food & Agriculture 2 (1), pp. 1166995. Cited by: §1.
  • [28] P. K. Saha, G. Borgefors, and G. S. di Baja (2017) Skeletonization and its applications — a review. In Skeletonization, pp. 3–42. Cited by: §3.3.
  • [29] D. Shaked and A. M. Bruckstein (1998) Pruning medial axes. Computer Vision and Image Understanding 69 (2), pp. 156–169. Cited by: §3.4.
  • [30] N. Shakoor, S. Lee, and T. Mockler (2017-08) High throughput phenotyping to accelerate crop breeding and monitoring of diseases in the field. Current Opinion in Plant Biology 38, pp. 184–192. External Links: Document Cited by: §1.
  • [31] F. Tardieu, L. Cabrera-Bosquet, T. Pridmore, and M. Bennett (2017) Plant phenomics, from sensors to knowledge. Current Biology 27 (15), pp. R770–R783. Cited by: §1.
  • [32] R. Tillett (1991) Image analysis for agricultural processes: a review of potential opportunities. Journal of Agricultural Engineering Research 50, pp. 247–258. Cited by: §1.
  • [33] J. R. Ubbens and I. Stavness (2017) Deep plant phenomics: a deep learning platform for complex plant phenotyping tasks. Frontiers in Plant Science 8, pp. 1190. Cited by: §4.
  • [34] J. Ubbens (2017-07) Deep plant phenomics documentation. Note: github External Links: Link Cited by: §4.
  • [35] M. Wahabzada, A. Mahlein, C. Bauckhage, U. Steiner, E. Oerke, and K. Kersting (2016) Plant phenotyping using probabilistic topic models: uncovering the hyperspectral language of plants. Scientific Reports 6 (1), pp. 1–11. Cited by: §1.
  • [36] T. Zhang and C. Y. Suen (1984) A fast parallel algorithm for thinning digital patterns. Communications of the ACM 27 (3), pp. 236–239. Cited by: §3.3.