Morphological reconstruction (MR)  is a powerful operation in mathematical morphology. It has been widely used in image filtering , image segmentation , and feature extraction , etc. Among these applications, one of the most important applications is that MR is often used in seeded segmentation algorithms ,  such as watershed transformation (WT)  and power watershed (PW)  to reduce over-segmentation caused by image noise and details. However, there are two drawbacks ,  when MR is used in seeded segmentation algorithms.
It is difficult to reduce over-segmentation while obtaining a high segmentation accuracy for seeded segmentation algorithms (we use MR-WT to denote MR-based watershed transform and use MR-PW to denote MR-based power watershed). Although MR is able to filter noise in gradient images, some important contour details are smoothed out as well.
MR is sensitive to the scale of structuring elements. In practical applications, if the scale is too small, the reconstructed gradient image suffers from a serious over-segmentation. Oppositely, if the scale is too large, the reconstructed gradient image suffers from an under-segmentation.
Generally, MR is used in watershed transform to improve the segmentation effect by employing a structuring element to filter regional minima . However, it is very difficult to filter useless regional minima while preserving meaningful ones by simply considering one single-scale structuring element. Although
-min imposition  is a simple and efficient method for over-segmentation reduction, it relies on a threshold choice and is likely to miss some important boundaries. Region merging ,  is also a popular method for this, but it requires iterating and renewing edge weight leading to a high computing burden. In addition, some researchers employ reasonable contour detection methods, e.g., globalized probability of boundary (gPb)  that combines the multiscale information from brightness, color and texture, to achieve better image segmentation. However, the gPb is computationally expensive because it combines too many feature cues for contour detection. To speed up the algorithm of contour detection, Dollar and Zitnick  took the advantage of the structure present in regional image patches and random decision forests, and proposed a fast structured edge (SE) detection approach using structured forests. This algorithm obtains real-time performance and state-of-the-art edge detection but requires a huge amount of memory for training data. To reduce memory requirement, Hallman and Fowlkes  proposed a simple and efficient model to learn contour detection, namely oriented edge forests (OEF). Although these improved contour detectors are superior to traditional detectors, e.g. Sobel or Canny, and they are helpful for improving subsequent image segmentation, they still generate a large number of seeds leading to serious over-segmentations.
In practice, contour detection methods are usually combined with other approaches to improve image segmentation effect. For example, Fu et al.
 proposed a robust image segmentation approach using contour-guided color palettes by integrating contour and color cues, where SE, mean-shift algorithm , region merging, and spectral clustering  are combined to achieve better segmentation results. However, the approach is complex because it combines several different algorithms that requires many parameters.
In this paper, we propose an adaptive morphological reconstruction (AMR) operation that is able to generate a better seed image than MR to improve seeded segmentation algorithms. Firstly, AMR employs multiscale structuring elements to reconstruct a gradient image. Secondly, a pointwise maximum operation on these reconstructed gradient images is performed to obtain the final adaptive reconstruction result. Because AMR employs small structuring elements to reconstruct pixels of large gradient magnitudes while employing large structuring elements to reconstruct pixels of small gradient magnitudes in a gradient image, AMR is able to obtain better seed images to improve the seeded segmentation algorithms. Our main contributions are summarized as follows.
Multiscale structuring elements are employed by AMR, and different scaled structuring elements are adaptively adopted by pixels of different gradient magnitudes without computing the local features of a gradient image.
AMR has a convergence property and a monotonic increasing property, the two properties help seeded segmentation algorithms to achieve a hierarchical segmentation.
AMR has a low computational complexity, and it can help seed-based spectral segmentation to achieve better image segmentation results than the-state-of-art algorithms.
The rest of the paper is organized as follows. In the next section, the research background related with AMR is introduced and analyzed. On this basis, AMR is proposed, and its two properties, monotonic increasingness and convergence are carefully analyzed in Section III. To demonstrate the superiority of AMR, AMR is used for seeded image segmentation and seed-based spectral segmentation. Experiments are presented in Section IV, followed by the conclusion in Section V.
Ii-a Morphological Reconstruction
MR is an image transformation that requires two input images, a marker image and a mask image. Let two grayscale images and denote the marker image that is the starting point for the transformation and the mask image that constrains the transformation, respectively . If , which means is pointwise less than or equal to , the morphological dilation reconstruction of from is denoted by
where , for , satisfies . The symbol represents the elementary morphological dilation operation, and stands for the pointwise minimum at each pixel of two images as shown in Fig. 1(a).
Similarly, if , the morphological erosion reconstruction of from , which is the dual operation of , is defined as
where , for , satisfies . The symbol represents the elementary morphological erosion operation, and stands for the pointwise maximum at per pixel of two images as shown in Fig. 1(b).
To further illustrate the principle of MR for image transformation, we present an example for the binary MR as shown in Fig. 2, where the red regions denote seeds, i.e., the marker image .
According to Fig. 2 and (1)-(2), a suitable marker image is important for MR. We have known that for while for . Thus, there are lots of choices for . Different marker images corresponds to different reconstruction results as shown in Fig. 3. To obtain an efficient in practice, the marker image is usually obtained by performing a transformation on the corresponding mask image -. For example, the erosion or dilation result of a mask image is often considered as a marker image , i.e., or , where is a disk shaped structuring element, the radius of is , . Therefore, MR is sensitive to the parameter because the marker image is decided by the scale of the structuring element.
As compositional morphological opening and closing operations show better performance than elementary morphological erosion and dilation operations for image filtering, feature extraction, etc., we present the definition of compositional morphological opening and closing reconstructions ( and ) of from as follows
Ii-B Multiscale and Adaptive Mathematical Morphology
For image filtering and enhancement using morphological operators, a large-scale structuring element can suppress noise but may also blur the image details, whereas a small-scale structuring element can preserve image details but may fail to suppress noise. Some researchers proposed multiscale and adaptive morphological operators to improve the performance of traditional morphological operators. However, most multiscale morphological operators ,  such as morphological gradient operators and morphological filtering operators, average all scales of morphological operation results as final output
where is the final output result, is the radius of the structuring element, , , . Although the average result is superior to the result based on single-scale morphological operators, it causes contour offset and mistakes. Some researchers improved multiscale morphological operators by introducing a weighted coefficient to (4), and they defined adaptive multiscale morphological operators as follows 
where is the weighted coefficient on the th scale result. However, because the computing of weighted coefficients is complex, the adaptive multiscale morphological operators have a low computational efficiency. Moreover, the weighted average result is similar to average result because it is difficult to obtain the optimal weighted coefficient, even though the former is slightly better than the latter.
Although lots of adaptive multiscale morphological operators - have been proposed, it can be seen from (4)-(5) that both the multiscale and adaptive morphological operators employ a linear combination of different-scale results to improve single-scale morphological gradient or filtering operators. Because the linear combination is unsuitable for multiscale morphological reconstruction operation, in this paper, we try to employ a non-linear combination (i.e., the pointwise maximum operation denoted by ) to design adaptive morphological reconstruction operators. These operators are different from conventional multiscale and adaptive morphological operators employing linear combination in (4)-(5). We use non-linear operation instead of linear combination since the former is more suitable than the later for the removal of useless seeds in seeded image segmentation.
Ii-C Seeded Segmentation
Seeded segmentation algorithms, such as graph cuts , random walker , watersheds , and power watershed  have been widely used in complex image segmentation tasks due to their good performance . It is not required to give seed images for both graph cuts and random walker because they usually consider each pixel as a seed. However, a seed image is necessary for WT and PW by computing the regional minima of a gradient image.
Since both WT and PW obtain seeds from a gradient image that often includes a huge number of seeds generated by noise and unimportant texture details, they usually suffer from over-segmentation. A larger number of approaches for addressing over-segmentation was proposed, and these approaches can be categorized into two groups.
Feature extraction or feature learning is used to obtain a better gradient image that enhances important contours while smoothing noise and texture details -.
MR is used for gradient reconstruction to reduce the number of regional minima -.
For the first group of approaches, gPb, OEF, and SE are popular for reducing over-segmentation as shown in Figs. 4-5. In Fig. 5, although gPb, OEF, and SE provide better gradient images that can reduce over-segmentation for WT and PW, the segmentation results are still poor compared to ground truths shown in Fig. 4.
The second group of approaches depends on MR and WT, it is denoted by MR-WT. Najman and Schmitt  employed MR to remove regional minima to reduce over-segmentation. Furthermore, a dynamic threshold is used to change the gradient magnitude that is smaller than the threshold, and then a hierarchical segmentation result is obtained. Wang  proposed a multiscale morphological gradient algorithm (MMG) for image segmentation using watersheds. The proposed MMG employs multiple structuring elements to obtain a better gradient image, and uses MR to remove regional minima to improve watershed segmentation.
Fig. 6 illustrates the principle of a seeded segmentation framework based on MR-WT. We can see that the number of the regional minima in the gradient image decreases rapidly with the increase of , but the boundary is also destroyed simultaneously. It is clear that the larger structuring element corresponds to fewer seeds. One major reason is that MR employs a single-scale structuring element, which equally treats all pixels of different gradient magnitudes in the gradient image. For example, in dilation reconstruction, the marker image converges to the minimum grayscale value of pixels in the mask image as the value of increases. Obviously, both large and small structuring elements lead to poor reconstruction results while a moderate-sized structuring element achieves a rough balance via sacrificing contour precision. Therefore, it is difficult to obtain a good seed image by employing a single-scale structuring element. Although many researchers employ multiscale structuring elements to generate a better gradient image, there are few studies on multiscale MR for gradient images. Moreover, the fusion of different-scale results is also a problem.
Ii-D Spectral Segmentation
It is well-known that spectral clustering  is greatly successful due to the fact that it does not make strong assumptions on data distribution, and it is implemented efficiently even for large datasets, as long as we make sure that the affinity matrix is sparse. However, since the size of the affinity matrix isfor an image of size
, and it is not sparse because of Gaussian similarity measure, spectral clustering is often inefficient for image segmentation due to eigenvalue decomposition of the huge affinity matrix. To address the issue, a great number of algorithms have been proposed to construct a smaller affinity matrix and thus to improve the computational efficiency of spectral clustering -. Most of these algorithms employ pre-segmentation (superpixel) methods such as the simple linear iterative clustering (SLIC) , mean-shift , linear spectral clustering (LSC) , and superpixel hierarchy , to reduce the number of pixels of the original image and, in turn, reduces the size of the affinity matrix. As an example, Zhanget al.  proposed a fast image segmentation approach that is a re-examination of spectral clustering on image segmentation. The approach provides better image segmentation results yet requires a long running time.
The popular superpixel approaches have some drawbacks for spectral segmentation. Firstly, mean-shift algorithm involves three parameters and it is sensitive to these parameters. Secondly, SLIC only generates superpixels that include regular regions, and these regions have a similar shape and size. Finally, LSC is superior to SLIC because LSC successfully connects a local feature with a global optimization objective function, so that LSC can generate more reasonable segmentation results. However, similar to SLIC, LSC also provides superpixels that include regular regions with a similar shape and size.
As seed-based spectral segmentation algorithms are sensitive to pre-segmentation results, an excellent pre-segmentation algorithm can improve segmentation results generated by seed-based spectral segmentation algorithms.
Iii Adaptive morphological reconstruction
Iii-a The Proposed AMR
To overcome the drawback of MR on regional minima filtering, we propose an AMR that is able to filter useless regional minima and maintains meaningful ones generated by salient objects. Fig. 7 shows the motivation of AMR in which multiscale structuring elements are employed to reconstruct a gradient image, i.e., small structuring elements are adopted by pixels of large gradient magnitude while large structuring elements are adopted by pixels of small gradient magnitude.
Definition 1. Let be a series of nested structuring elements, where is the scale parameter of a structuring element, , . For a gradient image such that and , the adaptive morphological reconstruction denoted by of from is defined as
Note that the pointwise maximum operation is only suitable for , but not suitable for . Because (the proof is presented in Appendix A) and , is unable to obtain a significantly convergent gradient image if .
We apply AMR to the gradient image shown in Fig. 6. The reconstruction and segmentation results are shown in Fig. 8, where the adopted structuring elements are disk and . More detailed comparisons are shown in Fig. 9. By comparing Fig. 8 with Fig. 6, it is obvious that AMR obtains better seed images than MR due to the fact that the non-linear operation is able to remove efficiently useless seeds.
To further show the influence of on AMR, Fig. 10 shows segmentation results provided by AMR through changing the value of . We can see that there are some small segmented areas when the value of is small. These small areas are merged by increasing the value of . However, although a large leads to the merge of small areas, the precision of object contours will be decreased as shown in Fig. 6. Therefore, we usually set for a moderate-sized image.
Iii-B The Monotonic Increasingness Property of AMR
AMR is an algorithm that aims at finding meaningful regional minima by merging or filtering useless regional minima. AMR includes two parameters and . When we increase the value of , gradient images reconstructed by AMR keep the increasing order as shown in Theorem 1.
Theorem 1. Let be an adaptive morphological reconstruction operator, is increasing with respect to the scale of structuring elements, i.e., for a gradient image such that and , , , , we have
The proof of Theorem 1 is presented in Appendix B. Theorem 1 shows that the gradient image processed by AMR is monotonous increasing with the increase of . Fig. 9 demonstrates Theorem 1. We can see that if is enlarged, the more unimportant seeds are removed, and important seeds are preserved. Actually, the result is equivalent to region merging. However, the method is simpler than region merging. According to the result, it can be seen that AMR can help seeded segmentation algorithms to achieve a hierarchical segmentation , . Hierarchical segmentation is a multilevel segmentation scheme, and it usually outputs a coarse-to-fine hierarchy of segments ordered by the level of details. Multiscale combinatorial grouping (MCG) proposed by Pont-Tuset et al.  is an excellent hierarchical segmentation approach that employs a fast normalized cut algorithm and an efficient algorithm for combinatorial merging of hierarchical regions. Based on the hierarchical segmentation results provided by MCG, some improved approaches are also proposed , . These improved approaches achieve better segmentation effect but have lower computational efficiency than MCG.
Before analyzing the relationship between AMR-WT and hierarchical segmentation, we first review some basic concepts of hierarchical segmentation. Let be a finite set. A hierarchy on is a set of parts of such that
For every , .
For each pair , or .
Note that is a chain of nested partitions. Let be the initial partition of , which corresponds to the finest partition of , and be the coarsest partition of , which segments the images as one single region. A partition , , on has the property that
where denotes the partition. is finer than the partition . Derived from Theorem 1 and Fig. 9, we obtain
where denotes seeded segmentation algorithms such as WT or PW. Suppose that , , and then
According to (12), the principle of the hierarchical segmentation based on AMR is shown in Fig. 11, in which the data points represent regions obtained by the hierarchical segmentation at different levels.
Iii-C The Convergence Property of AMR
By comparing Fig. 6 with Fig. 8, it can be observed that AMR provides significant gradient images and AMR-WT generates convergent segmentation results via enlarging the scale of structuring elements. An important convergence property of AMR is described in the following.
Theorem 2. Let be an adaptive morphological reconstruction operator, is convergent when increasing the scale parameter , i.e., for any gradient images and such that , if then
i.e., , and the proof is presented in Appendix C.
According to Fig 9, it can be seen that the gradient result and the corresponding seed image will remain unchanged when . This empirically illustrates that the gradient image reconstructed by AMR is convergent when increasing the value of . Besides, the large gradient magnitude is unchanged while the small gradient magnitude converges to ones larger than itself for AMR. However, the large gradient magnitude converges to one smaller than itself while the small gradient magnitude converges to one larger than itself for MR when the structuring element is small. With the increase of the value of , the value of gradient magnitudes finally converges to the minimum of the original gradient image, i.e., (see Appendix A). Consequently, MR removes all regional minima while AMR only filters useless regional minima and preserves significant ones when .
Furthermore, we analyze how to determine the parameter for AMR. The computational efficiency of AMR is influenced by the parameter . A small means a low computational complexity. According to Theorem 2, the reconstructed gradient image and the corresponding segmentation result are unchanged when , but the obtained is usually large. As the paper aims at employing AMR to improve seeded segmentation algorithms, we replace the convergence condition with checking the difference between and . We propose an objective function for justifying the convergence of AMR
where , . It is clear that the segmentation result will remain unchanged when , is a minimal threshold error, and it is a constant used for , but is a variant for . Consequently, only a parameter needs to be tuned for obtaining different reconstruction results.
Iii-D The Algorithm of AMR
AMR only involves the parameter and , as described in the detailed steps of AMR in Algorithm 1111Source code is available at https://github.com/SUST-reynole/AMR. To speed up the convergence of Algorithm 1, the three parameters , , and are used for AMR because the iteration can be stopped according to or . The computational complexity of AMR depends on the values of or . A large value of corresponds to a small value of . The larger is the value of , the longer is the execution time of AMR. Since we have known that AMR has a fast convergent property as shown in Fig. 8, a small is enough for moderate-sized images in practical applications. A small m indicates that AMR has a low computational complexity.
Note that the parameter is unnecessary theoretically, we use two convergent condition and to speed up the convergence of Algorithm 1. We applied Algorithm 1 to images with complex texture content to demonstrate that the proposed AMR is effective for reducing over-segmentation as shown in Fig. 12. AMR-WT not only overcomes the problem of over-segmentation but also obtains better contours than MR-WT and state-of-the-art superpixel methods. Furthermore, we test Algorithm 1 on images with text to show the monotonic increasing and convergent properties of AMR. Fig. 13 shows the comparison results. We can see that the segmentation results are nested, which demonstrates the monotonic increasing property of AMR. Moreover, the segmentation results are unchanged when , which demonstrates the convergent property of AMR.
To demonstrate the effectiveness and efficiency of the proposed AMR, we apply AMR to seeded image segmentation and spectral segmentation. We conduct experiments on the BSDS500 dataset. The experiments are performed on a workstation with an Intel Core (TM) i7-6700, 3.4GHz CPU and 16GB memory.
We compare the proposed algorithms with state-of-the-art algorithms including a multiscale morphological gradient for watersheds (MMG-WT) , multiscale ncut (MNCut) , oriented-watershed transform-ultrametric contour map (gPb-owt-ucm) , the algorithm recovering occlusion boundaries from an image proposed by Hoiem (gPb-Hoiem) , spectral segmentation algorithms proposed by Kim et al. (FNCut, cPb-owt-ucm) , Higher-order correlation clustering (HO-CC) , global/regional affinity graph (GL-graph) , single-scale combinatorial grouping (SCG) , and multiscale combinatorial grouping (MCG) . The open source codes and model parameters suggested by the corresponding authors are used. Because the author did not present specific parameter values for MMGR-WT, we set and , where is a threshold and it is used to generate a marker image, and is the radius of the structuring element used for MR. For the proposed approaches, we set , , and .
We report the experimental results using three evaluation metrics to quantitatively measure the performance of segmentation algorithms: probabilistic rand index (PRI), segmentation covering (CV), and variation of information (VI). The PRI and CV are similarity measures, and they are large while the VI is small when the final segmentation is close to ground truth segmentation.
Iv-a Seeded Image Segmentation
AMR is useful for improving seeded image segmentation because it employs multiscale structuring elements to obtain a convergent seed image without pre-setting many parameters. To show the capability of AMR, it is applied to different gradient images to filter seeds. Fig. 14 shows reconstructed gradient images by AMR and the corresponding segmentation results by WT/PW. These results are clearly better than the ones shown in Fig. 5. The problem of over-segmentation for seeded segmentation algorithms is therefore addressed. Furthermore, compared Fig. 6 to Fig. 14, although both MR and AMR are able to filter seeds, AMR is able to maintain meaningful seeds that correspond to important contours.
Furthermore, Table I shows the number of seeds generated by gradient images. We can see that the reconstructed gradient images generate fewer seeds than original gradient images, which demonstrates AMR is efficient for the filtering of useless seeds. Moreover, AMR is robust for different gradient images obtained by Sobel, gPb, OEF, and SE because the final segmentation results are similar.
In Fig. 14, we set because the segmentation result includes too many small regions when . Clearly, controls the number of small regions in segmentation results. Generally, the value of depends on the resolution of the images to be segmented, e.g., for BSDS500.
To demonstrate that the proposed AMR is robust for different images, we implement AMR-WT/PW on the BSDS500. Fig. 15 shows the comparison of segmentation results using different algorithms, i.e., Sobel-AMR-WT/PW, gPb-AMR-WT/PW, OEF-AMR-WT/PW, and SE-AMR-WT/PW. The segmentation results demonstrate the effectiveness of AMR for the filtering of useless seeds, Moreover, AMR is effective for both WT and PW.
To compare the performance of different algorithms on the BSDS500, Table II shows experimental results of three evaluation metrics: PRI, CV, and VI. We can see that AMR is more efficient for improving segmentation results obtained by WT or PW compared to MR. MR is sensitive to while AMR is insensitive to . Although MMG-MR-WT/PW is effective for the over-segmentation reduction by introducing the parameter , segmentation results are sensitive to both and . The gPb-AMR-WT/PW, OEF-AMR-WT/PW, and SE-AMR-WT/PW obtain better performance than Soble-AMR-WT/PW since the former provides better gradient images. The SE-AMR-WT/PW obtains the best performance. In addition, AMR-WT obtains higher PRI, CV, and lower VI than AMR-PW in the same situation.
Because AMR converges quickly, AMR has a high computation efficiency for gradient reconstruction. Table III shows the comparison of running time of AMR-WT on different gradient images obtained by Sobel, gPb, OEF, and SE, respectively. We only present the running time of AMR-WT here because AMR-PW has a similar running time as AMR-WT. It can be seen from Table III that AMR-WT has a short running time to achieve image segmentation on the BSDS500. The SE-AMR-WT requires the shortest running time because the corresponding gradient image converges quicker under AMR. Tables II-III show AMR is effective and efficient for improving seeded segmentation algorithms such as WT and PW.
Additional evidence of the superiority of AMR can be found in Fig. 16 which shows experimental results on images with rich texture and faded boundaries. According to Figs. 15-16, we can see that the proposed AMR is effective for different kinds of images.
Iv-B Seed-based Spectral Segmentation
In this section, we directly construct the affinity matrix on a pre-segmentation image provided by AMR-WT to reduce the size of the affinity matrix, and then compute the subsequent steps of spectral segmentation (we name it AMR-SC). Note that we employ AMR-WT rather than AMR-PW because the former is able to provide better pre-segmentation results than the latter as shown in Table II. As the pre-segmentation image only consists of dozens of regions, we consider color feature in CIELAB color space and Gaussian function as the criterion to measure the similarity of two regions. Throughout the paper, we use . It is clear that the affinity matrix produced by AMR is a small matrix. Therefore, the clusters can be detected easily and fast with the -means algorithm.
In this paper, the pre-segmentation depends on AMR. According to Table II, we set and , and we set the number of clusters for the -means according to , . The proposed AMR-SC is evaluated on BSDS500 and compared with algorithms such as gPb-owt-ucm, FNCut, GL-graph, SCG and MCG. Figs. 17-18 show that the proposed AMR-SC generates better segmentation results than those comparative algorithms. The result demonstrates that AMR is useful for improving spectral segmentation due to two reasons. The first is that the regional spatial information of an image provided by pre-segmentation is integrated into spectral segmentation, and the second is that the affinity graph is reduced efficiently by removing useless seeds.
Furthermore, we employ the three measures: PRI, CV and VI to compare the proposed AMR-SC with nine state-of-the-art image segmentation algorithms. Table VI shows the region benchmarks on the BSDS500. In Table VI, the proposed AMR-SC clearly dominates other algorithms on PRI and CV, and is on par with SCG on VI mainly due to accurate pre-segmentation provided by AMR-WT. The OEF-AMR-SC and SE-AMR-SC provide better performance than gPb-AMR-SC and Sobel-AMR-SC because OEF and SE obtain better gradient images than gPb and Sobel. In addition, AMR-SC is insensitive to the parameter .
We tested the running time complexity on the BSDS500 dataset. The running time comparison is shown in Table IV. On average, generating a pre-segmentation result with SE-AMR-WT takes 0.54 seconds (SE generates a gradient image requiring 0.06 seconds. AMR-WT takes 0.48 seconds, and ), and constructing an affinity graph and spectral clustering take 0.059 seconds. Consequently, SE-AMR-SC takes about 0.60 second to segment an image from the BSDS500. In contrast, the gPb-owt-ucm takes almost 106.38 seconds, FNCut takes about 10.58 seconds. As GL-graph has four steps, i.e., over-segmentation, feature extraction, bipartite graph construction and graph partition using spectral clustering, it is more complex than SE-AMR-SC, and takes almost 7.41 seconds. MCG takes about 18.60s per image to compute the multiscale hierarchy but SCG takes only 2.21s per image. It is clear that our SE-AMR-SC is the fastest because AMR-SC only depends on the gradient information, and the generated affinity matrix is small.
AMR has two parameters, and . relates to the convergent condition. Generally, a large value of means a few iterations (a small , where is the number of iterations) while a small value of corresponds to many iterations (a large ). Table V shows the influence of on for test images. We can see that increases with the decrease of but is unchanged when .
Furthermore, to show the influence of on AMR, we implement AMR on the BSDS500 by setting different values of , and Tables VI-VII show the results. It is clear that the number of iterations for AMR-WT is smaller and running time is shorter if the value of is larger. However, the number of iterations and running time are unchanged when . Therefore, in practical application, users can select different values of according to their requirements.
Furthermore, we implemented SE-AMR-WT on BSDS500 with different values of . The performance indices of segmentations are shown in Table VIII. By comparing Tables V-VIII, we can see that the average number of iterations, running time, and segmentation accuracy are unchanged for AMR-WT when . Therefore, the proposed AMR is insensitive to .
The value of controls the initial gradient value of images. A large will cause the contour offset while a small value of will cause too many unexpected small regions. Therefore, we choose and for the BSDS500 in Table IV. To further show the influence of on AMR, Table IX shows the performance indices of segmentations on BSDS500 by setting different values of . It can be seen from Table IX that SE-AMR-WT is insensitive to if .
In this work, we have studied the advantages and disadvantages of MR on seeded segmentation algorithms. We proposed an efficient AMR algorithm that can preferably improve seeded segmentation algorithms. The proposed AMR has two significant properties, the monotonic increasingness and the convergence. The monotonic increasingness helps AMR to achieve a hierarchical segmentation. The convergence is able to alleviate the drawback of MR for the filtering of useless regional minima in a gradient image, and guarantees a convergent result. Moreover, we have explored the applications of AMR and have found that AMR is not only able to improve seeded image segmentation results, but also can obtain better spectral segmentation results than state-of-the-art algorithms. Furthermore, the proposed AMR-SC is computationally efficient because a small affinity matrix is used for spectral clustering. Experimental results clearly demonstrate that the proposed AMR-WT generates satisfactory and convergent segmentation results without hard-tuning parameters, and the AMR-SC outperforms most of the state-of-the-art algorithms for image segmentation, and it performs the best in two metrics: PRI and CV.
The segmentation results generated by AMR-WT or AMR-SC can be directly used in object recognition and scene labeling. However, AMR-WT or AMR-SC cannot obtain semantic segmentation results compared to the popular convolutional neural network (CNN), e.g., fully convolutional network (FCN) . To further improve the contour quality of segmentation results, traditional algorithms such as conditional random field , image superpixel , and spatial pyramid pooling , are used to improve the performance of CNN on image segmentation. AMR can be also used in CNN to improve semantic segmentation results. For our future work, we plan to investigate how to combine AMR and FCN effectively and efficiently.
Appendix A Proof of
According to , in (1), we get
In terms of the duality of morphological operation,
Appendix B Proof of theorem 1
Let , from Definition 1, we have
Appendix C Proof of theorem 2
From Definition 1, we have
Since and from Appendix A, we get
We have known that , thus
-  L. Vincent, “Morphological grayscale reconstruction in image analysis: applications and efficient algorithms,” IEEE Trans. Image Process., vol. 2, no. 2, pp. 176-201, Apr. 1993.
-  J. Shi, Y. Wang, D. Xu, C. Yan, T. Chen, Y. He, L. Tang, M. Nie, P. Duan, D. Yan, H. Feng, and J. Yao, “Terahertz imaging based on morphological reconstruction,” IEEE J. Sel. Topics Quantum Electron., vol. 23, no. 4, pp. 1-7, July-Aug. 2017.
-  S. Roychowdhury, D. D. Koozekanani, and K. K. Parhi, “Iterative vessel segmentation of fundus images,” IEEE Trans. Biomed. Eng., vol. 62, no. 7, pp. 1738-1749, Jul. 2015.
-  W. Liao, R. Bellens, A. Pizurica, W. Philips, and Y. Pi, “Classification of hyperspectral data over urban areas using directional morphological profiles and semi-supervised feature extraction,” IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens., vol. 5, no. 4, pp. 1177-1190, Aug. 2012.
Z. Huang, X. Wang, J. Wang, W. Liu, and J. Wang, “Weakly-supervised semantic
segmentation network with deep seeded region growing,” in
Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2018, pp. 7014-7023.
-  W. Casaca, L. G. Nonato, and G. Taubin, “Laplacian coordinates for seeded image segmentation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2014, pp. 384-391.
-  L. Vincent and P. Soille, “Watersheds in digital spaces: an efficient algorithm based on immersion simulations,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 13, no. 6, pp. 583-598, Jun. 1991.
-  C. Couprie, L. Grady, L. Najman, and H. Talbot, “Power watershed: A unifying graph-based optimization framework,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 7, pp. 1384-1399, Jul. 2011.
-  E. O. Rodrigues, L. Torok, P. Liatsis, J. Viterbo, and A. Conci, “K-MS: a novel clustering algorithm based on morphological reconstruction,” Pattern Recognit., vol. 66, pp. 392-403, Jun. 2017.
-  T. Lei, X. Jia, Y. Zhang, L. He, H. Meng, and A. K. Nandi, “Significantly fast and robust fuzzy c-means clustering algorithm based on morphological reconstruction and membership filtering,” IEEE Trans. Fuzzy Syst., vol. 26, no. 5, pp. 3027-3041, Oct. 2018.
-  T. Lei, Y. Zhang, Y. Wang, S. Liu, and Z. Guo, “A conditionally invariant mathematical morphological framework for color images,” Inf. Sci., vol. 387, pp. 34-52, May 2017.
-  J. Cheng and J. C. Rajapakse, “Segmentation of clustered nuclei with shape markers and marking function,” IEEE Trans. Biomed. Eng., vol. 56, no. 3, pp. 741-748, Mar. 2009.
-  B. Peng, L. Zhang, and D. Zhang, “Automatic image segmentation by dynamic region merging,” IEEE Trans. Image Process., vol. 20, no. 12, pp. 3592-3605, Dec. 2011.
-  M. Bosch, C. M. Gifford, A. G. Dress, C. W. Lau, J. G. Skibo, and G. A. Christie, “Improved image segmentation via cost minimization of multiple hypotheses,” arXiv preprint arXiv:1802.00088, 2018.
-  P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik, “Contour detection and hierarchical image segmentation,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 5, pp. 898-916, May 2011.
-  P. Dollar and C. L. Zitnick, “Fast edge detection using structured forests,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 37, no. 8, pp. 1558-1570, Aug. 2015.
-  S. Hallman and C. C. Fowlkes, “Oriented edge forests for boundary detection,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2015, pp. 1732-1740.
-  X. Fu, C. Wang, C. Chen, C. Wang, and C.C. Jay Kuo, “Robust image segmentation using contour-guided color palettes,” in Proc. IEEE Conf. Int. Comput. Vis. (ICCV), Dec. 2015, pp. 1618–1625.
-  D. Comaniciu and P. Meer, “Mean shift: a robust approach toward feature space analysis,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 24, no. 5, pp. 603-619, May 2002.
A. Y. Ng, M. I. Jordan, and Y. Weiss, “On spectral clustering: Analysis and an algorithm,” inProc. Adv. Neural Inf. Proc. Syst. (NIPS), Dec. 2002, pp. 849-856.
-  J. Serra, Image Analysis and Mathematical Morphology, Part II: Theoretical Advances. London: Academic Press, 1988.
-  M. E. Plissiti, C. Nikou, and A. Charchanti, “Automated detection of cell nuclei in pap smear images using morphological reconstruction and clustering,” IEEE Trans. Inf. Technol. Biomed., vol. 15, no. 2, pp. 233-241, Mar. 2011.
-  P. Soille and M. Pesaresi, “Advances in mathematical morphology applied to geoscience and remote sensing,” IEEE Trans. Geosci. Remote Sens., vol. 40, no. 9, pp. 2042-2055, Sept. 2002.
-  W. Liao, M. D. Mura, J. Chanussot, R. Bellens, and W. Philips, “Morphological attribute profiles with partial reconstruction,” IEEE Trans. Geosci. Remote Sens., vol. 54, no. 3, pp. 1738-1756, Mar. 2016.
-  P. Soille, Morphological image analysis: principles and applications. Springer Science & Business Media, 2013.
-  D. Wang, “A multiscale gradient algorithm for image segmentation using watershelds,” Pattern Recognit., vol. 30, no. 12, pp. 2043-2052, Dec. 1997.
-  M. S. Miri and A. Mahloojifar, “Retinal image analysis using curvelet transform and multistructure elements morphology by reconstruction,” IEEE Trans. Biomed. Eng., vol. 58, no. 5, pp. 1183-1192, May 2011.
-  Y. Li, M. Xu, X. Liang, and W. Huang, “Application of bandwidth EMD and adaptive multiscale morphology analysis for incipient fault diagnosis of rolling bearings,” IEEE Trans. Ind. Electron., vol. 64, no. 8, pp. 6506-6517, Aug. 2017.
-  Y. Li, M. J. Zuo, J. Lin, and J. Liu, “Fault detection method for railway wheel flat using an adaptive multiscale morphological filter,” Mech. Syst. Signal Process., vol. 84, pp. 642-658, Feb. 2017.
-  J. Chen, C. Su, W. E. L. Grimson, J. Liu, and D. Shiue, “Object segmentation of database images by dual multiscale morphological reconstructions and retrieval applications,” IEEE Trans. Image Process., vol. 21, no. 2, pp. 828-843, Feb. 2012.
-  H. Shih and E. Liu, “Automatic reference color selection for adaptive mathematical morphology and application in image segmentation,” IEEE Trans. Image Process., vol. 25, no. 10, pp. 4665-4676, Oct. 2016.
-  Y. Teng, Y. Zhang, Y. Chen, and C. Ti, “Adaptive morphological filtering method for structural fusion restoration of hyperspectral images,” IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens., vol. 9, no. 2, pp. 655-667, Feb. 2016.
Y. Boykov and G. Funka-Lea, “Graph cuts and efficient nd image segmentation,”
Int. J. Comput. Vision, vol. 70, no. 2, pp. 109-131, Nov. 2006.
-  L. Grady, “Random walks for image segmentation,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 11, pp. 1768-1783, Nov. 2006.
-  M. Huang, W. Yang, Y. Wu, J. Jiang, W. Chen, and Q. Feng, “Brain tumor segmentation based on local independent projection-based classification,” IEEE Trans. Biomed. Eng., vol. 61, no. 10, pp. 2633-2645, Oct. 2014.
-  L. Najman and M. Schmitt, “Geodesic saliency of watershed contours and hierarchical segmentation,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 18, no. 12, pp. 1163-1173, Dec. 1996.
-  V. Grau, A. U. J. Mewes, M. Alcaniz, R. Kikinis, and S. K. Warfield, “Improved watershed transform for medical image segmentation using prior information,” IEEE Trans. Med. Imag., vol. 23, no. 4, pp. 447-458, Apr. 2004.
-  A. M. Mendonca and A. Campilho, “Segmentation of retinal blood vessels by combining the detection of centerlines and morphological reconstruction,” IEEE Trans. Med. Imag., vol. 25, no. 9, pp. 1200-1213, Sept. 2006.
-  T. H. Kim, K. M. Lee, and S. U. Lee, “Learning full pairwise affinities for spectral segmentation,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 7, pp. 1690-1703, Jul. 2013.
-  F. Galasso, M. Keuper, T. Brox, and B. Schiele, “Spectral graph reduction for efficient image and streaming video segmentation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2014, pp. 49-56.
-  C. Yang, L. Bruzzone, H. Zhao, Y. Tan, and R. Guan, “Superpixel-based unsupervised band selection for classification of hyperspectral images,” IEEE Trans. Geosci. Remote Sens., vol. 56, no. 12, pp. 7230-7245, Dec. 2018.
-  P. Theologou, I. Pratikakis, and T. Theoharis, “Unsupervised spectral mesh segmentation driven by heterogeneous graphs,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 39, no. 2, pp. 397-410, Feb. 2017.
-  R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, and S. Süsstrunk, “Slic superpixels compared to state-of-the-art superpixel methods,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 11, pp. 2274-2282, Nov. 2012.
-  J. Chen, Z. Li, and B. Huang, “Linear spectral clustering superpixel,” IEEE Trans. Image Process., vol. 26, no. 7, pp. 3317-3330, Jul. 2017.
-  X. Wei, Q. Yang, Y. Gong, N. Ahuja, and M. Yang, “Superpixel Hierarchy,” IEEE Trans. Image Process., vol. 27, no. 10, pp. 4838-4849, Oct. 2018.
-  Z. Zhang, F. Xing, H. Wang, Y. Yan, Y. Huang, X. Shi, and L. Yang, “Revisiting graph construction for fast image segmentation,” Pattern Rcognit., vol. 78, pp. 344-357, Jun. 2018.
-  Y. Xu, E. Carlinet, T. Géraud, and L. Najman, “Hierarchical segmentation using tree-based shape spaces,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 39, no. 3, pp. 457-469, Mar. 2017.
-  J. Syu, S. Wang, and L. Wang, “Hierarchical image segmentation based on iterative contraction and merging,” IEEE Trans. Image Process., vol. 26, no. 5, pp. 2246-2260, May 2017.
-  J. Pont-Tuset, P. Arbelaez, J. T. Barron, F. Marques, and J. Malik, “Multiscale combinatorial grouping for image segmentation and object proposal generation,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 39, no. 1, pp. 128-140, Jan. 2017.
-  Y. Chen, D. Dai, J. Pont-Tuset, and L. V. Gool, “Scale-aware alignment of hierarchical image segmentation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit.(CVPR), Jun. 2016, pp. 364-372.
-  F. Zohrizadeh, M. Kheirandishfard, and F. Kamangar, “Image segmentation using sparse subset selection,” in Proc. IEEE Winter Conf. Appl. Comput. Vis.(WACV), Mar. 2018, pp. 1470-1479.
-  T. Cour, F. Benezit, and J. Shi, “Spectral segmentation with multiscale graph decomposition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Sept. 2005, pp. 1124-1131.
-  D. Hoiem, A. A. Efros, and M. Hebert, “Recovering occlusion boundaries from an image,” Int. J. Comput. Vision, vol. 91, no. 3, pp. 328-346, Feb. 2011.
-  S. Kim, C. D. Yoo, S. Nowozin, and P. Kohli, “Image segmentation using higher-order correlation clustering,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 36, no. 9, pp. 1761-1774, Sept. 2014.
-  X. Wang, Y. Tang, S. Masnou, and L. Chen, “A global/local affinity graph for image segmentation,” IEEE Trans. Image Process., vol. 24, no. 4, pp. 1399-1411, Apr. 2015.
-  J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2015, pp. 3431-3440.
S. Zheng, S. Jayasumana, B. Romera-Paredes, V. Vineet, Z. Su, D. Du, C. Huang, and P. H. S. Torr, “Conditional random fields as recurrent neural networks,” inProc. IEEE Intl. Conf. Comput. Vis. (ICCV), Dec. 2015, pp. 1529-1537.
-  R. Gadde, V. Jampani, M. Kiefel, D. Kappler, and P. V. Gehler, “Superpixel convolutional networks using bilateral inceptions,” in Proc. Eur. Conf. Comput. Vis. (ECCV), Oct. 2016, pp. 597-613.
-  L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 40, no. 4, pp. 834-848, Apr. 2018.