The decomposition of an image into homogeneous areas, called superpixels
, has become a very popular pre-processing step in many image processing and computer vision frameworks. For most superpixel-based applications such as object recognition[1, 2], labeling  or tracking [4, 5], the use of regular decompositions [6, 7, 8, 9, 10, 11] where superpixels roughly have the same size and regular shapes is necessary. With such regularity, accurate superpixel features can be computed and relevant information can be extracted from their boundaries, while with irregular decompositions [12, 13, 14, 15, 16], superpixels can have different sizes, noisy boundaries and stretched shapes.
Most of recent methods allow the user to tune a compactness parameter [6, 15, 7, 8, 9, 11] to produce superpixels of variable regularity, that may affect the performances for a given application. The search for optimal results and comparison between methods should thus be performed for several regularity settings . Moreover, most superpixel decomposition methods compute a trade-off between adherence to contours and shape compactness. The regularity is hence an argument to compare decompositions with similar contour adherence performances. Hence, clear regularity definition and measure are necessary to evaluate superpixel methods.
In the superpixel literature, the regularity notion is usually called compactness, and is mainly evaluated by the circularity metric . Although the circularity metric has since been considered in many works [18, 15, 10, 11], and large benchmarks [19, 20], the literature usually refers to the regularity as the ability to produce convex shapes with non noisy boundaries. Moreover, for tracking application, the aim is to find one-to-one superpixel associations across decompositions, so compactness should be high for convex shapes with balanced pixel repartition. In , the circularity is also discussed since it does not consider the square as a highly regular shape. Figure 1 compares two decompositions computed with  using initial square and hexagonal grids. Hexagons provide much higher circularity, although both shapes should be considered as regular. Since most methods start from a square grid and iteratively refine the superpixel borders, it would make sense to have high regularity for a square decomposition, and to have a measure that is in line with the compactness parameter evolution, that produces squares when set to maximum value.
|C = SRC =||C = SRC =|
|C = SRC =||C = SRC =|
Contributions. In this work, we first demonstrate that the circularity is not adapted to the superpixel context. We propose a new shape regularity criteria (SRC), that better expresses the regularity notion by considering the following aspects: shape convexity, balanced pixel repartition and contour smoothness. Finally, the relevance and the robustness of SRC are demonstrated with state-of-the-art superpixel methods applied to images of the Berkeley segmentation dataset .
2 Shape Regularity Criteria
2.1 Regularity Definition
The circularity C, introduced in  for superpixel evaluation, expresses the compactness of a shape as follows:
where is the shape perimeter and denotes the cardinality. This metric considers the compactness as the resolution of the isoperimetrical problem that aims to find the largest shape for a given boundary length. As stated in the introduction, the circularity does not express the shape regularity but only favor circular shapes. We propose a new shape regularity criteria (SRC) composed of three metrics, that each evaluates an aspect of the superpixel regularity.
Solidity. To evaluate the global convexity of a shape, we propose to consider its solidity (SO), i.e., the overlap with its convex hull . Such convex hull, containing the whole shape , is illustrated in Figure 2 and can be computed using Delaunay triangulation. Perfectly convex shapes such as squares or circles will get the highest solidity:
The overlap with the convex hull is not sufficient to express the global regularity. Convex shapes such as ellipses or lines have the highest SO, but should be considered as perfectly regular only with a balanced pixel repartition. To measure it, we define a variance term:
, the standard deviations of the pixel positionsand within . if, and only if, . In this case, the spatial repartition of the pixels around the barycenter is considered as well balanced.
Contour smoothness. Finally, the regularity of the superpixel borders must be considered. The convexity measure (CO) compares the number of boundary pixels of the shape and the one of its convex hull. Although this measure is generally in line with SO, it is mostly dependent on the border smoothness and penalizes noisy superpixels:
The proposed shape regularity criteria (SRC) is a combination of all regularity aspects and is defined as follows:
where is composed of superpixels , whose sizes are considered to reflect the overall regularity. The combination of SO and CO is related to the ratio between the Cheeger constant of the shape and its convex hull . As stated in , the Cheeger measure is a pertinent tool to evaluate both convexity and boundary smoothness.
2.2 Circularity vs SRC
Figure 3 compares the different metrics on synthetic shapes split into three groups, and generated with smooth (top) and noisy borders (bottom). The circularity presents several drawbacks. First, it is much lower for the Square than for the Circle and Hexagon, and even less than for the Ellipse. As shown in Figure 1, this is an issue when comparing superpixel methods starting from square and hexagonal grids, such as . It also drops for the Cross and Bean although they are visually regular. In the bottom part of Figure 3, the circularity appears to be very dependent on the boundary smoothness, since noisy shapes have similar circularity and cannot be differentiated. Finally, standard shapes with smooth borders can have much higher circularity than regular ones. For instance, the noisy Square has a lower circularity than the Bean.
As can be seen in Figure 3, SO, and CO independently taken are not sufficient to express the compactness of a shape. The proposed SRC combines all defined regularity properties. For instance, SO is representative for all shapes, except for the Ellipse and W, since they both have large overlap with their convex hull. penalizes the Ellipse since it does not have a balanced pixel repartition, and CO considers the large amount of contour pixels in the W shape.
The three regular shapes get the highest SRC (), and the standard shapes have similar measures. Since our metric is less sensitive to slight contour smoothness variations, SRC also clearly separates the three shape groups in the noisy case, contrary to C. Moreover, regular but noisy shapes, have comparable SRC to smooth standard ones, whereas noisy standard shapes still have higher SRC than irregular shapes with smooth contours, which can be considered as a relevant evaluation of regularity. Figure 4 also represents the regularity measures, where SRC appears to more clearly separates the three shape groups in both smooth and noisy cases.
|Regular shapes||Standard shapes||Irregular shapes|
Finally, circularity appears to very dependent on the shape size in Figure 5. As observed in , due to discrete computation, it can be superior to 1 (we threshold its value in Figure 5), and it drops with larger shapes. Hence, comparisons of methods on this metric would be relevant only with decompositions having the same superpixel number, i.e., superpixels with approximately the same size. Contrary to circularity, SRC provides much more consistent measure according to the superpixel size, e.g., the Square always has a SRC equal to 1.
Finally, a decomposition into superpixels can be seen as a preprocessing, similar to the multi-resolution approach, but where the regions are no longer regular blocks. Since most decomposition methods starts from a square grid, and iteratively refine the superpixel borders, it would thus make sense to obtain the highest regularity measure with a square decomposition, i.e., regularity parameter set to infinite.
3 Improved Evaluation of Superpixel Methods
3.1 Validation Framework
To compare results of state-of-the-art methods, we consider the standard Berkeley segmentation dataset (BSD) , containing 200 test images of
pixels. At least 5 human segmentations are provided to compute evaluation metrics of contour adherence and respect of image objects.
|C = SRC =||C = SRC =||C = SRC =||C = SRC =||C = SRC =|
|SLIC ||ERGC ||WP ||LSC ||SCALP |
Methods. We compare the results of the following state-of-the-art methods that enable to set the regularity of their decompositions: SLIC , CRS [conrad2013], ETPS  and SCALP . All these methods refine the superpixel borders of an initial grid, using color, spatial distance or contour intensity terms (see papers for method specificities). As in Figure 1, we compare the results of SLIC initialized with a square (SLIC-s) and hexagonal grid (SLIC-o). A decomposition example of each considered method is shown in Figure 6.
|C = SRC =||C = SRC =||C = SRC =||C = SRC =||C = SRC =|
3.2 Evaluation of Superpixel Methods
In this section, we consider state-of-the-art methods that enable to set a compactness parameter: SLIC , ERGC , Waterpixels (WP) , LSC  and SCALP . Decomposition examples with the associated Delaunay graph are illustrated in Figure 6. In Table 2, we demonstrate that SRC provides a more robust regularity measure of superpixel methods. We compute decompositions at several scales (from 50 to 1000 superpixels), with the default compactness settings, and average the results on the 200 BSD images. SRC is more robust to the superpixel scale, since it reports lower variance.
We also consider the standard undersegmentation error (UE) metric measuring the number of pixels that belong to several image objects, and the boundary recall (BR), that measures the detection of ground truth contours (defined for instance in ). The scale is set to 300 superpixels, and the UE and BR results are averaged on the BSD images, decomposed with different compactness settings. Hence, for each method, UE and BR results are computed on several regularity levels. We consider the optimal UE and BR of each method, and the standard deviation between the corresponding C and SRC measures are respectively and for UE, and and for BR. The reduced variance demonstrates the relevance of our metric, since optimal decomposition performances of different methods, evaluated with UE and BR, are obtained for more similar SRC.
Since superpixels define a lower-level representation,
they can be used to compress the image content [levinshtein2009, zeng2011].
Each superpixel color channel
is approximated by a polynomial of n-th order
and contained into the polynomial coefficients.
We propose to compute these coefficients by a least mean square approach
and to evaluate the mean square error between the reconstructed and initial images.
A compression example is illustrated in Figure 9.
|C = SRC =||C = SRC =||C = SRC =|
3.3 Robustness to Noise
In this section, we further demonstrate the robustness of SRC to noisy boundaries. We randomly perturb the trade-off between contour adherence and compactness of SLIC  to generate noisy decompositions (see Figure 10). We compute decompositions for different values of compactness parameter , and we report the C and SRC measures averaged on the BSD in Figure 11. The circularity appears to be impacted by the noisy borders, to such an extent that it does not express the global shape regularity that increases with , since it drops from . Nevertheless, SRC is proportional to the evolution of the compactness parameter of , demonstrating that it better expresses the regularity of a decomposition, and is not mainly sensitive to contour smoothness.
3.4 Global Regularity Evaluation
This work and , that introduced circularity, focus on a local compactness definition, where each superpixel is independently evaluated. Such local evaluation seems in line with the visual regularity of the graphs (see Figure 6), but it does not consider the global size regularity within the whole decomposition. Although most methods such as [6, 7, 11] produce superpixels approximately containing the same number of pixels, other methods may produce partitions of irregular sizes [12, 14, 16]. In Figure 12, we represent an example of SLIC superpixels  and standard quadtree partition, which produces larger squares in areas with lower color variance. Decompositions are shown with their associated Delaunay graph, connecting barycenters of adjacent superpixels. Since it only produces square areas, the local regularity of such quadtree partition is high (), although the associated graph shows inconsistent distances between the barycenters of connected superpixels, contrary to the one of . Numerical evaluation of graph regularity is a complex issue , and future works will investigate the global regularity notion and measure in the superpixel context.
In this paper, we focus on the notion of regularity, i.e., compactness in the superpixel context. We consider that a regular shape should verify these aspects: convexity, balanced repartition and contour smoothness, and we define a new metric that better expresses the local regularity, and is robust to scale and noise. Most of decomposition methods tend to achieve a trade-off between segmentation accuracy and shape regularity. This work enables to relevantly compare superpixel algorithms and provides accurate regularity information on the decomposition inputs of superpixel-based pipelines. Nevertheless, local compactness measure does not express the global regularity notion, which will be investigated in future works.
Finally, we further demonstrate the SRC relevance by showing its correlation with the performances of a superpixel-based labeling framework. In , a structure to compare superpixel neighborhoods is defined and applied to exemplar-based labeling. The method uses the Labeled Faces in the Wild (LFW) dataset [huang2007], which contains 927 test images to label and 1500 training images with superpixel labels (face, hair, background). For such application, the regularity of the adjacency graph increases the comparison accuracy between superpixels, the matching relevance, and therefore, the labeling performances .
The images are decomposed using SLIC  to produce smooth and noisy superpixels with noisy borders (see Figure 13). We compute the labeling performances for several regularity settings and report these results in Figure 14, with the corresponding average regularity. measured by circularity C and by our shape regularity criteria SRC. The circularity is greatly impacted by the noisy borders although it does not alter the global regularity of the superpixel shapes or the adjacency graph complexity. Nevertheless, in both cases, our metric is in line with the labeling performances, demonstrating that it better expresses the regularity of a decomposition.
|C=0.345 SRC=0.454||C=0.345 SRC=0.454|
|Initial image||SLIC (smooth)||SLIC (noisy)|
-  J. Tighe and S. Lazebnik, “SuperParsing: Scalable nonparametric image parsing with superpixels,” in Proc. of European Conference on Computer Vision, 2010, pp. 352–365.
-  S. Gould, J. Zhao, X. He, and Y. Zhang, “Superpixel graph label transfer with learned distance metric,” in Proc. of European Conference on Computer Vision, 2014, pp. 632–647.
-  R. Giraud, V.-T. Ta, A. Bugeau, P. Coupé, and N. Papadakis, “SuperPatchMatch: An algorithm for robust correspondences using superpixel patches,” IEEE Trans. on Image Processing, 2017.
-  S. Wang, H. Lu, F. Yang, and M. H. Yang, “Superpixel tracking,” in Proc. of IEEE International Conference on Computer Vision, 2011, pp. 1323–1330.
J. Chang, D. Wei, and J. W. Fisher,
“A video representation using temporal superpixels,”
Proc. of IEEE Conf. on Computer Vision and Pattern Recognition, 2013, pp. 2051–2058.
-  R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, and S. Süsstrunk, “SLIC superpixels compared to state-of-the-art superpixel methods,” IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 34, no. 11, pp. 2274–2282, 2012.
-  V. Machairas, M. Faessel, D. Cárdenas-Peña, T. Chabardes, T. Walter, and E. Decencière, “Waterpixels,” IEEE Trans. on Image Processing, vol. 24, no. 11, pp. 3707–3716, 2015.
Z. Li and J. Chen,
“Superpixel segmentation using linear spectral clustering,”in Proc. of IEEE Conf. on Computer Vision and Pattern Recognition, 2015, pp. 1356–1363.
-  J. Yao, M. Boben, S. Fidler, and R. Urtasun, “Real-time coarse-to-fine topologically preserving segmentation,” in Proc. of IEEE Conf. on Computer Vision and Pattern Recognition, 2015, pp. 2947–2955.
-  Y. Zhang, X. Li, X. Gao, and C. Zhang, “A simple algorithm of superpixel segmentation with boundary constraint,” IEEE Trans. on Circuits and Systems for Video Technology, 2016.
-  R. Giraud, V.-T. Ta, and N. Papadakis, “SCALP: Superpixels with contour adherence using linear path,” in International Conference on Pattern Recognition, 2016, pp. 2374–2379.
-  P. Felzenszwalb and D. Huttenlocher, “Efficient graph-based image segmentation,” International Journal of Computer Vision, vol. 59, no. 2, pp. 167–181, 2004.
-  M. Y. Liu, O. Tuzel, S. Ramalingam, and R. Chellappa, “Entropy rate superpixel segmentation,” in Proc. of IEEE Conf. on Computer Vision and Pattern Recognition, 2011, pp. 2097–2104.
-  M. Van den Bergh, X. Boix, G. Roig, B. de Capitani, and L. Van Gool, “SEEDS: Superpixels extracted via energy-driven sampling,” in Proc. of European Conference on Computer Vision, 2012, pp. 13–26.
-  P. Buyssens, M. Toutain, A. Elmoataz, and O. Lézoray, “Eikonal-based vertices growing and iterative seeding for efficient graph-based segmentation,” in Proc. of IEEE International Conference on Image Processing, 2014, pp. 4368–4372.
-  A. Rubio, L. Yu, E. Simo-Serra, and F. Moreno-Noguer, “BASS: Boundary-aware superpixel segmentation,” in International Conference on Pattern Recognition, 2016, pp. 2824–2829.
-  A. Schick, M. Fischer, and R. Stiefelhagen, “Measuring and evaluating the compactness of superpixels,” in International Conference on Pattern Recognition, 2012, pp. 930–934.
-  M. Reso, J. Jachalsky, B. Rosenhahn, and J. Ostermann, “Temporally consistent superpixels,” in Proc. of IEEE International Conference on Computer Vision, 2013, pp. 385–392.
-  A. Schick, M. Fischer, and R. Stiefelhagen, “An evaluation of the compactness of superpixels,” Pattern Recognition Letters, vol. 43, pp. 71–80, 2014.
-  D. Stutz, A. Hermans, and B. Leibe, “Superpixels: An evaluation of the state-of-the-art,” Computer Vision and Image Understanding, 2017.
-  D. Martin, C. Fowlkes, D. Tal, and J. Malik, “A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics,” in Proc. of IEEE International Conference on Computer Vision, 2001, vol. 2, pp. 416–423.
-  V. Caselles, G. Facciolo, and E. Meinhardt, “Anisotropic cheeger sets and applications,” SIAM Journal on Imaging Sciences, vol. 2, no. 4, pp. 1211–1254, 2009.
-  T. Roussillon, I. Sivignon, and L. Tougne, “Measure of circularity for parts of digital boundaries and its fast computation,” Pattern Recognition, vol. 43, no. 1, pp. 37–46, 2010.
-  R. Diestel, “Graph theory,” Graduate Texts in Mathematics, 1997.