I Introduction
The growing in resolution and quantity of image data has highlighted the need for efficient under representations to reduce the computational load of computer vision pipelines. In this context, superpixels were popularized with [1] to reduce the image domain to irregular regions having approximately the same size and homogeneous colors. Contrary to regular multiresolution schemes, a result at the superpixel scale can be very close to the optimal one at the pixel scale. Superpixels have been successfully used in many applications such as: semantic segmentation [23, 26]
, optical flow estimation
[18] or style transfer [15]. The main issue to deal with is the irregularity between all regions that may prevent from using the standard neighborhoodbased tools. Nevertheless, this issue has been addressed in graphbased approaches [11], using neighborhood structure [8], or within deep learning frameworks
[24].At the same time, the use of new acquisition devices capturing wide angles, such as fish eyes, generally covering a 360 field of view has become more and more popular. These devices offer a global capture of the environment, particularly interesting for applications such as autonomous driving. With a depthaware system, the intensity can be projected on a 3D point cloud. Otherwise, the image sphere is generally projected on a discrete 2D plane to generate an equirectangular image inducing distorsions [34]. In this context, several works, e.g., [5, 21] have used standard planar superpixels although they do not consider the geometry distorsions in the equirectangular image, that may limit the segmentation accuracy and their interpretation on the spherical acquisition space [33].
Many superpixel approaches have been proposed over the years, most exclusively to segment standard planar images. These methods use watershed [17], region growing [14], eikonalbased [4], graphbased energy [16], or even coarsetofine algorithms [31]. A significant breakthrough was obtained with the SLIC method [1], locally adapting a means algorithm on a tradeoff between distances in the spatial and CIELab color space to generate superpixels. The method has few parameters and a low processing time, but is limited to adjust to different image content and to accurately capture object borders. Many improvements of SLIC have been proposed using boundary constraint [32], advanced feature space [6], noniterative clustering [2], a shortest path approach [10], or even deep learning processes [24] although these last methods present the usual limitations, in terms of resources, training time, large dataset needed, and applicability to other images.
For spherical images, The unsupervised segmentation approach of [7] has been extended in [30], but generates very irregular regions, not considered as superpixels. More recently, the SLIC method was extended to produce spherically regular superpixels [33]. Pixels are projected on the unit sphere for computing the spatial constraints and produce regular superpixels in the spherical space. Besides the display interest, the respect of the acquisition space geometry enables to more accurately segment the image objects [33]. Nevertheless, this approach comes with the same limitations as SLIC, i.e., limited adaptability to different contexts with severe non robustness to textures or noise due to the use of a standard color feature space, and no explicit integration of contour information. These limitations are addressed in [10], for which authors obtain significantly higher accuracy for standard planar image segmentation by considering the color and contour features along the shortest path between the pixel and the superpixel.
Contributions
In this paper, we address the limitations of the spherical approach of [33], by proposing in Section II a new superpixel method called SphSPS (Spherical Shortest Pathbased Superpixels). SphSPS is based on the same spherical means approach of [33] but exploites more advanced features [6] and generalizes the notion of shortest path [10], to the acquisition space, here the spherical one. To this end, a dedicated fast shortest path algorithm is defined to integrate the information of this large number of pixels into the method.
SphSPS generates in very limited processing time accurate and regular spherical superpixels (see Figure 1). To relevantly evaluate the regularity aspect in the spherical space, we also propose a generalization of the global regularity measure [9] (Section III). SphSPS obtains higher segmentation performance than the stateoftheart methods on the reference 360 spherical panorama segmentation dataset [25] (Section IV).
Standard planar superpixels using [6]  

Spherical superpixels using the proposed SphSPS method 
Ii Spherical Shortest Pathbased Superpixels
To introduce SphSPS, we first present the means method [1] (Section IIA) and its spherical adaptation [33] (Section IIB
). Then, we present the feature extraction method on a planar shortest path
[10] (Section IIC1) and our generalization to the spherical space (Sections IIC2 and IIC3).Iia Planar Kmeans Iterative Clustering
SphSPS is based on the SLIC algorithm [1] using an iteratively constrained means clustering of pixels. Superpixels are first initialized over the image as blocks of size , described by the average CIELab colors and barycenter position of pixels in . The clustering for each pixel relies on a color , and a spatial distance . At each iteration, each superpixel is compared to all pixels , of color at position , within a area around its barycenter . A pixel is associated to the superpixel minimizing the distance defined as:
(1) 
with , the tradeoff parameter setting the shape regularity. Finally, a postprocessing step ensures the region connectivity.
IiB Spherical Geometry
In the spherical acquisition space, vertical and horizontal coordinates are respectively projected to the meridians and circles of latitude, so the spherical image has a width twice superior to its height. SphSPS is based on the same adaptation of the planar means method to the spherical geometry as [33], that requires three steps. The first one is the initialization of the superpixels. To spread the barycenters along the sphere, we also use the Hammersley sampling [27]. The second step is the search area that must consider the proximity of pixels in the spherical space. For instance, superpixels on the image top and bottom have larger search areas. This area , is defined for each superpixel of barycenter as:
(2) 
with the polar angle corresponding to the th row for an image of height and width , and the average superpixel size . The 360 geometry aspect must also be handled to horizontally connect the pixels. This is done with a mirror effect when the search region falls outside the image boundaries [33]. The third aspect is the computation of the spatial distance, that must also be done in the spherical space. For each image pixel the projection on the 3D acquisition space is computed as:
(3) 
Note that , when computed from , so we map on the image domain with , if .
IiC Generalized Shortest Path Method
IiC1 Feature extraction on a shortest path
In [10], color and contour information of pixels on the planar shortest path between a pixel and a superpixel are used to improve segmentation accuracy and regularity. SphSPS also integrates these features and has the same clustering distance than [10]. Nevertheless, in the following, the shortest path differs since we compute it in the spherical space.
First, to relevantly increase the regularity and prevent non convex shapes to appear, the color distance of the pixels on the path is added to the color distance such that:
(4) 
with a tradeoff parameter usually set to .
The contour information can also be considered to increase the respect of objects borders using a contour map , with values between 0 and 1. A contour term is defined as:
(5) 
with the parameter penalizing the crossing of a contour.
The final clustering distance of SphSPS is defined as:
(6) 
with the spherical spatial distance using the cosine dissimilarity as [33], and the proposed spherical shortest path computed as follows.
IiC2 Generalized shortest path
In Figure 2, we compare shortest paths in the planar space, as in [10], and in the spherical one as in SphSPS. With planar images, since no distorsions are introduced between the acquisition and the image space (), they are considered equivalent. Hence, the shortest path reduces to a linear path and can be easily computed with a discrete algorithm [3]. Nevertheless, in general, the shortest path should be computed in the acquisition space, than can be spherical or even circular using fisheyes with different capture angles. Hence, the generalized formulation of the shortest path problem computes it in the acquisition space () and projects it back to the planar image space:
(7) 
IiC3 Shortest path in the spherical space
The spherical shortest path consists in following the geodesic along the sphere [12], lying on a great circle (in orange color in Figures 2 and 3), containing the two points and the sphere center. Tangential methods to extract waypoints on the great circle have been formalized for instance in [13]. Nevertheless, such theoretical approaches use many trigonometric computations that impact the performance. In the following we propose a simpler reformulation of the spherical geodesic path problem.
Fast geodesic path implementation
For each comparison of a pixel at to a superpixel of barycenter , we propose to first compute an orthogonal coordinate system within their great circle. To build such system, we perform an orthogonalization process to get the position
, creating an orthogonal vector to
within the great circle such as:(8) 
with the scalar product already computed for the spatial distance . Then, the angle between the two points is simply obtained with . Finally, the geodesic path is defined within , by starting from the pixel position, and linearly increasing the angle shift from to , to reach the superpixel barycenter such as:
(9) 
with , intermediate angles to linearly sample points between the two positions. The geodesic path is finally projected in the planar space (3) to get (7). By this way, we obtain the shortest spherical path coordinates with simple calculations, dividing the processing time by a factor 2 compared to tangential approaches. An example of spherical shortest path on a great circle with the the computation of the corresponding coordinate system is illustrated in Figure 3.
Optimization
First, for each superpixel, we can store the color distance computed to each tested pixel, reducing the processing time by . Then, contrary to the planar linear path algorithm [3] we can exploit path redundancy. If the path of a pixel to a superpixel crosses a previously computed path to the same superpixel, the rest of the path should be the same since they lie on the same great circle. So we can also store the average color and contour information on the path for each crossed pixel. This is done efficiently using recursive implementation. By this way, for many pixels we are able to directly access the large quantity of information contained in the shortest path, again reducing the processing time by .
Iii Generalized Global Regularity Measure
Superpixels tend to optimize a color and spatial tradeoff, so metrics should mainly evaluate object segmentation and regularity performances. This last aspect has been sparsely evaluated although most methods have a regularity parameter that may significantly impact superpixelbased pipelines. Moreover, the standard compactness metric [22], which is the only one extended to the spherical space [33] was proven very limited [9]. In this section, we propose a new way to relevantly evaluate the regularity in the acquisition space.
Iiia Limitation of the Compactness Measure
In [33], the compactness measure C [22] is extended to the spherical case. The regularity of a segmentation is only seen as a notion of circularity, computed as:
(10) 
with = the spherical isoperimetric quotient [20]. Hence, each superpixel is independently compared to a circular shape, such that for instance, ellipses can have higher C measures than squares. In [9], this metric has been proven highly sensitive to boundary noise and inconsistent with the superpixel size. Moreover, in [33] it even fails to differentiate spherical and planarbased methods.
IiiB Generalized Global Regularity Metric
IiiB1 Global regularity metric
In [9], a global regularity metric (GR) is introduced, to address the issues of the compactness. First, the Shape Regularity Criteria (SRC) is defined to robustly evaluate the convexity, the contour smoothness, and the 2D balanced repartition of each superpixel. Convexity and smoothness properties are computed with respect to the discrete convex hull containing the shape.
As for the compactness C (10), SRC is independently computed for each superpixel, so [9] also introduces a Smooth Matching Factor (SMF) to evaluate the consistency of superpixel shapes. Each superpixel is compared, after registration on its barycenter, to the average superpixel shape, created from the superposition of all registered superpixels.
Finally, the notion of regularity is defined by the GR (Global Regularity) metric combining these two metrics such that:
(11) 
IiiB2 Generalization in the acquisition space
Ideally the regularity should be evaluated in the acquisition space. In our context, in the spherical acquisition space gives , a set of 3D positions on the unit sphere (3). GR being based on the computation of convex hull, and barycenter registration, it cannot be directly applied to such point clouds in .
To generalize the metric, we propose to simply project the 3D points of on a discrete 2D plan, and then apply the initial GR. The whole process is illustrated in Figure 4. To do so, we first project a superpixel in the discrete image space to its acquisition one, here to get a spherical point cloud
. Then, we apply a Principal Component Analysis (PCA) on
, and project the points on its two most significant eigenvectors to reduce to a 2D point cloud
. Finally, a downsampling is performed to obtain a discrete 2D shape . By this way, each superpixel shape has a relevant discrete projection in the acquisition space. The proposed Generalized Global Regularity (GGR) metric is defined as:(12) 
With the proposed GGR metric, a gap is now visible such that no planar methods have higher regularity than spherical ones for a given number of superpixels (Section IVC).
Iv Results
Iva Validation Framework
IvA1 Dataset
We consider the Panorama Segmentation Dataset (PSD) [25], containing 75 360 equirectangular images of pixels, having between 115 and 1085 segmented objects with an average size of 1334 pixels. These images are taken from the standard spherical dataset SUN360 [28], and accurate groundtruth segmentations are provided by [25].
IvA2 Metrics
To relevantly evaluate SphSPS performances and compare to stateoftheart methods, we use the superpixel metrics recommended in [9], for several superpixel numbers. The main aspects to evaluate are the object segmentation and spatial regularity performances, which is robustly evaluated in the acquisition space with the proposed GGR metric (12).
For the segmentation aspect, the standard measure is the Achievable Segmentation Accuracy (ASA) [16], highly correlated to the Undersegmentation Error [19] as shown in [9]. The ASA measures the overlap of a superpixel segmentation with the ground truth objects, denoted , such as:
(13) 
The BoundaryRecall (BR) is the commonly employed metric to evaluate the detection of the ground truth contours by the boundaries of the superpixels such that:
(14) 
with a distance threshold set to pixels [9], and when is true and otherwise. To prevent methods generating superpixels with fuzzy borders to get high performances [9], BR results are compared to the Contour Density (CD), i.e., the number of pixels of superpixel borders.
The standard PrecisionRecall curves can also be represented to illustrate the overall object contour detection performances. These are computed on a contour probability map
generated by averaging the superpixel borders obtained at different scales . This map is thresholded by several intensities to get a binary contour map. For each threshold, the Precision (PR), the percentage of accurate detection among the superpixel borders, is computed with the BR measure. For all PR curves, to synthesize the contour detection performance, we also report the maximum on all thresholds of the Fmeasure defined as:(15) 
IvA3 Parameter settings
SphSPS was implemented with MATLAB using CMEX code, on a standard Linux computer with 12 cores at 2.6 GHz with 64GB of RAM. Contrary to [33], using the 3 average color features of the CIELab space, we use the 6 CIELab dimension space of [6], also including the features of neighboring pixels [10]. In the shortest path, pixels are considered (9). The number of iterations is set to 5, and the parameter (4), setting the tradeoff between the central pixel and the ones on the shortest path, is set to as in [10]. When used, the contour prior is computed from [29] and set to (5). Finally, the parameter (6) is empirically set to to provide a visually satisfying tradeoff between the respect of object contours and spatial regularity.
(a) Initial image  (b) 3Lab, == 
6Lab==  6Lab ==  6Lab= = 
IvB Impact of Contributions
In this section, we show the impact of contributions within SphSPS. We report for different distance settings the contour detection PR/BR curves, with the maximum Fmeasure (15), and the regularity GGR (12) curves in Figure 5, and a zoom on a segmentation example in Figure 6. With a 3 feature dimension space, SphSPS reduces to the spherical SLIC algorithm [33]. With the 6 dimension space, SphSPS uses the CIELab features of [6], and the neighboring pixels information as in [10], with the color distance (4), and , the contour information on the shortest path (5).
We demonstrate that each contribution enables to improve the segmentation performances. We can especially observe that the color distance on the shortest path, that strengthens the superpixel convexity and homogeneity, indeed provides much more regular superpixels while also increasing the accuracy.
IvC Comparison with the StateoftheArt Methods
We compare the performances of the proposed SphSPS approach to the ones of the stateoftheart methods. We consider the planar methods SLIC [1], LSC [6], SNIC [2] and SCALP [10], and the spherical approach SphSLIC [33] in 2 different settings, i.e., considering the Euclidean (SphSLICEuc) and the Cosine (SphSLICCos) distances (see Section IIB). To ensure fair comparison, planar and SphSLICEuc methods are used with their default settings, since they provide a good tradeoff between accuracy and regularity. Note that for the SphSLICCos method [33], results are reported for the regularity setting optimizing the segmentation accuracy, since low performances were obtained with default settings.
In Figure 7, we report the contour detection results measured by PR/BR curves with Fmeasure (15), and BR/CD (14), the segmentation of objects with ASA (13), and regularity with the proposed GGR metric (12), obtained for several numbers of superpixels. SphSPS overall obtains the best segmentation results, with for instance the higher Fmeasure (), and significantly outperforms the other spherical method SphSLIC, in both distance modes, while producing very regular superpixels. Note that even without contour prior (), i.e., only using color information on the shortest path, SphSPS still significantly outperforms the other stateoftheart methods. We also observe that by using a linear path approach, SCALP [10] degrades the segmentation accuracy of LSC [6]. This result highlights the need for considering our spherical shortest path instead of the linear one.
The regularity measured with the proposed GGR (12) appears to be very relevant and able to differentiate planar and spherical methods. It evaluates the convexity and contour smoothness of each superpixel along with their consistency, while C (10) is only based on a non robust and independent circularity assumption. Hence, with GGR, the regularity in the spherical space is accurately measured such that no planar methods have higher regularity than spherical ones for a given number of superpixels, contrary to C [33].
In Figure 8, we show segmentation examples of SphSPS compared to the stateoftheart methods, on 360 equirectangular images and projected on the unit sphere. SphSPS produces regular superpixels in the spherical space and accurately captures the object contours compared to the other methods.
Finally, in terms of processing time, the relevance of our features enables SphSPS to fastly converge in a low number of iterations. For instance, only using the 6 dimensional feature space [6], SphSPS generates superpixels in s per image of size pixels and already obtains higher accuracy () than the stateoftheart methods (see Figure 5). With the significant optimizations proposed in Section IIC3, SphSPS can use the information on the shortest path to obtain significantly higher accuracy in only s, i.e., faster than existing spherical approaches [33]. Moreover, with basic multithreading, we easily reduce the processing time of our implementation to s to further facilitate the use of SphSPS^{1}^{1}1Available code at: https://github.com/rgiraud/sphsps.
LSC [6] 
SphSLICEuc [33] 


SNIC [2] 
SphSLICCos [33] 

SCALP [10] 
SphSPS 

LSC [6] 
SphSLICEuc [33] 

SNIC [2] 
SphSLICCos [33] 

SCALP [10] 
SphSPS 
V Conclusion
In this work, we generalize the shortest path approach between a pixel and a superpixel barycenter [9] to the case of spherical images. We show that the complexity resulting of the large number of pixels to process can be extremely reduced using the path redundancy on the 3D sphere. Color features on this path enable to generate both very accurate and regular superpixels. Moreover, SphSPS can consider a contour prior information to further improve its performances.
To ensure a relevant evaluation of regularity, we introduce a generalized metric measuring the spatial convexity and consistency in the 3D spherical space. While providing regular results in the acquisition space, SphSPS significantly outperforms both planar and spherical stateoftheart methods.
Accuracy and regularity in the acquisition space are crucial for relevant display and for computer vision preprocessing. Future works will extend our method to spherical videos and other acquisition spaces, e.g., circular or polarimetric.
References
 [1] (2012) SLIC superpixels compared to stateoftheart superpixel methods. IEEE Transactions on Pattern Analysis and Machine Intelligence 34, pp. 2274–2282. Cited by: §I, §I, §IIA, §II, §IVC.

[2]
(2017)
Superpixels and polygons using simple noniterative clustering.
In
IEEE Conference on Computer Vision and Pattern Recognition
, pp. 4895–4904. Cited by: §I, Fig. 8, §IVC.  [3] (1965) Algorithm for computer control of a digital plotter. IBM Systems Journal 4 (1), pp. 25–30. Cited by: §IIC2, §IIC.
 [4] (2014) Eikonalbased vertices growing and iterative seeding for efficient graphbased segmentation. In Proc. of IEEE International Conference on Image Processing, pp. 4368–4372. Cited by: §I.
 [5] (2014) Piecewise planar and compact floorplan reconstruction from images. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 628–635. Cited by: §I.

[6]
(2017)
Linear spectral clustering superpixel
. IEEE Transactions on Image Processing 26, pp. 3317–3330. Cited by: Fig. 1, §I, §I, Fig. 8, §IVA3, §IVB, §IVC, §IVC, §IVC.  [7] (2004) Efficient graphbased image segmentation. International Journal of Computer Vision 59 (2), pp. 167–181. Cited by: §I.
 [8] (2017) SuperPatchMatch: an algorithm for robust correspondences using superpixel patches. IEEE Transactions on Image Processing 26 (8), pp. 4068–4078. Cited by: §I.
 [9] (2017) Evaluation framework of superpixel methods with a global regularity measure. Journal of Electronic Imaging 26 (6). Cited by: §I, §IIIA, §IIIB1, §IIIB1, §III, §IVA2, §IVA2, §IVA2, §V.
 [10] (2018) Robust superpixels using color and contour features along linear path. Computer Vision and Image Understanding 170, pp. 1–13. Cited by: §I, §I, §I, §IIC1, §IIC2, §II, Fig. 8, §IVA3, §IVB, §IVC, §IVC.
 [11] (2014) Superpixel graph label transfer with learned distance metric. In European Conference on Computer Vision, pp. 632–647. Cited by: §I.
 [12] (1983) Filling riemannian manifolds. Journal of Differential Geometry 18 (1), pp. 1–147. Cited by: §IIC3.
 [13] (2013) Algorithms for geodesics. Journal of Geodesy 87 (1), pp. 43–55. Cited by: §IIC3.
 [14] (2009) TurboPixels: fast superpixels using geometric flows. IEEE Transactions on Pattern Analysis and Machine Intelligence 31 (12), pp. 2290–2297. Cited by: §I.
 [15] (2017) Photo stylistic brush: robust style transfer via superpixelbased bipartite graph. IEEE Trans. on Multimedia 20 (7), pp. 1724–1737. Cited by: §I.
 [16] (2011) Entropy rate superpixel segmentation. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 2097–2104. Cited by: §I, §IVA2.
 [17] (2015) Waterpixels. IEEE Transactions on Image Processing 24 (11), pp. 3707–3716. Cited by: §I.
 [18] (2015) Object scene flow for autonomous vehicles. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 3061–3070. Cited by: §I.
 [19] (2012) Superpixel benchmark and comparison. In Forum Bildverarbeitung, pp. 1–12. Cited by: §IVA2.
 [20] (1978) The isoperimetric inequality. Bulletin of the American Mathematical Society 84 (6), pp. 1182–1238. Cited by: §IIIA.
 [21] (2015) Change detection from a street image pair using cnn features and superpixel segmentation.. In British Machine Vision Conference, pp. 61–1. Cited by: §I.
 [22] (2012) Measuring and evaluating the compactness of superpixels. In International Conference on Pattern Recognition, pp. 930–934. Cited by: §IIIA, §III.
 [23] (2010) SuperParsing: scalable nonparametric image parsing with superpixels. In European Conference on Computer Vision, pp. 352–365. Cited by: §I.
 [24] (2018) Learning superpixels with segmentationaware affinity loss. In IEEE Conference on Computer Vision and Pattern Recognition, Cited by: §I, §I.
 [25] (2018) Spherical superpixels: benchmark and evaluation. In Asian Conference on Computer Vision, pp. 703–717. Cited by: §I, §IVA1.
 [26] (2013) Multiatlas segmentation without registration: a supervoxelbased approach. International Conference on Medical Image Computing and ComputerAssisted Intervention, pp. 535–542. Cited by: §I.
 [27] (1997) Sampling with hammersley and halton points. Journal of Graphics Tools 2 (2), pp. 9–24. Cited by: §IIB.
 [28] (2012) Recognizing scene viewpoint using panoramic place representation. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 2695–2702. Cited by: §IVA1.
 [29] (2015) Holisticallynested edge detection. In IEEE International Conference on Computer Vision, pp. 1395–1403. Cited by: §IVA3.
 [30] (2016) Efficient 3d room shape recovery from a single panorama. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 5422–5430. Cited by: §I.
 [31] (2015) Realtime coarsetofine topologically preserving segmentation. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 2947–2955. Cited by: §I.
 [32] (2016) A simple algorithm of superpixel segmentation with boundary constraint. IEEE Transactions on Circuits and Systems for Video Technology (99). Cited by: §I.
 [33] (2018) Spherical superpixel segmentation. IEEE Trans. on Multimedia 20 (6), pp. 1406–1417. Cited by: §I, §I, §I, §IIB, §IIB, §IIB, §IIC1, §II, §IIIA, §IIIA, §III, Fig. 7, Fig. 8, §IVA3, §IVB, §IVC, §IVC, §IVC.
 [34] (1995) Correction of geometric perceptual distortions in pictures. In International Conf. on Computer Graphics and Interactive Techniques, pp. 257–264. Cited by: §I.
Comments
There are no comments yet.