Orientation-aware Semantic Segmentation on Icosahedron Spheres

by   Chao Zhang, et al.

We address semantic segmentation on omnidirectional images, to leverage a holistic understanding of the surrounding scene for applications like autonomous driving systems. For the spherical domain, several methods recently adopt an icosahedron mesh, but systems are typically rotation invariant or require significant memory and parameters, thus enabling execution only at very low resolutions. In our work, we propose an orientation-aware CNN framework for the icosahedron mesh. Our representation allows for fast network operations, as our design simplifies to standard network operations of classical CNNs, but under consideration of north-aligned kernel convolutions for features on the sphere. We implement our representation and demonstrate its memory efficiency up-to a level-8 resolution mesh (equivalent to 640 x 1024 equirectangular images). Finally, since our kernels operate on the tangent of the sphere, standard feature weights, pretrained on perspective data, can be directly transferred with only small need for weight refinement. In our evaluation our orientation-aware CNN becomes a new state of the art for the recent 2D3DS dataset, and our Omni-SYNTHIA version of SYNTHIA. Rotation invariant classification and segmentation tasks are additionally presented for comparison to prior art.



There are no comments yet.


page 1

page 3

page 4

page 6

page 7

page 9


Land cover mapping at very high resolution with rotation equivariant CNNs: towards small yet accurate models

In remote sensing images, the absolute orientation of objects is arbitra...

DeepSphere: Efficient spherical Convolutional Neural Network with HEALPix sampling for cosmological applications

Convolutional Neural Networks (CNNs) are a cornerstone of the Deep Learn...

Labeling Panoramas with Spherical Hourglass Networks

With the recent proliferation of consumer-grade 360 cameras, it is worth...

PRIN: Pointwise Rotation-Invariant Network

In recent years, point clouds have earned quite some research interests ...

Gauge Equivariant Mesh CNNs: Anisotropic convolutions on geometric graphs

A common approach to define convolutions on meshes is to interpret them ...

DualConvMesh-Net: Joint Geodesic and Euclidean Convolutions on 3D Meshes

We propose DualConvMesh-Nets (DCM-Net) a family of deep hierarchical con...

Single Level Feature-to-Feature Forecasting with Deformable Convolutions

Future anticipation is of vital importance in autonomous driving and oth...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.