FISHING Net: Future Inference of Semantic Heatmaps In Grids

06/17/2020
by   Noureldin Hendy, et al.
5

For autonomous robots to navigate a complex environment, it is crucial to understand the surrounding scene both geometrically and semantically. Modern autonomous robots employ multiple sets of sensors, including lidars, radars, and cameras. Managing the different reference frames and characteristics of the sensors, and merging their observations into a single representation complicates perception. Choosing a single unified representation for all sensors simplifies the task of perception and fusion. In this work, we present an end-to-end pipeline that performs semantic segmentation and short term prediction using a top-down representation. Our approach consists of an ensemble of neural networks which take in sensor data from different sensor modalities and transform them into a single common top-down semantic grid representation. We find this representation favorable as it is agnostic to sensor-specific reference frames and captures both the semantic and geometric information for the surrounding scene. Because the modalities share a single output representation, they can be easily aggregated to produce a fused output. In this work we predict short-term semantic grids but the framework can be extended to other tasks. This approach offers a simple, extensible, end-to-end approach for multi-modal perception and prediction.

READ FULL TEXT

page 1

page 3

page 4

page 8

research
03/21/2019

Short-Term Prediction and Multi-Camera Fusion on Semantic Grids

An environment representation (ER) is a substantial part of every autono...
research
08/15/2023

UniTR: A Unified and Efficient Multi-Modal Transformer for Bird's-Eye-View Representation

Jointly processing information from multiple sensors is crucial to achie...
research
02/21/2019

Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges

Recent advancements in the perception for autonomous driving are driven ...
research
05/03/2022

3D Semantic Scene Perception using Distributed Smart Edge Sensors

We present a system for 3D semantic scene perception consisting of a net...
research
07/04/2012

Map-aided Fusion Using Evidential Grids for Mobile Perception in Urban Environment

Evidential grids have been recently used for mobile object perception. T...
research
05/22/2018

Towards Inverse Sensor Mapping in Agriculture

In recent years, the drive of the Industry 4.0 initiative has enriched i...
research
04/24/2023

USA-Net: Unified Semantic and Affordance Representations for Robot Memory

In order for robots to follow open-ended instructions like "go open the ...

Please sign up or login with your details

Forgot password? Click here to reset