Exploiting Natural Language for Efficient Risk-Aware Multi-robot SaR Planning

by   Vikram Shree, et al.

The ability to develop a high-level understanding of a scene, such as perceiving danger levels, can prove valuable in planning multi-robot search and rescue (SaR) missions. In this work, we propose to uniquely leverage natural language descriptions from the mission commander in chief and image data captured by robots to estimate scene danger. Given a description and an image, a state-of-the-art deep neural network is used to assess a corresponding similarity score, which is then converted into a probabilistic distribution of danger levels. Because commonly used visio-linguistic datasets do not represent SaR missions well, we collect a large-scale image-description dataset from synthetic images taken from realistic disaster scenes and use it to train our machine learning model. A risk-aware variant of the Multi-robot Efficient Search Path Planning (MESPP) problem is then formulated to use the danger estimates in order to account for high-risk locations in the environment when planning the searchers' paths. The problem is solved via a distributed approach based on Mixed-Integer Linear Programming. Our experiments demonstrate that our framework allows to plan safer yet highly successful search missions, abiding to the two most important aspects of SaR missions: to ensure both searchers' and victim safety.



There are no comments yet.


page 1

page 3

page 8


Mixed-Integer Linear Programming Models for Multi-Robot Non-Adversarial Search

In this letter, we consider the Multi-Robot Efficient Search Path Planni...

Balance Scene Learning Mechanism for Offshore and Inshore Ship Detection in SAR Images

This letter proposes a novel Balance Scene Learning Mechanism (BSLM) for...

Exploiting multi-temporal information for improved speckle reduction of Sentinel-1 SAR images by deep learning

Deep learning approaches show unprecedented results for speckle reductio...

Risk-Aware Planning and Assignment for Ground Vehicles using Uncertain Perception from Aerial Vehicles

We propose a risk-aware framework for multi-robot, multi-demand assignme...

Collaborative Multi-Robot Systems for Search and Rescue: Coordination and Perception

Autonomous or teleoperated robots have been playing increasingly importa...

Counterexamples for Robotic Planning Explained in Structured Language

Automated techniques such as model checking have been used to verify mod...

An Autonomous Drone for Search and Rescue in Forests using Airborne Optical Sectioning

Drones will play an essential role in human-machine teaming in future se...

Code Repositories


Image-description dataset for multimodal scene perception.

view repo
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.