Visual Attention Prediction Improves Performance of Autonomous Drone Racing Agents

by   Christian Pfeiffer, et al.

Humans race drones faster than neural networks trained for end-to-end autonomous flight. This may be related to the ability of human pilots to select task-relevant visual information effectively. This work investigates whether neural networks capable of imitating human eye gaze behavior and attention can improve neural network performance for the challenging task of vision-based autonomous drone racing. We hypothesize that gaze-based attention prediction can be an efficient mechanism for visual information selection and decision making in a simulator-based drone racing task. We test this hypothesis using eye gaze and flight trajectory data from 18 human drone pilots to train a visual attention prediction model. We then use this visual attention prediction model to train an end-to-end controller for vision-based autonomous drone racing using imitation learning. We compare the drone racing performance of the attention-prediction controller to those using raw image inputs and image-based abstractions (i.e., feature tracks). Our results show that attention-prediction based controllers outperform the baselines and are able to complete a challenging race track consistently with up to 88 visual attention-prediction and feature-track based models showed better generalization performance than image-based models when evaluated on hold-out reference trajectories. Our results demonstrate that human visual attention prediction improves the performance of autonomous vision-based drone racing agents and provides an essential step towards vision-based, fast, and agile autonomous flight that eventually can reach and even exceed human performances.


page 4

page 5

page 6


Human-Piloted Drone Racing: Visual Processing and Control

Humans race drones faster than algorithms, despite being limited to a fi...

Learning Agile, Vision-based Drone Flight: from Simulation to Reality

We present our latest research in learning deep sensorimotor policies fo...

Learning Vision-based Flight in Drone Swarms by Imitation

Decentralized drone swarms deployed today either rely on sharing of posi...

Learning Vision-based Cohesive Flight in Drone Swarms

This paper presents a data-driven approach to learning vision-based coll...

Cooperative Flight Control Using Visual-Attention – Air-Guardian

The cooperation of a human pilot with an autonomous agent during flight ...

Enhancing Robot Learning through Learned Human-Attention Feature Maps

Robust and efficient learning remains a challenging problem in robotics,...

Gaze Training by Modulated Dropout Improves Imitation Learning

Imitation learning by behavioral cloning is a prevalent method which has...

Please sign up or login with your details

Forgot password? Click here to reset