DeepAI AI Chat
Log In Sign Up

Visual Attention Prediction Improves Performance of Autonomous Drone Racing Agents

by   Christian Pfeiffer, et al.

Humans race drones faster than neural networks trained for end-to-end autonomous flight. This may be related to the ability of human pilots to select task-relevant visual information effectively. This work investigates whether neural networks capable of imitating human eye gaze behavior and attention can improve neural network performance for the challenging task of vision-based autonomous drone racing. We hypothesize that gaze-based attention prediction can be an efficient mechanism for visual information selection and decision making in a simulator-based drone racing task. We test this hypothesis using eye gaze and flight trajectory data from 18 human drone pilots to train a visual attention prediction model. We then use this visual attention prediction model to train an end-to-end controller for vision-based autonomous drone racing using imitation learning. We compare the drone racing performance of the attention-prediction controller to those using raw image inputs and image-based abstractions (i.e., feature tracks). Our results show that attention-prediction based controllers outperform the baselines and are able to complete a challenging race track consistently with up to 88 visual attention-prediction and feature-track based models showed better generalization performance than image-based models when evaluated on hold-out reference trajectories. Our results demonstrate that human visual attention prediction improves the performance of autonomous vision-based drone racing agents and provides an essential step towards vision-based, fast, and agile autonomous flight that eventually can reach and even exceed human performances.


page 4

page 5

page 6


Human-Piloted Drone Racing: Visual Processing and Control

Humans race drones faster than algorithms, despite being limited to a fi...

Learning Vision-based Flight in Drone Swarms by Imitation

Decentralized drone swarms deployed today either rely on sharing of posi...

Cooperative Flight Control Using Visual-Attention – Air-Guardian

The cooperation of a human pilot with an autonomous agent during flight ...

Learning Deep Sensorimotor Policies for Vision-based Autonomous Drone Racing

Autonomous drones can operate in remote and unstructured environments, e...

Learning Vision-based Cohesive Flight in Drone Swarms

This paper presents a data-driven approach to learning vision-based coll...

Gaze Training by Modulated Dropout Improves Imitation Learning

Imitation learning by behavioral cloning is a prevalent method which has...

Learning a Controller Fusion Network by Online Trajectory Filtering for Vision-based UAV Racing

Autonomous UAV racing has recently emerged as an interesting research pr...