DiffStack: A Differentiable and Modular Control Stack for Autonomous Vehicles

by   Peter Karkus, et al.

Autonomous vehicle (AV) stacks are typically built in a modular fashion, with explicit components performing detection, tracking, prediction, planning, control, etc. While modularity improves reusability, interpretability, and generalizability, it also suffers from compounding errors, information bottlenecks, and integration challenges. To overcome these challenges, a prominent approach is to convert the AV stack into an end-to-end neural network and train it with data. While such approaches have achieved impressive results, they typically lack interpretability and reusability, and they eschew principled analytical components, such as planning and control, in favor of deep neural networks. To enable the joint optimization of AV stacks while retaining modularity, we present DiffStack, a differentiable and modular stack for prediction, planning, and control. Crucially, our model-based planning and control algorithms leverage recent advancements in differentiable optimization to produce gradients, enabling optimization of upstream components, such as prediction, via backpropagation through planning and control. Our results on the nuScenes dataset indicate that end-to-end training with DiffStack yields substantial improvements in open-loop and closed-loop planning metrics by, e.g., learning to make fewer prediction errors that would affect planning. Beyond these immediate benefits, DiffStack opens up new opportunities for fully data-driven yet modular and interpretable AV architectures. Project website: https://sites.google.com/view/diffstack


Differentiable Trajectory Generation for Car-like Robots with Interpolating Radial Basis Function Networks

The design of Autonomous Vehicle software has largely followed the Sense...

End-to-end Autonomous Driving: Challenges and Frontiers

The autonomous driving community has witnessed a rapid growth in approac...

Control-Aware Prediction Objectives for Autonomous Driving

Autonomous vehicle software is typically structured as a modular pipelin...

Designing a Recurrent Neural Network to Learn a Motion Planner for High-Dimensional Inputs

The use of machine learning in the self-driving industry has boosted a n...

A Survey of Deep Reinforcement Learning Algorithms for Motion Planning and Control of Autonomous Vehicles

In this survey, we systematically summarize the current literature on st...

NeuralSim: Augmenting Differentiable Simulators with Neural Networks

Differentiable simulators provide an avenue for closing the sim-to-real ...

Multimodal Sensor Fusion with Differentiable Filters

Leveraging multimodal information with recursive Bayesian filters improv...

Please sign up or login with your details

Forgot password? Click here to reset