Multimodal dynamics modeling for off-road autonomous vehicles

by   Jean-François Tremblay, et al.

Dynamics modeling in outdoor and unstructured environments is difficult because different elements in the environment interact with the robot in ways that can be hard to predict. Leveraging multiple sensors to perceive maximal information about the robot's environment is thus crucial when building a model to perform predictions about the robot's dynamics with the goal of doing motion planning. We design a model capable of long-horizon motion predictions, leveraging vision, lidar and proprioception, which is robust to arbitrarily missing modalities at test time. We demonstrate in simulation that our model is able to leverage vision to predict traction changes. We then test our model using a real-world challenging dataset of a robot navigating through a forest, performing predictions in trajectories unseen during training. We try different modality combinations at test time and show that, while our model performs best when all modalities are present, it is still able to perform better than the baseline even when receiving only raw vision input and no proprioception, as well as when only receiving proprioception. Overall, our study demonstrates the importance of leveraging multiple sensors when doing dynamics modeling in outdoor conditions.



There are no comments yet.


page 4

page 5

page 6


Reactive motion planning with probabilistics safety guarantees

Motion planning in environments with multiple agents is critical to many...

Multimodal representation models for prediction and control from partial information

Similar to humans, robots benefit from interacting with their environmen...

Choosing Smartly: Adaptive Multimodal Fusion for Object Detection in Changing Environments

Object detection is an essential task for autonomous robots operating in...

ObserveNet Control: A Vision-Dynamics Learning Approach to Predictive Control in Autonomous Vehicles

A key component in autonomous driving is the ability of the self-driving...

A Framework for Multisensory Foresight for Embodied Agents

Predicting future sensory states is crucial for learning agents such as ...

Learning to Navigate Sidewalks in Outdoor Environments

Outdoor navigation on sidewalks in urban environments is the key technol...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.