Visual Navigation for Autonomous Vehicles: An Open-source Hands-on Robotics Course at MIT

by   Luca Carlone, et al.

This paper reports on the development, execution, and open-sourcing of a new robotics course at MIT. The course is a modern take on "Visual Navigation for Autonomous Vehicles" (VNAV) and targets first-year graduate students and senior undergraduates with prior exposure to robotics. VNAV has the goal of preparing the students to perform research in robotics and vision-based navigation, with emphasis on drones and self-driving cars. The course spans the entire autonomous navigation pipeline; as such, it covers a broad set of topics, including geometric control and trajectory optimization, 2D and 3D computer vision, visual and visual-inertial odometry, place recognition, simultaneous localization and mapping, and geometric deep learning for perception. VNAV has three key features. First, it bridges traditional computer vision and robotics courses by exposing the challenges that are specific to embodied intelligence, e.g., limited computation and need for just-in-time and robust perception to close the loop over control and decision making. Second, it strikes a balance between depth and breadth by combining rigorous technical notes (including topics that are less explored in typical robotics courses, e.g., on-manifold optimization) with slides and videos showcasing the latest research results. Third, it provides a compelling approach to hands-on robotics education by leveraging a physical drone platform (mostly suitable for small residential courses) and a photo-realistic Unity-based simulator (open-source and scalable to large online courses). VNAV has been offered at MIT in the Falls of 2018-2021 and is now publicly available on MIT OpenCourseWare (OCW).


page 1

page 3


A Simulation Benchmark for Vision-based Autonomous Navigation

This work introduces a simulator benchmark for vision-based autonomous n...

Steps Towards a Theory of Visual Information: Active Perception, Signal-to-Symbol Conversion and the Interplay Between Sensing and Control

This manuscript describes the elements of a theory of information tailor...

A Review of Visual Odometry Methods and Its Applications for Autonomous Driving

The research into autonomous driving applications has observed an increa...

VPAIR – Aerial Visual Place Recognition and Localization in Large-scale Outdoor Environments

Visual Place Recognition and Visual Localization are essential component...

Learning Synthetic to Real Transfer for Localization and Navigational Tasks

Autonomous navigation consists in an agent being able to navigate withou...

HOPPY: An open-source and low-cost kit for dynamic robotics education

This letter introduces HOPPY, an open-source, low-cost, robust, and modu...

Semantics for Robotic Mapping, Perception and Interaction: A Survey

For robots to navigate and interact more richly with the world around th...

Please sign up or login with your details

Forgot password? Click here to reset