The Foundation Model Path to Open-World Robots

Dhruv Shah

Robot learning methods typically rely either on learning from large-scale simulation modeling and transferring to real-world settings or by collecting real-world interaction data on the target robot. While this paradigm has been successful for solving simple tasks in structured environments, it may fall short for tasks that are hard to simulate accurately (e.g. in off-road racing) and where data collection may be expensive (e.g. micro UAVs with <5 minute battery life). My research proposes an alternative paradigm of “cross-embodiment” robot learning, building algorithms and systems that can leverage internet-scale data to learn intelligent behaviors in unstructured, open-world environments. In this talk, I will discuss the unique challenges and opportunities that motivate building “robot foundation models”, and present the first instantiation of such a model for the task of visual navigation. I will then discuss how such a model can serve as a pre-trained backbone for a variety of downstream applications, such as autonomous off-road racing and socially-compliant navigation, as well as bootstrap learning for entirely new robots such as drones, quadrupeds, and manipulators. Finally, I will discuss how these robot foundation models can be empowered with current vision and language foundation models using a  novel planning framework to build robust robotic systems capable of “in-the-wild” deployment of intelligent robotic systems.

Dhruv Shah is a final year PhD candidate in EECS at UC Berkeley, where he is advised by Sergey Levine. His research spans the fields of machine learning and robotics, with the goal of building autonomous robots that can combine large-scale learning with real-world deployment. Dhruv is a Microsoft Future Leader in Robotics & AI (2024), Berkeley Fellow, and his research was a finalist for the Best Systems Paper Award at RSS 2022. His work has also been featured in several media outlets, including IEEE Spectrum, TechXplore, Two Minute Papers, and ZDNet, along with several international venues.

https://columbiauniversity.zoom.us/j/97996448552?pwd=c09Lem45amxmWXRLMFo5QjFPb0hNUT09