MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge

06/17/2022
by   Linxi Fan, et al.
19

Autonomous agents have made great strides in specialist domains like Atari games and Go. However, they typically learn tabula rasa in isolated environments with limited and manually conceived objectives, thus failing to generalize across a wide spectrum of tasks and capabilities. Inspired by how humans continually learn and adapt in the open world, we advocate a trinity of ingredients for building generalist agents: 1) an environment that supports a multitude of tasks and goals, 2) a large-scale database of multimodal knowledge, and 3) a flexible and scalable agent architecture. We introduce MineDojo, a new framework built on the popular Minecraft game that features a simulation suite with thousands of diverse open-ended tasks and an internet-scale knowledge base with Minecraft videos, tutorials, wiki pages, and forum discussions. Using MineDojo's data, we propose a novel agent learning algorithm that leverages large pre-trained video-language models as a learned reward function. Our agent is able to solve a variety of open-ended tasks specified in free-form language without any manually designed dense shaping reward. We open-source the simulation suite and knowledge bases (https://minedojo.org) to promote research towards the goal of generally capable embodied agents.

READ FULL TEXT

page 2

page 4

page 6

page 9

page 25

page 26

page 27

research
07/12/2020

OtoWorld: Towards Learning to Separate by Learning to Move

We present OtoWorld, an interactive environment in which agents must lea...
research
08/13/2018

Large-Scale Study of Curiosity-Driven Learning

Reinforcement learning algorithms rely on carefully engineering environm...
research
05/25/2023

Ghost in the Minecraft: Generally Capable Agents for Open-World Enviroments via Large Language Models with Text-based Knowledge and Memory

The captivating realm of Minecraft has attracted substantial research in...
research
11/25/2020

Open-World Learning Without Labels

Open-world learning is a problem where an autonomous agent detects thing...
research
09/14/2021

Benchmarking the Spectrum of Agent Capabilities

Evaluating the general abilities of intelligent agents requires complex ...
research
07/28/2021

Growing knowledge culturally across generations to solve novel, complex tasks

Knowledge built culturally across generations allows humans to learn far...
research
10/25/2021

What Would Jiminy Cricket Do? Towards Agents That Behave Morally

When making everyday decisions, people are guided by their conscience, a...

Please sign up or login with your details

Forgot password? Click here to reset