VECA : A Toolkit for Building Virtual Environments to Train and Test Human-like Agents

by   Kwanyoung Park, et al.

Building human-like agent, which aims to learn and think like human intelligence, has long been an important research topic in AI. To train and test human-like agents, we need an environment that imposes the agent to rich multimodal perception and allows comprehensive interactions for the agent, while also easily extensible to develop custom tasks. However, existing approaches do not support comprehensive interaction with the environment or lack variety in modalities. Also, most of the approaches are difficult or even impossible to implement custom tasks. In this paper, we propose a novel VR-based toolkit, VECA, which enables building fruitful virtual environments to train and test human-like agents. In particular, VECA provides a humanoid agent and an environment manager, enabling the agent to receive rich human-like perception and perform comprehensive interactions. To motivate VECA, we also provide 24 interactive tasks, which represent (but are not limited to) four essential aspects in early human development: joint-level locomotion and control, understanding contexts of objects, multimodal learning, and multi-agent learning. To show the usefulness of VECA on training and testing human-like learning agents, we conduct experiments on VECA and show that users can build challenging tasks for engaging human-like algorithms, and the features supported by VECA are critical on training human-like agents.


page 3

page 6

page 11

page 13

page 14

page 15


Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration

In this paper, we introduce Watch-And-Help (WAH), a challenge for testin...

Arena: A General Evaluation Platform and Building Toolkit for Multi-Agent Intelligence

Learning agents that are not only capable of taking tests but also innov...

VRKitchen: an Interactive 3D Virtual Environment for Task-oriented Learning

One of the main challenges of advancing task-oriented learning such as v...

Toddler-Guidance Learning: Impacts of Critical Period on Multimodal AI Agents

Critical periods are phases during which a toddler's brain develops in s...

Madeup: A Mobile Development Environment for Programming 3-D Models

Constructionism is a learning theory that states that we learn more when...

ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation

We introduce ThreeDWorld (TDW), a platform for interactive multi-modal p...

AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation Framework

This technical report presents AutoGen, a new framework that enables dev...

Please sign up or login with your details

Forgot password? Click here to reset