ProgPrompt: Generating Situated Robot Task Plans using Large Language Models

by   Ishika Singh, et al.
University of Southern California

Task planning can require defining myriad domain knowledge about the world in which a robot needs to act. To ameliorate that effort, large language models (LLMs) can be used to score potential next actions during task planning, and even generate action sequences directly, given an instruction in natural language with no additional domain information. However, such methods either require enumerating all possible next steps for scoring, or generate free-form text that may contain actions not possible on a given robot in its current context. We present a programmatic LLM prompt structure that enables plan generation functional across situated environments, robot capabilities, and tasks. Our key insight is to prompt the LLM with program-like specifications of the available actions and objects in an environment, as well as with example programs that can be executed. We make concrete recommendations about prompt structure and generation constraints through ablation experiments, demonstrate state of the art success rates in VirtualHome household tasks, and deploy our method on a physical robot arm for tabletop tasks. Website at


page 1

page 2

page 6


Generating Executable Action Plans with Environmentally-Aware Language Models

Large Language Models (LLMs) trained using massive text datasets have re...

Semantic Task Planning for Service Robots in Open World

In this paper, we present a planning system based on semantic reasoning ...

Errors are Useful Prompts: Instruction Guided Task Programming with Verifier-Assisted Iterative Prompting

Generating low-level robot task plans from high-level natural language i...

Correcting Robot Plans with Natural Language Feedback

When humans design cost or goal specifications for robots, they often pr...

Robot Behavior-Tree-Based Task Generation with Large Language Models

Nowadays, the behavior tree is gaining popularity as a representation fo...

Generalized Planning in PDDL Domains with Pretrained Large Language Models

Recent work has considered whether large language models (LLMs) can func...

Interactively Robot Action Planning with Uncertainty Analysis and Active Questioning by Large Language Model

The application of the Large Language Model (LLM) to robot action planni...

Please sign up or login with your details

Forgot password? Click here to reset