Learning to Navigate the Web

by   Izzeddin Gür, et al.

Learning in environments with large state and action spaces, and sparse rewards, can hinder a Reinforcement Learning (RL) agent's learning through trial-and-error. For instance, following natural language instructions on the Web (such as booking a flight ticket) leads to RL settings where input vocabulary and number of actionable elements on a page can grow very large. Even though recent approaches improve the success rate on relatively simple environments with the help of human demonstrations to guide the exploration, they still fail in environments where the set of possible instructions can reach millions. We approach the aforementioned problems from a different perspective and propose guided RL approaches that can generate unbounded amount of experience for an agent to learn from. Instead of learning from a complicated instruction with a large vocabulary, we decompose it into multiple sub-instructions and schedule a curriculum in which an agent is tasked with a gradually increasing subset of these relatively easier sub-instructions. In addition, when the expert demonstrations are not available, we propose a novel meta-learning framework that generates new instruction following tasks and trains the agent more effectively. We train DQN, deep reinforcement learning agent, with Q-value function approximated with a novel QWeb neural network architecture on these smaller, synthetic instructions. We evaluate the ability of our agent to generalize to new instructions on World of Bits benchmark, on forms with up to 100 elements, supporting 14 million possible instructions. The QWeb agent outperforms the baseline without using any human demonstration achieving 100


Learning from Symmetry: Meta-Reinforcement Learning with Symmetric Data and Language Instructions

Meta-reinforcement learning (meta-RL) is a promising approach that enabl...

FollowNet: Robot Navigation by Following Natural Language Directions with Deep Reinforcement Learning

Understanding and following directions provided by humans can enable rob...

Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration

Reinforcement learning (RL) agents improve through trial-and-error, but ...

Meta-Reinforcement Learning via Language Instructions

Although deep reinforcement learning has recently been very successful a...

Language-Conditioned Reinforcement Learning to Solve Misunderstandings with Action Corrections

Human-to-human conversation is not just talking and listening. It is an ...

Reinforcement Learning of Action and Query Policies with LTL Instructions under Uncertain Event Detector

Reinforcement learning (RL) with linear temporal logic (LTL) objectives ...

LTL2Action: Generalizing LTL Instructions for Multi-Task RL

We address the problem of teaching a deep reinforcement learning (RL) ag...

Please sign up or login with your details

Forgot password? Click here to reset