URLB: Unsupervised Reinforcement Learning Benchmark

1 Introduction

Deep Reinforcement Learning (RL) has been at the source of a number of breakthroughs in autonomous control over the last five years. RL algorithms have been used to train agents to play Atari video games directly from pixels mnih2015human; mnih2016a2c, learn robotic locomotion trpo; schulman2015high; ppo and manipulation akkaya2019solving policies from raw sensory input, master the game of Go alphago; alphazero, and play large-scale multiplayer video games berner2019dota; alphastar. While these results were significant advances in autonomous decision making, a deeper look reveals a fundamental limitation. The above algorithms produced agents capable of only solving the single task they were trained to solve. As a result, current RL approaches produce brittle policies with poor generalization capabilities cobbe2020leveraging, which limits their applicability to many problems of interest Gleave2020Adversarial. It is therefore important to move beyond today’s powerful but narrow RL systems toward generalist systems capable of quickly adapting to new downstream tasks.

In contrast, in the fields of Computer Vision (CV) and Natural Language Processing (NLP), large-scale unsupervised pre-training has enabled sample-efficient few-shot adaptation. In NLP, unsupervised sequential modeling has produced powerful few-shot learners

brown2020language; devlin2018bert; radford2019language. In CV, unsupervised representation learning techniques such as contrastive learning have produced algorithms that are dramatically more label-efficient than their supervised counterparts chen2020simclr; he2020moco; henaff2019cpcv2; grill2020byol

and more capable of adapting to a host of downstream supervised tasks such as classification, segmentation, and object detection. While these advances in unsupervised learning have also benefited RL in terms of learning efficiently from images

laskin2020reinforcement; laskin2020curl; schwarzer2021dataefficient; stooke2020decoupling; yarats2021image as well as introducing new architectures for RL chen_lu2021dt; janner2021tto, the resulting agents have remained narrow since they still optimize a single extrinsic reward as before.

Figure 1: Unlike supervised RL which requires reward interaction at every step, unsupervised RL has two phases: (i) reward-free pre-training and (ii) fine-tuning to an extrinsic reward. During phase (i) an agent explores the environment through reward-free interaction with the environment. The quality of exploration depends on the intrinsic reward that the agent sets for itself. During phase (ii) the quality of pre-training is evaluated by its adaptation efficiency to a downstream task.

Fully unsupervised training of RL algorithms requires not only learning self-supervised representations but also learning policies without access to extrinsic rewards. Recently, unsupervised RL algorithms have begun to show progress toward more generalist systems by training policies without extrinsic rewards. Exploration with self-supervised prediction has enabled agents to explore video games from pixels pathak2017curiosity; Pathak19disagreement, mutual information-based approaches have demonstrated self-supervised skill discovery and generalization to downstream tasks in continuous control domains EysenbachGIL19diayn; hansen20visr; liu21aps; SharmaGLKH20dads, and maximal entropy RL has yielded policies capable of diverse exploration liu2021unsupervised; seo21re3; yarats21protorl. However, comparing and developing new algorithms has been challenging due to a lack of a unified evaluation benchmark. Reward-free RL algorithms often use different optimization schemes, different tasks for evaluation, and have different evaluation procedures. Additionally, unlike more mature supervised RL algorithms sac; hessel18rainbow; ppo, there does not exist a unified codebase for unsupervised RL that can be used to develop new methods quickly.

To make benchmarking and developing new unsupervised RL approaches easier, we introduce the Unsupervised Reinforcement Learning Benchmark (URLB). Built on top of the widely adopted DeepMind Control Suite tassa2018deepmind, URLB provides a suite of domains of varying difficulty for unsupervised pre-training with diverse downstream evaluation tasks. URLB standardizes evaluation of unsupervised RL algorithms by defining fixed pre-training and fine-tuning procedures across all baselines. Perhaps most importantly, we open-source code for URLB environments as well as 8 leading baselines that represent the main approaches taken towards unsupervised pre-training in RL to date. Unlike prior code releases for unsupervised RL, URLB uses the same exact optimization algorithm for each baseline which enables transparent benchmarking and lowers the barrier to entry for developing new algorithms. We summarize the main contributions of this paper below:

We introduce URLB, a new benchmark for evaluating unsupervised RL algorithms, which consists of three domains and twelve continuous control tasks of varying difficulty to evaluate the adaptation efficiency of unsupervised RL algorithms.
We open-source a unified codebase for eight leading unsupervised RL algorithms. Each algorithm is trained with the same optimization backbone for fairness of comparison.
We find that while the implemented baselines make progress on the proposed benchmark, no existing unsupervised RL algorithm can solve URLB, and consequently identify promising research directions to progress unsupervised RL.

The benchmark environments, algorithmic baselines, and pre-training and evaluation scripts are available at https://github.com/rll-research/url_benchmark. We believe that URLB will make the development of unsupervised RL agents easier and more transparent by providing a unified set of evaluation environments, systematic procedures for pre-training and evaluation, and algorithmic baselines that share the same optimization backbone.

2 Preliminaries and Notation

Markov Decision Process:

We consider the typical Reinforcement Learning setting where an agent’s interaction with the environment is modeled through a Markov Decision Process (MDP)

sutton2018reinforcement. In this work, we benchmark unsupervised RL algorithms in both fully observable MDPs where the agent learns from coordinate state as well as partially observable MDPs (POMDPs) where the agent learns from partially observable image observations. For simplicity we refer to both image and state-based observations as

o

. At every timestep

t

, the agent sees an observation

o_{t}

and selects an action

a_{t}

based on its policy

a_{t} \sim π_{θ} (\cdot | o_{t})

. The agent then sees the next observation

o_{t + 1}

and an extrinsic reward

r_{t}^{ext}

provided by the environment (supervised RL) or an intrinsic reward

r_{t}^{int}

defined through a self-supervised objective (unsupervised RL). In this work, we pre-train agents with intrinsic rewards

r_{t}^{ext}

and fine-tune them to downstream tasks with extrinsic rewards

r_{t}^{ext}

. Some algorithms considered in this work condition the agent on a learned task vector which we denote as

w

Learning from pixels vs states: We benchmark unsupervised RL where environment observations $o_{t}$ can be either proprioceptive states or RGB images. When learning from pixels, rather than defining the self-supervised task directly as a function of image observations, it is usually more convenient to first embed the image and compute the intrinsic reward as a function of these lower dimensional features burda2018exploration; liu2021unsupervised; liu21aps; pathak2017curiosity. We therefore define an embedding as $z_{t} = f_{ξ} (o_{t})$ where $f_{ξ} (o_{t})$ is an encoder function. We employ different encoder $f_{ξ}$ architectures depending on whether the algorithm receives pixel or state-based input. For pixel-based inputs we use the convolutional encoder architecture from SAC-AE (yarats2019improving), while for state-based inputs we use the identity function by default unless the unsupervised RL algorithm explicitly specifies a different encoding. The intrinsic reward $r_{t}^{int}$ can be a function of any and all $(z_{t}, a_{t}, w_{t})$ depending on the algorithm. Finally, note that the encoder $f_{ξ}$ may or may not be shared with components of the base RL algorithm such as the actor and critic.

3 URLB: Evaluation and Environments

Figure 2: The three domains (walker, quadruped, jaco arm) and twelve downstream tasks considered in URLB. The environments include tasks of varying complexity and require an agent pre-trained on a given domain to adapt efficiently to the downstream tasks within that domain.

3.1 Standardized of Pre-training and Fine-tuning Procedures

One reason why unsupervised RL has been hard to benchmark to date is that there is no agreed upon procedure for training and evaluating unsupervised RL agents. To this end, we standardize pre-training, fine-tuning, and evaluation in URLB. We split pre-training and fine-tuning into two phases consisting of $N_{P T}$ and $N_{F T}$ environment steps respectively. During pre-training, we checkpoint agents at 100k, 500k, 1M, 2M steps in order to evaluate downstream performance as a function of pre-training steps. For adapting the pre-trained policy to downstream tasks, we evaluate in the data-efficient regime where $N_{F T}$ is 100k, since we are interested in agents that are quick to adapt.

3.2 Evaluation

We evaluate the performance of an unsupervised RL algorithm by measuring how quickly it adapts to a downstream task. For each fine-tuning task, we initialize the agent with the pre-trained network parameters, fine-tune the agent for 100k steps and measure its performance on the downstream task. This evaluation procedure is similar to how pre-trained networks in CV and NLP are fine-tuned to downstream tasks such as classification, object detection, and summarization. There exist other means of evaluating the quality of pre-trained RL agents such as measuring the diversity of data collected during exploration or zero-shot generalization of goal-conditioned agents. However, it is challenging to produce a general method to measure data diversity, and while zero-shot generalization with goal-conditioned agents can be powerful such a benchmark would be limited to goal-conditioned RL. For these reasons, data diversity and goal-conditioned zero-shot generalization are less common evaluation metrics. In an effort to provide a general benchmark, we focus on the fine-tuning efficiency of the agent after pre-training which allows us to evaluate a diverse set of baselines.

Unlike unsupervised methods in CV and NLP which focus solely on representation learning, unsupervised pre-training in RL requires both representation learning and behavior learning. For this reason, URLB benchmarks performance for both state-based and pixel-based agents. Benchmarking both state and pixel-based RL separately is important because it allows us to decouple unsupervised behavior learning from unsupervised representation learning. In state-based RL, the agent receives a near-optimal representation of the world through coordinate states. Evaluating state-based unsupervised RL agents allows us to isolate unsupervised behavior discovery without worrying about representation learning as confounding factor. Evaluating pixel-based unsupervised RL agents provides insight into how representations and behaviors can be learned jointly.

3.3 URLB Environments

We release a set of domains and downstream tasks for URLB that are based on the DeepMind Control Suite (DMC) tassa2018deepmind. The three reasons for building URLB on top of DMC are (i) DMC is already widely adopted and familiar to RL practitioners; (ii) DMC environments can be used with both state and pixel-based inputs; (iii) DMC features environments of varying difficulty which is useful for designing a benchmark that contains both challenging and feasible tasks. URLB evaluates performance on 12 continuous control tasks (3 domains with 4 downstream tasks per domain). From easiest to hardest, the URLB domains and tasks are:

Walker (Stand, Walk, Flip, Run): A biped constrained to a 2D vertical plane. Walker is a challenging introduction domain for unsupervised RL because it requires the unsupervised agent to learn balancing and locomotion skills in order to fine-tune efficiently. Quadruped (Stand, Walk, Jump, Run): A quadruped within a a 3D space. Like walker, quadruped requires the agent to learn to balance and move but is harder due to a high-dimensional state and action spaces and 3D environment. Jaco Arm (Reach top left, Reach top right, Reach bottom left, Reach bottom right): Jaco Arm is a 6-DOF robotic arm with a three-finger gripper. This environment tests the unsupervised RL agent’s ability to control the robot arm without locking and perform simple manipulation tasks. It was recently shown that this environment is particularly challenging for unsupervised RL yarats21protorl.

1:Randomly initialized actor

π_{θ}

, critic

Q_{ϕ}

, and encoder

f_{ξ}

networks, replay buffer

D

2:Intrinsic

r^{i n t}

and extrinsic

r^{e x t}

reward functions, discount factor

γ

3:Environment (env),

M

downstream tasks

T_{k}, k \in [1, \dots, M]

4:pre-train

N_{P T}

and fine-tune

N_{F T}

steps.

5:for

t = 1.. N_{P T}

▹

Part 1: Unsupervised Pre-training

a_{t} \leftarrow π_{θ} (f_{ξ} (o_{t})) + ϵ

and

ϵ \sim N (0, σ^{2})

o_{t + 1} \sim P (\cdot | o_{t}, a_{t})

D \leftarrow D \cup (o_{t}, a_{t}, o_{t + 1})

9: Update

π_{θ}

Q_{ϕ}

, and

f_{ξ}

using minibatches from

D

and intrinsic reward

r^{i n t}

according to Eqs. 1 and 2.

10:end for

11:Outputs pre-trained parameters

θ_{P T}, ϕ_{P T}

, and

ξ_{P T}

12:for

T_{k} \in [T_{1}, \dots, T_{M}

] do

▹

Part 2: Supervised Fine-tuning

13: initialize

θ \leftarrow θ_{P T}

ϕ \leftarrow ϕ_{P T}

ξ \leftarrow ξ_{P T}

, reset

D

14: for

t = 1.. N_{F T}

15:

a_{t} \leftarrow π_{θ} (f_{ξ} (o_{t})) + ϵ

and

ϵ \sim N (0, σ^{2})

16:

o_{t + 1}, r_{t}^{e x t} \sim P (\cdot | o_{t}, a_{t})

17:

D \leftarrow D \cup (o_{t}, a_{t}, r_{t}^{e x t}, o_{t + 1})

18: Update

π_{θ}

Q_{ϕ}

, and

f_{ξ}

using minibatches from

D

according to Eqs. 1 and 2.

19: end for

20: Evaluate performance of RL agent on task

T_{k}

21:end for

Algorithm 1 Unsupervised RL: Unsupervised Pre-training and Supervised Fine-tuning

4 URLB: Algorithmic Baselines for Unsupervised RL

In addition to introducing URLB, the other primary contribution of this work is open-sourcing a unified codebase for eight leading unsupervised RL algorithms. To date, unsupervised RL algorithms have been hard to compare due to confounding factors such as different evaluation procedures and optimization schemes. While URLB provides standardized pre-training, fine-tuning, and evaluation procedures, current algorithms are hard to compare since they rely on different optimization algorithms. For instance, Curiosity pathak2017curiosity utilizes PPO ppo while APT liu2021unsupervised uses SAC sac for optimization. Moreover, even if two unsupervised RL methods use the same optimization algorithm, small differences in implementation can result in large performance differences that are independent of the pre-training algorithm. For this reason, it is important to provide a unified codebase with identical implementations of the optimization algorithm for each baseline. Providing such a unified codebase is one of the main contributions of this benchmark.

4.1 Backbone RL Algorithm

Since most of the above algorithms rely on off-policy optimization (and some cannot be optimized on-policy at all), we opt for a state-of-the-art off-policy optimization algorithm. While SAC sac has been the de facto off-policy RL algorithm for many RL methods in the last few years, it is prone to suffering from policy entropy collapse. DrQ-v2 (yarats2021drqv2) recently showed that using DDPG lillicrap15ddpg instead of SAC as a learning algorithm leads to a more robust performance on tasks from DMC. For this reason, we opt for DrQ-v2 (yarats2021drqv2) as our base optimization algorithm to learn from images, and DDPG, as implemented in DrQ-v2, to learn from states. DDPG is an actor-critic off-policy algorithm for continuous control tasks. The critic $Q_{ϕ}$ minimizes the Bellman error

L_{Q} (ϕ, D) = E_{(o_{t}, a_{t}, r_{t}, o_{t + 1}) \sim D} [(Q_{ϕ} (o_{t}, a_{t}) - r_{t} - γ Q_{¯ ϕ} (o_{t + 1}, π_{θ} (o_{t + 1}))^{2}],

(1)

where $¯ ϕ$ is an exponential moving average of the critic weights. The deterministic actor $π_{θ}$ is learned by maximizing the expected returns

L_{π} (θ, D) = E_{o_{t} \sim D} [Q_{ϕ} (o_{t}, π_{θ} (o_{t}))] .

(2)

4.2 Unsupervised RL Algorithms

As part of URLB, we open-source code for eight leading or well-known algorithms across all three of these categories all of which utilize the same optimization backbone. All algorithms provided with URLB differ only in their intrinsic reward while keeping all other parts of the RL architecture the same. We list all implemented baselines in Table 1 and provide a brief overview of the algorithms considered, which are binned into three categories – knowledge-based, data-based, and competence-based algorithms.¹¹1We borrow this terminology from the following unsupervised RL tutorial srinivas_abbeel_2021_icml_tutorial. For detailed descriptions of each method we refer the reader to Appendix A.

Name	Algo. Type	Intrinsic Reward
ICM pathak2017curiosity	Knowledge	$∥ g (z_{t + 1} \| z_{t}, a_{t}) - z_{t + 1} ∥^{2}$
Disagreement Pathak19disagreement	Knowledge	$Var {g_{i} (z_{t + 1} \| z_{t}, a_{t})} i = 1, \dots, N$
RND burda2018exploration	Knowledge	$∥ g (z_{t}, a_{t}) - ~ g (z_{t}, a_{t}) ∥_{2}^{2}$
APT liu2021unsupervised	Data	$\sum_{j \in random} log ∥ z_{t} - z_{j} ∥ j = 1, \dots, K$
ProtoRL yarats21protorl	Data	$\sum_{j \in prototypes} log ∥ z_{t} - z_{j} ∥ j = 1, \dots, K$
SMM lee2019smm	Competence	$log p^{*} (z) - log q_{w} (z) - log p (w) + log d (w \| z)$
DIAYN EysenbachGIL19diayn	Competence	$log q (w \| z) + const.$
APS liu21aps	Competence	$r_{t}^{APT} (z) + log q (z \| w)$

Table 1: Unsupervised RL Algorithms implemented in URLB.

Knowledge-based Baselines: Knowledge-based methods aim to increase knowledge about the world by maximizing prediction error. As part of the knowledge-based suite, we implement the Intrinsic Curiosity Module (ICM) pathak2017curiosity, Disagreement Pathak19disagreement, and Random Network Distillation (RND) burda2018exploration. All three methods utilize a function $g$ to either predict the dynamics $g (z_{t + 1} | z_{t}, a_{t})$ (ICM, Disagreement) or predict the output of a random network $g (z_{t}, a_{t})$ (RND), where $z$ is the encoding of $o$ . ICM and RND maximize prediction error while Disagreement maximizes prediction uncertainty.

Data-based Baselines: Data-based methods aim to achieve data diversity by maximizing entropy. We implement APT liu2021unsupervised and ProtoRL yarats21protorl both of which maximize entropy $H (z)$

in different ways. Both methods utilize a particle estimator

singh03entropy

to maximize the entropy by maximizing the distance between k-nearest neighbors (kNN) for each state or observation embedding

z

. Since computing kNN over the entire replay buffer is expensive, APT estimates entropy across transitions in a randomly sampled minibatch. ProtoRL improves on APT by clustering the replay buffer with a contrastive deep clustering algorithm SWaV caron20swav. The centroids of the clusters are called prototypes, which are used by ProtoRL to estimate entropy.

Competence-based Baselines: Competence-based algorithms, learn an explicit skill vector $w$ by maximizing the mutual information between the encoded observation and skill $I (z; w)$ . This mutual information can be decomposed in two ways, $I (z; w) = H (z) - H (z | w) = H (w) - H (w | z)$ . We provide baselines for both decompositions. The former decomposition is utilized in skill discovery algorithms such as DIAYN EysenbachGIL19diayn, VIC GregorRW17vic, VALOR achiam2018valor, which are conceptually similar. For URLB, we implement DIAYN. The latter decomposition, though less common, is implemented in the APS liu21aps, which uses a particle estimator for the entropy term and successor features to represent the conditional entropy hansen20visr. Lastly, we implement SMM lee2019smm which combines both decompositions into one objective. Note that the SMM paper describes both skill-based and skill-free variants, so it can be categorized as both competence and data-based.

5 Experiments

Figure 3: Aggregate results for each algorithm category after pre-training the agent with intrinsic rewards for 2M environment steps and finetuning with extrinisc rewards for 100k steps as described in Sec. 3.2
. Scores are normalized by the asymptotic performance on each task (i.e., DrQ-v2 and DDPG performance after training from 2M steps on pixels and states correspondingly) and we show the mean and standard error of each category. Each algorithm is evaluated across ten random seeds. To provide an aggregate view of each algorithm category, the scores are averaged over individual tasks and methods (see Appendix
C for detailed results for each algorithm and downstream task). The Random Init baseline represents DrQ-v2 and DDPG trained from a random initialization for 100k steps. Full results can be found in Section C.

We evaluate the algorithms listed in Table 1 by pre-training with the intrinsic reward objective and fine-tuning on the downstream task as described in Section 3.2. For DrQ-v2 optimization we fix the hyper-parameters from yarats2021drqv2 and for algorithm-specific hyper-parameters we perform a grid sweep and pick the best performing parameters. We benchmark both state and pixel-based experiments and keep all non-algorithm-specific architectural details the same with a full description available in Appendix B. Performance on each downstream task is evaluated over ten random seeds and we display the mean scores and standard errors. We summarize the main results of our evaluation in Figures 3 and 6, which show evaluation scores grouped by algorithm category, described in Section 4.2, and environment, described in Section 3.3. An extensive list of results across all algorithms considered in this work can be found in Appendix C.

By benchmarking a wide array of exploration algorithms on both state and pixel-based tasks we are able to get perspective on the current state of unsupervised RL. Overall, we find that while unsupervised RL shows promise, it is still far from solving the proposed benchmark and many open questions need to be addressed to make progress toward unsupervised pre-training for RL. We note that solving the benchmark means matching the asymptotic DrQ-v2 (for pixels) and DDPG (for states) performance within 100k steps of fine-tuning. The motivation for this definition is that unsupervised RL agents get access to unlimited reward-free environment interactions. After pre-training, we seek to develop agents that adapt quickly to the desired downstream task. We list our observations below:

O1: None of the implemented unsupervised RL algorithms solve the benchmark. Despite access to up to 2M pre-training steps, after 100k steps of fine-tuning no method matches asymptotic performance on most tasks. The best-performing benchmarked algorithms achieve $40 - 70 %$ normalized return whereas the benchmark is considered solved when the agent achieves near $100 %$ normalized returns. This suggests that we are still far as a community from efficient generalization in deep RL.

O2: Unsupervised RL is not universally better than random initialization. We also observe that fine-tuning an unsupervised RL baseline is not always preferable to fine-tuning from a random initialization. In particular when learning from states, a random initialization is competitive with most baselines. However, when learning from pixels fine-tuning from random initialization degrades suggesting that representation learning is an important component of unsupervised pre-training.

O3: There exists a large gap in performance between exploring from states and exploring from pixels. Another observation that supports representation learning as an important aspect of exploration is that exploration algorithms degrade substantially when learning from pixels compared to learning from state. Shown in Figure 3, most algorithms lose $20 - 50 %$ when learning from pixels compared to state and especially so on the harder environments (Quadruped, Jaco Arm). These results suggest that better representation learning during pre-training is an important research direction.

O4: In aggregate, competence-based approaches underperform knowledge-based and data-based approaches. While knowledge-based and data-based approaches both perform competitively across URLB, we find that competence-based approaches are lagging behind. Specifically, there is no competence-based approach that achieves state-of-the-art mean performance on any of the URLB tasks, which points to competence-based unsupervised RL as an impactful research direction with significant room for improvement.

O5: There is not a single leading unsupervised RL algorithm for both states and pixels. We observe that there is no single state-of-the-art algorithm for unsupervised RL. At 2M pre-training steps, APT liu2021unsupervised and ProtoRL yarats21protorl are the leading algorithms for state-based URLB while ICM pathak2017curiosity achieves leading performance on pixel-based URLB despite the existence of more sophisticated knowledge-based methods Pathak19disagreement; burda2018exploration (see Figure 7).

O6: For many unsupervised RL algorithms, rather than monotonically improving performance decays as a function of pre-training steps. We desire and would expect that the fine-tuning efficiency of unsupervised RL algorithms would improve as a function of pre-training steps. Surprisingly, we find that for 9 out of 18 experiments shown in Figure 6, performance either does not improve or even degrades as a function of pre-training steps. We see this as potentially the biggest drawback of current unsupervised RL approaches – they do not scale with the number of environment interactions. Developing algorithms that improve monotonically as a function of pre-training steps is an open and impactful line of research.

O7: New fine-tuning strategies will likely be needed for fast adaptation. While not investigated in depth in this benchmark, new fine-tuning strategies could play a large role in the adoption of unsupervised RL. Perhaps part of the issue raised in O6 could be addressed with better fine-tuning. The algorithms in URLB are all fine-tuned by initializing the actor-critic with the pre-trained weights and fine-tuning with an extrinsic reward. There are likely other better strategies for fine-tuning, particularly for competence based approaches that are conditioned on the skill $w$ .

6 Related work

Deep Reinforcement Learning Benchmarks. Part of the accelerated progress in deep RL over the last few years has been due to the existence of stable benchmarks. Specifically, the Atari Arcade Learning Environment bellemare2013arcade, the OpenAI gym brockman2016openai, and more recently the DeepMind Control (DMC) Suite tassa2018deepmind have become standard benchmarks for evaluating supervised RL agents in both state and pixel-based observation spaces and discrete and continuous action spaces. Open-sourcing code for algorithms has been another aspect that accelerated progress in deep RL. For instance, duan2016benchmarking not only presented a benchmark for continuous control but also provided baselines for common supervised RL algorithms, which led to the development of the widely used OpenAI gym benchmark brockman2016openai and baselines baselines. The combination of challenging yet feasible benchmarks and open-sourced code were important components in the discovery of many widely adopted RL algorithms sac; mnih2015human; trpo; schulman2015high; ppo.

In addition to Atari, OpenAi gym, and DeepMind control, there have been many other benchmarks designed to study different aspects of supervised RL. DeepMind lab beattie2016deepmind benchmarks 3D navigation from pixels, ProcGen cobbe2019quantifying; cobbe2020leveraging measures generalization of supervised agents in procedurally generated environments, D4RL fu2020d4rl and RL unplugged gulcehre2020rl benchmark performance of offline RL methods, B-Pref lee2021bpref benchmarks performance of preference-based RL methods, Metaworld yu2020meta measures the performance of multi-task and meta-RL algorithms, and SafetyGym ray2019benchmarking measures how RL agents can achieve tasks with safety constraints. However, while the existing benchmarks are suitable for supervised RL algorithms, there is no such benchmark and collections of easy-to-use baseline algorithms for unsupervised RL, which is our primary motivation for accelerating progress in unsupervised RL through URLB.

Unsupervised Reinforcement Learning. While investigations into unsupervised deep RL appeared shortly after the landmark DQN mnih2015human, the field has experienced accelerated progress over the last year, which has been in part due to advents in unsupervised representation learning in CV chen2020simclr; he2020moco; henaff2019cpcv2 and NLP brown2020language; devlin2018bert; radford2019language as well as the development for stable RL optimization algorithms sac; hessel18rainbow; lillicrap15ddpg; ppo. However, unlike CV and NLP which focus solely on unsupervised representation learning, unsupervised RL has required both unsupervised representation and behavioral learning.

Unsupervised Representation Learning for Deep RL: In order for an RL algorithm to learn a policy $π (a | s)$ it must first have a good representation for the state $s$ . When working with coordinate state, the representation is supplied by a the human task designer but when operating from image observations $o$ , we must first transform the observations into latent vectors $z$ . This transformation comprises the study of representation learning for RL. One of the first seminal works on unsupervised representation learning for RL showed that unsupervised auxiliary tasks improve performance of supervised RL jaderberg17unreal. Over the last two years, a series of works in unsupervised representation learning for RL with world models hafner2018learning; hafner2019dream contrastive learning laskin2020curl; stooke2020decoupling; yarats21protorl

, autoencoders

yarats2019improving, and data augmentation laskin2020reinforcement; yarats2021drqv2; yarats2021image have dramaticaly improved learning efficiency from pixels. On many tasks from the DMC suite, learning from pixels is now as data-efficient as learning from state laskin2020curl.

Unsupervised Behavioral Learning for Deep RL: One caveat is that the above algorithms are not fully unsupervised since they still optimize for an extrinsic reward but with an auxiliary unsupervised loss. Fully unsupervised RL also requires unsupervised learning of behaviors, which is typically achieved by optimizing for an intrinsic reward oudeyer2007intrinsic. Given that representation learning is already heavily benchmarked for RL hafner2018learning; laskin2020curl; yarats2021image, URLB focuses mostly on unsupervised behavior learning. Many recent algorithms have been proposed for intrinsic behavioral learning, which include prediction methods burda2018largescale; pathak2017curiosity; Pathak19disagreement, maximal entropy-based methods campos2021beyond; liu2021unsupervised; liu21aps; mutti2020policy; seo21re3; yarats21protorl, and maximal mutual information-based methods EysenbachGIL19diayn; hansen20visr; liu21aps; SharmaGLKH20dads. However, these methods use different pre-training and evaluation procedures, different optimization algorithms, and different environments. To make fully unsupervised RL algorithm comparisons transparent and easier to develop, we introduce URLB.

7 Conclusion

We presented URLB, a benchmark designed to measure the performance of unsupervised RL algorithms. URLB consists of a suite of twelve evaluation tasks of varying difficulty from three domains and standardized procedures for pre-training and evaluation. We’ve open-sourced implementations and evaluation scores for eight leading unsupervised RL algorithms from all major algorithm categories. To minimize confounding factors, we utilized the same optimization method across all baselines. While none of the implemented baselines solve URLB, many make substantial progress suggesting a number of fruitful directions for unsupervised RL research. We hope that this benchmark makes the development and comparison of unsupervised RL algorithms easier and clearer.

Limitations. There are a number of limitations for both URLB and unsupervised RL methods in general. While URLB tasks are designed to be challenging, they are far from the visual and combinatorial complexity of real-world robotics. However, existing algorithms are unable to solve the benchmark meaning there is substantial room for improvement on the URLB tasks before moving on to even more challenging ones. While we present standardized pre-training and evaluation procedures, there can be many other ways of measuring the quality of the exploration algorithm. For instance, the quality of pre-training can be evaluated not only through policy adaptation but also through dataset diversity which we do not consider in this paper. In this work, similar to the Atari bellemare2013arcade and DMC tassa2018deepmind benchmarks for supervised RL we do not consider goal-conditioned RL which can be quite powerful for exploration ecoffet2020goexplore. For generality, we chose the currently most commonly used evaluation procedure that allowed us to benchmark a diverse set of leading exploration algorithms but, of course, other choices are available and would be interesting to investigate in future work.

Potential negative impacts. Unsupervised RL has the benefits of requiring zero extrinsic reward interactions during pre-training, and due to this the resulting agents may develop policies that are not aligned with human intent. This could be problematic in the long-term if not addressed early and carefully because as unsupervised robotics get more capable they can inadvertently inflict harm on themselves or the environment. Methods for constraining exploration within a broad set of human preferences (e.g. explore without harming the environment) is an interesting and important direction for future research in order to produced safe agents.

Acknowledgements

This work was partially supported by Berkeley DeepDrive, BAIR, the Berkeley Center for Human-Compatible AI, the Office of Naval Research grant N00014-21-1-2769, and DARPA through the Machine Common Sense Program.

References

Checklist

For all authors…
1. Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
2. Did you describe the limitations of your work? See Section 7.
3. Did you discuss any potential negative societal impacts of your work? See Section 7.
4. Have you read the ethics review guidelines and ensured that your paper conforms to them?
If you are including theoretical results…
1. Did you state the full set of assumptions of all theoretical results?
2. Did you include complete proofs of all theoretical results?
If you ran experiments (e.g. for benchmarks)…
1. Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? See Section 5
  , the appendix for hyperparameters. You can access the code with full instructions in the supplementary materials or using this link
  https://github.com/rll-research/url_benchmark.
2. Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? See Section 5 and 3.2 and the supplementary material.
3. Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? See Section 5 and the supplementary material.
4. Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? See Section F.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets…
1. If your work uses existing assets, did you cite the creators?
2. Did you mention the license of the assets?
3. Did you include any new assets either in the supplemental material or as a URL? See custom_dmc_tasks folder in supplementary materials codebase.
4. Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
5. Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
If you used crowdsourcing or conducted research with human subjects…
1. Did you include the full text of instructions given to participants and screenshots, if applicable?
2. Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
3. Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Appendix:

Unsupervised Reinforcement Learning Benchmark

Appendix A Unsupervised Reinforcement Learning Baselines

a.1 Knowledge-based Baselines

Prediction methods train a forward dynamics model $f (o_{t + 1} | o_{t}, a_{t})$ and define a self-supervised task based on the outputs of the model prediction.

Curiosity pathak2017curiosity: The Intrinsic Curiosity Module (ICM) defines the self-supervised task as the error between the state prediction of a learned dynamics model ${^z}^{'} \sim g (z^{'} | z, a)$ and the observation. The intuition is that parts of the state space that are hard to predict are good to explore because they were likely to be unseen before. An issue with Curiosity is that it is susceptible to the noisy TV problem wherein stochastic elements of the environment will always cause high prediction error while not being informative for exploration:

r_{t}^{ICM} \propto ∥ g (z_{t + 1} | z_{t}, a_{t}) - z_{t + 1} ∥^{2} .

Disagreement Pathak19disagreement:

Disagreement is similar to ICM but instead trains an ensemble of forward models and defines the intrinsic reward as the variance (or disagreement) among the models. Disagreement has the favorable property of not being susceptible to the noisy TV problem, since high stochasticity in the environment will result high prediction error but low variance if it has been thoroughly explored:

r_{t}^{Disagreement} \propto Var {g_{i} (z_{t + 1} | z_{t}, a_{t})} i = 1, \dots, N .

RND burda2018exploration:

Random Network Distillation (RND) defines the self-supervised task by predicting the output of a frozen randomly initialized neural network

~ f

. This differs from ICM only in that instead of predicting the next state, which is effectively an environment-defined function, it tries to predict the vector output of a randomly defined function. Similar to ICM, RND can suffer from the noisy TV problem:

r_{t}^{RND} \propto ∥ g (z_{t}, a_{t}) - ~ g (z_{t}, a_{t}) ∥_{2}^{2} .

a.2 Data-based Baselines

Recently, exploration through state entropy maximization has resulted in simple yet effective algorithms for unsupervised pre-training. We implement two leading variants of this approach for URLB.

APT liu2021unsupervised: Active Pre-training (APT) utilizes a particle-based estimator singh03entropy that uses K nearest-neighbors to estimate entropy for a given state or image embedding. Since APT does not itself perform representation learning, it requires an auxiliary representation learning loss to provide latent vectors for entropy estimation, although it is also possible to use random network embeddings seo21re3. We provide implementations of APT with the forward $g (z_{t + 1} | z_{t}, a_{t})$ and inverse dynamics $h (a_{t} | z_{t + 1}, z_{t})$ representation learning losses:

r_{t}^{APT} \propto \sum j \in random log ∥ z_{t} - z_{j} ∥ j = 1, \dots, K .

ProtoRL yarats21protorl: ProtoRL devises a self-supervised pre-training scheme that allows to decouple representation learning and exploration to enable efficient downstream generalization to previously unseen tasks. For this, ProtoRL uses the contrastive clustering assignment loss from SWaV (caron20swav) and learns latent representations and a set of prototypes to form the basis of the latent space. The prototypes are then used for more accurate estimation of entropy of the state-visitation distribution via KNN particle-based estimator:

r_{t}^{Proto} \propto \sum j \in prototypes log ∥ z_{t} - z_{j} ∥ j = 1, \dots, K .

a.3 Competence-based Baselines

Competence-based approaches learn skills $w$ that maximize the mutual information between encoded observations (or states) and skills $I (z; w)$ . The mutual information has two decompositions $I (z; w) = H (w) - H (w | z) = H (z) - H (z | w)$ . We provide baselines for both decompositions.

SMM lee2019smm: SMM minimizes $D_{K L} (p_{π} (z) ∥ p^{*} (z))$ , which maximizes the state entropy, while minimizing the cross entropy from the state to the target state distribution. When using skills, $H (z)$ can be rewritten as $H (z | w) + I (z; w)$ . $H (z | w)$ can maximized by optimizing the reward $r = log q_{w} (z)$ , which is estimated using a VAE kingma2013auto that models the density of $z$ while executing skill $w$ . Similar to other mutual information methods that decompose $I (z; w) = H (w) - H (w | z)$ , SMM learns a discriminator $d (w | z)$ over a set of discrete skills with a uniform prior that maximizes $H (w)$ :

rSMMt\ensurestackMath\stackon[1pt]=Δlogp∗(z)−logqw(z)−logp(w)+logd(w|z).

DIAYN EysenbachGIL19diayn: DIAYN and similar algorithms such as VIC GregorRW17vic and VALOR achiam2018valor are perhaps the best competence-based exploration algorithms. These methods estimate the mutual information through the first decomposition $I (z; w) = H (w) - H (w | z)$ . $H (w)$ is kept maximal by drawing $w \sim p (w)$ from a discrete uniform prior distribution and the density $- H (w | z)$ is estimated with a discriminator $log q (w | z)$ .

r_{t}^{DIAYN} \propto log q (w | z) + const.

APS liu21aps: APS is a recent leading mutual information exploration method that uses the second decomposition $I (z; w) = H (z) - H (z | w)$ . $H (z)$ is estimated with a particle estimator as in APT liu2021unsupervised while $H (z | w)$ is estimated with successor features as in VISR hansen20visr.²²2In this benchmark, the generalized policy improvement(GPI) barreto2018transfer that is used in Atari games for APS and VISR is not implemented for continuous control experiments.

r_{t}^{APS} \propto r_{t}^{APT} (z) + log q (z | w)

Appendix B Hyper-parameters

In Table 2 we present a common set of hyper-parameters used in our experiments, while in table 3 we list individual hyper-parameters for each method.

Common hyper-parameter	Value
Replay buffer capacity	$10^{6}$
Action repeat	$1$ states-based and $2$ for pixels-based
Seed frames	$4000$
$n$ -step returns	$3$
Mini-batch size	$1024$ states-based and $256$ for pixels-based
Seed frames	$4000$
Discount ( $γ$ )	$0.99$
Optimizer	Adam
Learning rate	$10^{- 4}$
Agent update frequency	$2$
Critic target EMA rate ( $τ_{Q}$ )	$0.01$
Features dim.	$1024$ states-based and $50$ for pixels-based
Hidden dim.	$1024$
Exploration stddev clip	$0.3$
Exploration stddev value	$0.2$
Number pre-training frames	up to $2 \times 10^{6}$
Number fine-turning frames	$1 \times 10^{5}$

Table 2: A common set of hyper-parameters used in our experiments.

ICM hyper-parameter	Value
Representation dim.	$512$
Reward transformation	$log (r + 1.0)$
Forward net arch.	$(\| O \| + \| A \|) \to 1024 \to 1024 \to \| O \|$ ReLU MLP
Inverse net arch.	$(2 \times \| O \|) \to 1024 \to 1024 \to \| A \|$ ReLU MLP
Disagreement hyper-parameter	Value
Ensemble size	$5$
Forward net arch:	$(\| O \| + \| A \|) \to 1024 \to 1024 \to \| O \|$ ReLU MLP
RND hyper-parameter	Value
Representation dim.	$512$
Predictor & target net arch.	$\| O \| \to 1024 \to 1024 \to 512$ ReLU MLP
Normalized observation clipping	5
APT hyper-parameter	Value
Representation dim.	$512$
Reward transformation	$log (r + 1.0)$
Forward net arch.	$(512 + \| A \|) \to 1024 \to 512$ ReLU MLP
Inverse net arch.	$(2 \times 512) \to 1024 \to \| A \|$ ReLU MLP
$k$ in $N N$	$12$
Avg top $k$ in $N N$	True
ProtoRL hyper-parameter	Value
Predictor dim.	$128$
Projector dim.	$512$
Number of prototypes	$512$
Softmax temperature	$0.1$
$k$ in $N N$	$3$
Number of candidates per prototype	$4$
Encoder target EMA rate ( $τ_{e n c}$ )	$0.05$
SMM hyper-parameter	Value
Skill dim.	$4$
Skill discrim lr	$10^{- 3}$
VAE lr	$10^{- 2}$
DIAYN hyper-parameter	Value
Skill dim	16
Skill sampling frequency (steps)	50
Discriminator net arch.	$512 \to 1024 \to 1024 \to 16$ ReLU MLP
APS hyper-parameter	Value
Representation dim.	$512$
Reward transformation	$log (r + 1.0)$
Successor feature dim.	$10$
Successor feature net arch.	$\| O \| \to 1024 \to 1024 \to 10$ ReLU MLP
$k$ in $N N$	$12$
Avg top $k$ in $N N$	True
Least square batch size	$4096$

Table 3: Per algorithm sets of hyper-parameters used in our experiments.

Appendix C Per-domain Individual Results

Individual fine-tuning results for each methods are shown in Figure 7. Furthermore, Figures 8 and 9 demonstrate individual results of states and pixels based fine-tuning performance as a function of pre-training steps for each considered method and task.

Figure 7: Individual results of fine-tuning for 100k steps after different degrees of pre-training for each considered method. The performance is aggregated across all the tasks within a domain and normalized with respect to the optimal performance.

Figure 8: Individual results of fine-tuning efficiency as a function of pre-training steps for states-based learning.

Figure 9: Individual results of fine-tuning efficiency as a function of pre-training steps for pixels-based learning.

Appendix D Finetuning Learning Curves

We provide finetuning learning curves for agents pre-trained for 2M steps with intrinsic rewards.

Figure 10: Finetuning curves for each evaluated unsupervised algorithm for each task considered in this benchmark after the agent has been pre-trained with intrinsic rewards.

Appendix E Individual Numerical Results

The individual numerical results of fine-tuning for each task and each method are presented in Table 4 for states-based learning, and in Table 5 for pixels-based learning.

Pre-trainining for $1 \times 10^{5}$ frames
Domain	Task	DDPG (DrQ-v2)	ICM	Disagreement	RND	APT	ProtoRL	SMM	DIAYN	APS
Walker	Flip	538 $\pm$ 27	535 $\pm$ 19	559 $\pm$ 20	581 $\pm$ 21	580 $\pm$ 28	523 $\pm$ 16	388 $\pm$ 36	352 $\pm$ 22	638 $\pm$ 33
	Run	325 $\pm$ 25	384 $\pm$ 24	437 $\pm$ 24	437 $\pm$ 33	424 $\pm$ 28	250 $\pm$ 34	244 $\pm$ 28	259 $\pm$ 26	428 $\pm$ 26
	Stand	899 $\pm$ 23	944 $\pm$ 5	937 $\pm$ 6	947 $\pm$ 6	925 $\pm$ 9	926 $\pm$ 24	738 $\pm$ 61	784 $\pm$ 68	872 $\pm$ 34
	Walk	748 $\pm$ 47	805 $\pm$ 46	911 $\pm$ 14	857 $\pm$ 32	888 $\pm$ 19	831 $\pm$ 31	592 $\pm$ 53	584 $\pm$ 25	731 $\pm$ 70
Quadruped	Jump	236 $\pm$ 48	291 $\pm$ 35	261 $\pm$ 45	383 $\pm$ 57	334 $\pm$ 48	220 $\pm$ 33	386 $\pm$ 56	267 $\pm$ 34	589 $\pm$ 57
	Run	157 $\pm$ 31	195 $\pm$ 31	198 $\pm$ 40	203 $\pm$ 20	161 $\pm$ 27	138 $\pm$ 21	224 $\pm$ 33	179 $\pm$ 26	420 $\pm$ 49
	Stand	392 $\pm$ 73	390 $\pm$ 59	420 $\pm$ 76	446 $\pm$ 17	559 $\pm$ 57	425 $\pm$ 82	430 $\pm$ 86	350 $\pm$ 55	662 $\pm$ 64
	Walk	229 $\pm$ 57	185 $\pm$ 20	265 $\pm$ 45	229 $\pm$ 26	173 $\pm$ 29	141 $\pm$ 23	227 $\pm$ 47	193 $\pm$ 29	664 $\pm$ 56
Jaco	Reach bottom left	72 $\pm$ 22	117 $\pm$ 24	100 $\pm$ 20	121 $\pm$ 19	124 $\pm$ 19	86 $\pm$ 20	64 $\pm$ 13	64 $\pm$ 15	156 $\pm$ 9
	Reach bottom right	117 $\pm$ 18	155 $\pm$ 10	179 $\pm$ 6	161 $\pm$ 8	141 $\pm$ 14	82 $\pm$ 21	68 $\pm$ 17	44 $\pm$ 8	164 $\pm$ 10
	Reach top left	116 $\pm$ 22	152 $\pm$ 24	143 $\pm$ 14	141 $\pm$ 15	136 $\pm$ 23	110 $\pm$ 19	33 $\pm$ 6	26 $\pm$ 6	153 $\pm$ 13
	Reach top right	94 $\pm$ 18	159 $\pm$ 15	159 $\pm$ 15	168 $\pm$ 9	175 $\pm$ 6	116 $\pm$ 22	47 $\pm$ 12	59 $\pm$ 12	186 $\pm$ 7
Pre-trainining for $5 \times 10^{5}$ frames
Domain	Task	DDPG	ICM	Disagreement	RND	APT	ProtoRL	SMM	DIAYN	APS
Walker	Flip	538 $\pm$ 27	554 $\pm$ 27	568 $\pm$ 42	685 $\pm$ 46	594 $\pm$ 29	501 $\pm$ 19	472 $\pm$ 30	380 $\pm$ 24	637 $\pm$ 36
	Run	325 $\pm$ 25	416 $\pm$ 18	485 $\pm$ 25	499 $\pm$ 21	410 $\pm$ 27	228 $\pm$ 18	328 $\pm$ 19	241 $\pm$ 19	337 $\pm$ 25
	Stand	899 $\pm$ 23	930 $\pm$ 7	940 $\pm$ 8	946 $\pm$ 5	930 $\pm$ 5	925 $\pm$ 17	906 $\pm$ 18	762 $\pm$ 44	869 $\pm$ 27
	Walk	748 $\pm$ 47	846 $\pm$ 30	923 $\pm$ 5	869 $\pm$ 23	826 $\pm$ 40	865 $\pm$ 16	791 $\pm$ 43	632 $\pm$ 34	778 $\pm$ 58
Quadruped	Jump	236 $\pm$ 48	252 $\pm$ 41	452 $\pm$ 45	542 $\pm$ 51	282 $\pm$ 48	225 $\pm$ 32	387 $\pm$ 56	350 $\pm$ 59	493 $\pm$ 55
	Run	157 $\pm$ 31	184 $\pm$ 42	368 $\pm$ 28	377 $\pm$ 28	182 $\pm$ 22	153 $\pm$ 29	205 $\pm$ 26	258 $\pm$ 24	347 $\pm$ 39
	Stand	392 $\pm$ 73	422 $\pm$ 49	649 $\pm$ 53	722 $\pm$ 49	470 $\pm$ 80	433 $\pm$ 63	499 $\pm$ 78	459 $\pm$ 54	743 $\pm$ 46
	Walk	229 $\pm$ 57	237 $\pm$ 43	412 $\pm$ 67	498 $\pm$ 68	217 $\pm$ 29	209 $\pm$ 47	238 $\pm$ 27	218 $\pm$ 24	553 $\pm$ 74
Jaco	Reach bottom left	72 $\pm$ 22	94 $\pm$ 16	145 $\pm$ 13	113 $\pm$ 11	123 $\pm$ 15	106 $\pm$ 18	61 $\pm$ 10	38 $\pm$ 9	134 $\pm$ 6
	Reach bottom right	117 $\pm$ 18	119 $\pm$ 15	136 $\pm$ 15	144 $\pm$ 6	136 $\pm$ 13	115 $\pm$ 19	82 $\pm$ 10	63 $\pm$ 11	131 $\pm$ 8
	Reach top left	116 $\pm$ 22	125 $\pm$ 18	165 $\pm$ 9	121 $\pm$ 16	118 $\pm$ 18	122 $\pm$ 23	57 $\pm$ 9	29 $\pm$ 10	124 $\pm$ 11
	Reach top right	94 $\pm$ 18	151 $\pm$ 8	181 $\pm$ 7	150 $\pm$ 8	170 $\pm$ 8	120 $\pm$ 22	60 $\pm$ 10	43 $\pm$ 6	106 $\pm$ 10
Pre-trainining for $1 \times 10^{6}$ frames
Domain	Task	DDPG	ICM	Disagreement	RND	APT	ProtoRL	SMM	DIAYN	APS
Walker	Flip	538 $\pm$ 27	524 $\pm$ 20	586 $\pm$ 36	610 $\pm$ 27	505 $\pm$ 22	480 $\pm$ 17	486 $\pm$ 17	338 $\pm$ 23	531 $\pm$ 24
	Run	325 $\pm$ 25	344 $\pm$ 30	488 $\pm$ 23	482 $\pm$ 23	373 $\pm$ 22	254 $\pm$ 29	332 $\pm$ 40	249 $\pm$ 24	352 $\pm$ 31
	Stand	899 $\pm$ 23	922 $\pm$ 15	919 $\pm$ 11	946 $\pm$ 5	916 $\pm$ 20	905 $\pm$ 25	903 $\pm$ 18	870 $\pm$ 34	846 $\pm$ 28
	Walk	748 $\pm$ 47	845 $\pm$ 27	880 $\pm$ 19	873 $\pm$ 24	821 $\pm$ 35	848 $\pm$ 29	746 $\pm$ 51	553 $\pm$ 33	808 $\pm$ 61
Quadruped	Jump	236 $\pm$ 48	306 $\pm$ 42	595 $\pm$ 39	615 $\pm$ 53	400 $\pm$ 53	287 $\pm$ 52	349 $\pm$ 63	365 $\pm$ 15	415 $\pm$ 46
	Run	157 $\pm$ 31	157 $\pm$ 24	444 $\pm$ 39	444 $\pm$ 42	237 $\pm$ 35	206 $\pm$ 32	280 $\pm$ 40	343 $\pm$ 25	400 $\pm$ 48
	Stand	392 $\pm$ 73	428 $\pm$ 79	736 $\pm$ 51	763 $\pm$ 79	526 $\pm$ 58	436 $\pm$ 62	391 $\pm$ 36	529 $\pm$ 52	712 $\pm$ 57
	Walk	229 $\pm$ 57	140 $\pm$ 24	729 $\pm$ 46	644 $\pm$ 70	246 $\pm$ 33	266 $\pm$ 64	312 $\pm$ 63	525 $\pm$ 76	505 $\pm$ 84
Jaco	Reach bottom left	72 $\pm$ 22	114 $\pm$ 9	144 $\pm$ 9	114 $\pm$ 10	125 $\pm$ 10	122 $\pm$ 22	58 $\pm$ 8	43 $\pm$ 13	87 $\pm$ 8
	Reach bottom right	117 $\pm$ 18	126 $\pm$ 10	129 $\pm$ 10	106 $\pm$ 13	128 $\pm$ 12	113 $\pm$ 20	62 $\pm$ 9	34 $\pm$ 6	109 $\pm$ 9
	Reach top left	116 $\pm$ 22	146 $\pm$ 11	156 $\pm$ 10	136 $\pm$ 13	110 $\pm$ 5	114 $\pm$ 19	61 $\pm$ 7	12 $\pm$ 2	108 $\pm$ 13
	Reach top right	94 $\pm$ 18	143 $\pm$ 10	159 $\pm$ 10	132 $\pm$ 10	149 $\pm$ 11	123 $\pm$ 21	61 $\pm$ 9	31 $\pm$ 9	101 $\pm$ 9
Pre-trainining for $2 \times 10^{6}$ frames
Domain	Task	DDPG	ICM	Disagreement	RND	APT	ProtoRL	SMM	DIAYN	APS
Walker	Flip	538 $\pm$ 27	514 $\pm$ 25	491 $\pm$ 21	515 $\pm$ 17	477 $\pm$ 16	480 $\pm$ 23	505 $\pm$ 26	381 $\pm$ 17	461 $\pm$ 24
	Run	325 $\pm$ 25	388 $\pm$ 30	444 $\pm$ 21	439 $\pm$ 34	344 $\pm$ 28	200 $\pm$ 15	430 $\pm$ 26	242 $\pm$ 11	257 $\pm$ 27
	Stand	899 $\pm$ 23	913 $\pm$ 12	907 $\pm$ 15	923 $\pm$ 9	914 $\pm$ 8	870 $\pm$ 23	877 $\pm$ 34	860 $\pm$ 26	835 $\pm$ 54
	Walk	748 $\pm$ 47	713 $\pm$ 31	782 $\pm$ 33	828 $\pm$ 29	759 $\pm$ 35	777 $\pm$ 33	821 $\pm$ 36	661 $\pm$ 26	711 $\pm$ 68
Quadruped	Jump	236 $\pm$ 48	205 $\pm$ 33	668 $\pm$ 24	590 $\pm$ 33	462 $\pm$ 48	425 $\pm$ 63	298 $\pm$ 39	578 $\pm$ 46	538 $\pm$ 42
	Run	157 $\pm$ 31	133 $\pm$ 20	461 $\pm$ 12	462 $\pm$ 23	339 $\pm$ 40	316 $\pm$ 36	220 $\pm$ 37	415 $\pm$ 28	465 $\pm$ 37
	Stand	392 $\pm$ 73	329 $\pm$ 58	840 $\pm$ 33	804 $\pm$ 50	622 $\pm$ 57	560 $\pm$ 71	367 $\pm$ 42	706 $\pm$ 48	714 $\pm$ 50
	Walk	229 $\pm$ 57	143 $\pm$ 31	721 $\pm$ 56	826 $\pm$ 19	434 $\pm$ 64	403 $\pm$ 91	184 $\pm$ 26	406 $\pm$ 64	602 $\pm$ 86
Jaco	Reach bottom left	72 $\pm$ 22	106 $\pm$ 8	134 $\pm$ 8	101 $\pm$ 12	88 $\pm$ 12	121 $\pm$ 22	40 $\pm$ 9	17 $\pm$ 5	96 $\pm$ 13
	Reach bottom right	117 $\pm$ 18	119 $\pm$ 9	122 $\pm$ 4	100 $\pm$ 10	115 $\pm$ 12	113 $\pm$ 16	50 $\pm$ 9	31 $\pm$ 4	93 $\pm$ 9
	Reach top left	116 $\pm$ 22	119 $\pm$ 12	117 $\pm$ 14	111 $\pm$ 10	112 $\pm$ 11	124 $\pm$ 20	50 $\pm$ 7	11 $\pm$ 3	65 $\pm$ 10
	Reach top right	94 $\pm$ 18	137 $\pm$ 9	140 $\pm$ 7	140 $\pm$ 10	136 $\pm$ 5	135 $\pm$ 19	37 $\pm$ 8	19 $\pm$ 4	81 $\pm$ 11

Table 4: Individual results of fine-tuning for

1 \times 10^{5}

frames after different levels of pre-training in the states-based settings.

Pre-trainining for $1 \times 10^{5}$ frames
Domain	Task	DrQ-v2	ICM	Disagreement	RND	APT	ProtoRL	SMM	DIAYN	APS
Walker	Flip	81 $\pm$ 23	252 $\pm$ 54	80 $\pm$ 33	214 $\pm$ 38	25 $\pm$ 1	293 $\pm$ 33	24 $\pm$ 1	132 $\pm$ 24	38 $\pm$ 7
	Run	41 $\pm$ 11	110 $\pm$ 21	57 $\pm$ 14	78 $\pm$ 13	25 $\pm$ 1	135 $\pm$ 13	22 $\pm$ 1	50 $\pm$ 6	26 $\pm$ 1
	Stand	212 $\pm$ 28	315 $\pm$ 67	250 $\pm$ 62	261 $\pm$ 34	162 $\pm$ 9	353 $\pm$ 67	133 $\pm$ 8	233 $\pm$ 22	162 $\pm$ 9
	Walk	141 $\pm$ 53	302 $\pm$ 45	192 $\pm$ 68	263 $\pm$ 43	43 $\pm$ 16	320 $\pm$ 52	23 $\pm$ 1	138 $\pm$ 25	29 $\pm$ 2
Quadruped	Jump	278 $\pm$ 35	226 $\pm$ 40	173 $\pm$ 15	223 $\pm$ 30	160 $\pm$ 24	246 $\pm$ 33	211 $\pm$ 25	204 $\pm$ 24	182 $\pm$ 32
	Run	156 $\pm$ 21	156 $\pm$ 13	112 $\pm$ 12	145 $\pm$ 17	134 $\pm$ 21	156 $\pm$ 27	148 $\pm$ 18	173 $\pm$ 23	133 $\pm$ 24
	Stand	309 $\pm$ 47	329 $\pm$ 49	259 $\pm$ 31	350 $\pm$ 43	266 $\pm$ 37	342 $\pm$ 35	297 $\pm$ 36	350 $\pm$ 48	265 $\pm$ 48
	Walk	151 $\pm$ 31	160 $\pm$ 10	134 $\pm$ 24	154 $\pm$ 16	119 $\pm$ 17	168 $\pm$ 24	149 $\pm$ 18	157 $\pm$ 23	161 $\pm$ 27
Jaco	Reach bottom left	23 $\pm$ 10	18 $\pm$ 7	12 $\pm$ 4	41 $\pm$ 7	0 $\pm$ 0	38 $\pm$ 11	1 $\pm$ 1	12 $\pm$ 4	0 $\pm$ 0
	Reach bottom right	23 $\pm$ 8	30 $\pm$ 12	23 $\pm$ 8	57 $\pm$ 8	0 $\pm$ 0	37 $\pm$ 9	1 $\pm$ 0	10 $\pm$ 3	0 $\pm$ 0
	Reach top left	40 $\pm$ 9	31 $\pm$ 11	30 $\pm$ 9	66 $\pm$ 9	0 $\pm$ 0	59 $\pm$ 14	2 $\pm$ 1	19 $\pm$ 4	2 $\pm$ 1
	Reach top right	37 $\pm$ 9	37 $\pm$ 13	22 $\pm$ 8	48 $\pm$ 7	3 $\pm$ 2	45 $\pm$ 16	4 $\pm$ 3	24 $\pm$ 8	4 $\pm$ 2
Pre-trainining for $5 \times 10^{5}$ frames
Domain	Task	DDPG	ICM	Disagreement	RND	APT	ProtoRL	SMM	DIAYN	APS
Walker	Flip	81 $\pm$ 23	260 $\pm$ 39	360 $\pm$ 16	222 $\pm$ 37	28 $\pm$ 2	210 $\pm$ 44	25 $\pm$ 1	117 $\pm$ 18	32 $\pm$ 2
	Run	41 $\pm$ 11	110 $\pm$ 15	131 $\pm$ 19	103 $\pm$ 13	26 $\pm$ 1	85 $\pm$ 16	23 $\pm$ 1	47 $\pm$ 4	27 $\pm$ 1
	Stand	212 $\pm$ 28	499 $\pm$ 62	398 $\pm$ 65	289 $\pm$ 26	155 $\pm$ 11	355 $\pm$ 61	139 $\pm$ 9	243 $\pm$ 16	161 $\pm$ 9
	Walk	141 $\pm$ 53	305 $\pm$ 51	348 $\pm$ 46	258 $\pm$ 33	37 $\pm$ 10	250 $\pm$ 54	23 $\pm$ 1	125 $\pm$ 19	48 $\pm$ 19
Quadruped	Jump	278 $\pm$ 35	286 $\pm$ 50	214 $\pm$ 24	366 $\pm$ 44	147 $\pm$ 26	229 $\pm$ 42	201 $\pm$ 20	248 $\pm$ 28	212 $\pm$ 27
	Run	156 $\pm$ 21	198 $\pm$ 29	153 $\pm$ 19	261 $\pm$ 38	112 $\pm$ 20	144 $\pm$ 27	138 $\pm$ 16	197 $\pm$ 24	178 $\pm$ 23
	Stand	309 $\pm$ 47	398 $\pm$ 69	298 $\pm$ 35	453 $\pm$ 47	229 $\pm$ 42	355 $\pm$ 67	279 $\pm$ 26	313 $\pm$ 31	281 $\pm$ 51
	Walk	151 $\pm$ 31	193 $\pm$ 31	129 $\pm$ 20	206 $\pm$ 30	111 $\pm$ 20	157 $\pm$ 25	139 $\pm$ 13	140 $\pm$ 19	141 $\pm$ 24
Jaco	Reach bottom left	23 $\pm$ 10	65 $\pm$ 20	30 $\pm$ 8	47 $\pm$ 8	0 $\pm$ 0	31 $\pm$ 14	1 $\pm$ 1	12 $\pm$ 3	0 $\pm$ 0
	Reach bottom right	23 $\pm$ 8	56 $\pm$ 16	34 $\pm$ 10	52 $\pm$ 6	0 $\pm$ 0	35 $\pm$ 11	1 $\pm$ 0	7 $\pm$ 2	0 $\pm$ 0
	Reach top left	40 $\pm$ 9	87 $\pm$ 22	47 $\pm$ 11	55 $\pm$ 8	1 $\pm$ 1	35 $\pm$ 15	2 $\pm$ 1	20 $\pm$ 4	2 $\pm$ 1
	Reach top right	37 $\pm$ 9	68 $\pm$ 17	33 $\pm$ 5	61 $\pm$ 8	2 $\pm$ 1	42 $\pm$ 14	4 $\pm$ 3	21 $\pm$ 5	1 $\pm$ 1
Pre-trainining for $1 \times 10^{6}$ frames
Domain	Task	DDPG	ICM	Disagreement	RND	APT	ProtoRL	SMM	DIAYN	APS
Walker	Flip	81 $\pm$ 23	256 $\pm$ 49	305 $\pm$ 34	250 $\pm$ 29	46 $\pm$ 17	244 $\pm$ 37	26 $\pm$ 1	126 $\pm$ 26	42 $\pm$ 12
	Run	41 $\pm$ 11	116 $\pm$ 16	155 $\pm$ 12	96 $\pm$ 13	26 $\pm$ 1	84 $\pm$ 11	24 $\pm$ 1	47 $\pm$ 4	30 $\pm$ 3
	Stand	212 $\pm$ 28	534 $\pm$ 73	565 $\pm$ 37	374 $\pm$ 37	150 $\pm$ 13	480 $\pm$ 63	145 $\pm$ 6	251 $\pm$ 19	170 $\pm$ 10
	Walk	141 $\pm$ 53	285 $\pm$ 45	433 $\pm$ 29	316 $\pm$ 41	36 $\pm$ 9	258 $\pm$ 39	25 $\pm$ 1	137 $\pm$ 25	54 $\pm$ 25
Quadruped	Jump	278 $\pm$ 35	345 $\pm$ 47	199 $\pm$ 31	368 $\pm$ 38	156 $\pm$ 27	237 $\pm$ 50	201 $\pm$ 20	319 $\pm$ 38	184 $\pm$ 29
	Run	156 $\pm$ 21	179 $\pm$ 11	140 $\pm$ 24	297 $\pm$ 36	115 $\pm$ 21	108 $\pm$ 16	139 $\pm$ 13	165 $\pm$ 17	155 $\pm$ 22
	Stand	309 $\pm$ 47	430 $\pm$ 44	257 $\pm$ 35	559 $\pm$ 51	229 $\pm$ 42	338 $\pm$ 71	279 $\pm$ 26	319 $\pm$ 27	275 $\pm$ 48
	Walk	151 $\pm$ 31	251 $\pm$ 25	104 $\pm$ 19	274 $\pm$ 19	115 $\pm$ 21	152 $\pm$ 28	139 $\pm$ 13	213 $\pm$ 20	146 $\pm$ 23
Jaco	Reach bottom left	23 $\pm$ 10	65 $\pm$ 19	42 $\pm$ 10	46 $\pm$ 9	0 $\pm$ 0	36 $\pm$ 13	1 $\pm$ 1	8 $\pm$ 3	0 $\pm$ 0
	Reach bottom right	23 $\pm$ 8	88 $\pm$ 23	58 $\pm$ 11	44 $\pm$ 9	0 $\pm$ 0	43 $\pm$ 10	1 $\pm$ 0	4 $\pm$ 1	0 $\pm$ 0
	Reach top left	40 $\pm$ 9	76 $\pm$ 19	89 $\pm$ 19	59 $\pm$ 7	6 $\pm$ 4	41 $\pm$ 10	2 $\pm$ 1	20 $\pm$ 4	2 $\pm$ 0
	Reach top right	37 $\pm$ 9	87 $\pm$ 24	49 $\pm$ 12	47 $\pm$ 7	2 $\pm$ 1	47 $\pm$ 12	4 $\pm$ 3	22 $\pm$ 6	5 $\pm$ 1
Pre-trainining for $2 \times 10^{6}$ frames
Domain	Task	DDPG	ICM	Disagreement	RND	APT	ProtoRL	SMM	DIAYN	APS
Walker	Flip	81 $\pm$ 23	231 $\pm$ 34	339 $\pm$ 16	280 $\pm$ 31	28 $\pm$ 2	223 $\pm$ 27	26 $\pm$ 1	114 $\pm$ 21	38 $\pm$ 9
	Run	41 $\pm$ 11	98 $\pm$ 11	154 $\pm$ 9	133 $\pm$ 15	25 $\pm$ 2	87 $\pm$ 18	24 $\pm$ 1	45 $\pm$ 3	30 $\pm$ 3
	Stand	212 $\pm$ 28	401 $\pm$ 40	552 $\pm$ 92	389 $\pm$ 49	155 $\pm$ 11	467 $\pm$ 69	145 $\pm$ 6	298 $\pm$ 71	172 $\pm$ 10
	Walk	141 $\pm$ 53	274 $\pm$ 44	424 $\pm$ 36	321 $\pm$ 43	35 $\pm$ 8	297 $\pm$ 48	25 $\pm$ 1	132 $\pm$ 19	37 $\pm$ 5
Quadruped	Jump	278 $\pm$ 35	312 $\pm$ 18	194 $\pm$ 21	383 $\pm$ 28	164 $\pm$ 27	197 $\pm$ 35	201 $\pm$ 20	262 $\pm$ 22	199 $\pm$ 29
	Run	156 $\pm$ 21	249 $\pm$ 21	143 $\pm$ 25	284 $\pm$ 18	121 $\pm$ 20	137 $\pm$ 35	139 $\pm$ 13	190 $\pm$ 18	156 $\pm$ 24
	Stand	309 $\pm$ 47	506 $\pm$ 40	305 $\pm$ 35	561 $\pm$ 43	243 $\pm$ 41	290 $\pm$ 56	279 $\pm$ 26	426 $\pm$ 40	331 $\pm$ 43
	Walk	151 $\pm$ 31	231 $\pm$ 15	145 $\pm$ 10	294 $\pm$ 20	122 $\pm$ 21	138 $\pm$ 35	139 $\pm$ 13	184 $\pm$ 23	146 $\pm$ 24
Jaco	Reach bottom left	23 $\pm$ 10	72 $\pm$ 20	106 $\pm$ 18	39 $\pm$ 3	0 $\pm$ 0	21 $\pm$ 5	1 $\pm$ 1	7 $\pm$ 2	1 $\pm$ 0
	Reach bottom right	23 $\pm$ 8	58 $\pm$ 19	90 $\pm$ 15	47 $\pm$ 9	0 $\pm$ 0	28 $\pm$ 7	1 $\pm$ 0	9 $\pm$ 3	1 $\pm$ 1
	Reach top left	40 $\pm$ 9	89 $\pm$ 22	127 $\pm$ 21	60 $\pm$ 6	0 $\pm$ 0	47 $\pm$ 16	2 $\pm$ 1	11 $\pm$ 2	2 $\pm$ 1
	Reach top right	37 $\pm$ 9	69 $\pm$ 18	118 $\pm$ 23	76 $\pm$ 11	1 $\pm$ 1	52 $\pm$ 12	4 $\pm$ 3	16 $\pm$ 3	10 $\pm$ 3

Table 5: Individual results of fine-tuning for

1 \times 10^{5}

frames after different levels of pre-training in the pixels-based settings.

Appendix F Compute Resources

URLB is designed to be accessible to the RL research community. Both state and pixel-based algorithms are implemented such that each algorithm requires a single GPU. For local debugging experiments we used NVIDIA RTX GPUs. For large-scale runs used to generate all results in this manuscripts, we used NVIDIA Tesla V100 GPU instances. All experiments were run on internal clusters. Each algorithm trains in roughly 30 mins - 12 hours depending on the snapshot (100k, 500k, 1M, 2M) and input (states, pixels). Since this benchmark required roughly 8k experiments (2 states / pixels, 12 tasks, 8 algorithms, 10 seeds, 4 snapshots) a total of 100 V100 GPUs were used to produce the results in this benchmark. Researchers who wish to build on URLB will, of course, not need to run this many experiments since they can utilize the results presented in this benchmark.

Appendix G Intuition on Competence-based Approaches Underperform on URLB

Across the three methods - data-based, knowledge-based, and competence-based - the best data-based and knowledge-based methods are competitive with one another. For instance, RND (a leading knowledge-based methods) and ProtoRL (a leading data-based method) achieve similar finetuning scores. Both are maximizing data diversity in two different ways - one through maximizing prediction error and the other through entropy maximization.

On the other hand, competence-based methods as a whole do much worse than data-based and knowledge-based ones. We hypothesize that this is due to current competence-based methods only supporting small skill spaces. Competence-based methods maximize a variational lower bound to the mutual information of the form:

I (τ; z) = H (z) - H (z | τ) = H (z) + E [log p (z | τ)] \geq H (z) + E [log q (z | τ)]

where $q (z | s)$

is called the discriminator. The discriminator can be interpreted as a classifier from

s \to z

(or vice versa depending on how you decompose

I (s; z)

). In order to have an accurate discriminator,

z

is chosen to be small in practice (DIAYN -

z

is a 16 dim one-hot, SMM -

z

is 4 dim continuous, APS -

z

is 10 dim continuous).

OpenAI gym environments for continuous control mask this limitation because they terminate if the agent falls over and hence leak extrinsic signal about the downstream task into the environment. This means that the agent learns only useful behaviors that keep it balanced and therefore a small skill vector is sufficient for classifying these behaviors. However, in DeepMind control (and hence URLB) the episodes have fixed length and therefore the set of possible behaviors is much larger. If the skill space is too small, the most likely skills to be classified are different configurations of the agent lying on the ground. We hypothesize that building more powerful discriminators would improve competence-based exploration.

URLB: Unsupervised Reinforcement Learning Benchmark

Authors

Evolving Rewards to Automate Reinforcement Learning

Behavior From the Void: Unsupervised Active Pre-Training

Parrot: Data-Driven Behavioral Priors for Reinforcement Learning

Reinforcement Learning with Unsupervised Auxiliary Tasks

Deep Reinforcement Learning for Closed-Loop Blood Glucose Control

Return-based Scaling: Yet Another Normalisation Trick for Deep RL

Memristor Hardware-Friendly Reinforcement Learning

1 Introduction

2 Preliminaries and Notation

3 URLB: Evaluation and Environments

3.1 Standardized of Pre-training and Fine-tuning Procedures

3.2 Evaluation

3.3 URLB Environments

4 URLB: Algorithmic Baselines for Unsupervised RL

4.1 Backbone RL Algorithm

4.2 Unsupervised RL Algorithms

5 Experiments

6 Related work

7 Conclusion

Acknowledgements

References

Checklist

Appendix A Unsupervised Reinforcement Learning Baselines

a.1 Knowledge-based Baselines

a.2 Data-based Baselines

a.3 Competence-based Baselines

Appendix B Hyper-parameters

Appendix C Per-domain Individual Results

Appendix D Finetuning Learning Curves

Appendix E Individual Numerical Results

Appendix F Compute Resources

Appendix G Intuition on Competence-based Approaches Underperform on URLB

URLB: Unsupervised Reinforcement Learning Benchmark

Authors

Related Research

Evolving Rewards to Automate Reinforcement Learning

Behavior From the Void: Unsupervised Active Pre-Training

Parrot: Data-Driven Behavioral Priors for Reinforcement Learning

Reinforcement Learning with Unsupervised Auxiliary Tasks

Deep Reinforcement Learning for Closed-Loop Blood Glucose Control

Return-based Scaling: Yet Another Normalisation Trick for Deep RL

Memristor Hardware-Friendly Reinforcement Learning

This week in AI

1 Introduction

2 Preliminaries and Notation

3 URLB: Evaluation and Environments

3.1 Standardized of Pre-training and Fine-tuning Procedures

3.2 Evaluation

3.3 URLB Environments

4 URLB: Algorithmic Baselines for Unsupervised RL

4.1 Backbone RL Algorithm

4.2 Unsupervised RL Algorithms

5 Experiments

6 Related work

7 Conclusion

Acknowledgements

References

Checklist

Appendix A Unsupervised Reinforcement Learning Baselines

a.1 Knowledge-based Baselines

a.2 Data-based Baselines

a.3 Competence-based Baselines

Appendix B Hyper-parameters

Appendix C Per-domain Individual Results

Appendix D Finetuning Learning Curves

Appendix E Individual Numerical Results

Appendix F Compute Resources

Appendix G Intuition on Competence-based Approaches Underperform on URLB