Recent successes in challenging computer games like StarCraft 2 (Vinyals et al., 2019) have raised interest in Multi-agent Reinforcement Learning (MARL). Here single units are modeled as individual agents, for example, in the recent open source StarCraft Multi-agent Challenge (SMAC, Samvelyan et al., 2019). In comparison to single-agent deep RL, MARL faces some unique challenges, in particular decentralization and coordination. In this paper we investigate the equally challenging problem of directed exploration.
Directed exploration in single-agent deep RL still poses many open questions, like how to generalize visitation counts
in large input spaces and how to change the exploration policy quickly towards newly discovered states. However, so far, there has been little work on exploration for deep MARL. Exploration in MARL differs from the single-agent setting in some challenging ways: (i) counting visitations of state-action pairs is infeasible for many agents, due to the large joint-action space; (ii) as unexpected outcomes can be caused by multiple agents, it is not clear which agent’s action should be reinforced; and (iii) partial observability decreases the reliability of count estimates.
Decentralization is required in MARL when the agents cannot communicate directly. Moreover, a centralized control policy is often infeasible, as the joint-action space grows exponentially in the number of agents. In line with SMAC, we consider the case where the state of the system is only partially observable by each agent, although during training the global state may be available. This is called centralized training with decentralized execution (Foerster et al., 2016). This paper focuses on the simplest value-based algorithm in this class, a variant of Independent Q-learning (IQL, Tan, 1993), where each agent acts on partial observations and assumes the other agents’ decisions are an unobserved, stationary part of the environment.
However, these simple decentralized agents lack coordination. Take the example of two predators, who trapped their prey in a corner. To catch it, both must attack simultaneously, as it will escape if only one predator attacks. From the perspective of each predator, the reward for attacking depends on the actions of the other. When the punishment for letting the prey escape is larger than the reward for catching it, neither agent will learn the optimal strategy independently. There are multiple methods using centralized training to mitigate this effect for decentralized policies, e.g., multi-agent credit assignment (COMA, Foerster et al., 2018) and bootstrapping with an approximation of the joint Q-value function. For example, Value Decomposition Networks (VDN, Sunehag et al., 2018) optimize the joint Q-value, but restrict the Q-value function to a sum of individual agents’ utilities. QMIX (Rashid et al., 2018) goes one step further and mixes the agents’ utilities with a non-linear hyper-network, that conditions on the global state. Both approaches can execute the decentralized learned policy by maximizing each agent’s utility. However, all the above techniques use relatively simple -greedy exploration.
Value-based algorithms that explore provably efficiently in a tabular setting (e.g. Jin et al., 2018) rely on optimism in the face of uncertainty. There are two major lines of research in the literature: to use intrinsic reward
to over-estimate uncertain state-action values or to use Thompson sampling from aBayesian posterior of the value function. Various techniques have been proposed to estimate the uncertainty of state-action values (summarized in Appendix B
), but whether it is used as an intrinsic reward or as the standard deviation of a Gaussian posterior, most works converge at an estimate that is supposed to scale with, where counts how often state and action have been observed at time . For large input spaces, however, these estimates are rough approximations of visitation counts and the resulting uncertainties are highly unreliable.
This paper investigates estimated intrinsic reward for decentralized IQL agents. We evaluate the variance of linear functions (O’Donoghue et al., 2018) as an uncertainty estimate in a novel predator-prey task that emphasizes exploration. We observe empirically that the intrinsic reward accelerates learning, but remains inherently unreliable, which prevents finding the optimal solution. To learn reliable decentralized policies in the face of unreliable reward, we propose to share control with a second agent that is discarded after training and can thus be centralized. Only the central agent receives intrinsic rewards, which prevents the decentralized agents from being distracted, while still improving their exploratory behavior. We show that this new approach to MARL exploration drastically speeds up learning of the decentralized agents, while converging to the optimal solution. This novel framework is general and can be applied to different estimators of intrinsic reward and/or off-policy MARL algorithms like VDN and QMIX.
We restrict ourselves to cooperative tasks, modeled as a Dec-POMDP (Oliehoek & Amato, 2016), that is, a tuple . The global state of the system is denoted as , and each of the agents chooses actions , which together form the joint action . After executing joint action in state at discrete time step , the next state is drawn from transition kernel , and a reward is determined by the reward function . While a centralized joint policy can choose joint actions conditioned on the current state , a decentralized agent policy draws only agent ’s action , based on the agent’s current trajectory of past actions and observations , which are drawn from the agent’s observation kernel , that is, . Execution of a joint policy yields an episode with return at time . Our goal is to find a decentralized joint policy , which maximizes the expected return for each observed trajectory. Partial observability of the policy can significantly slow down learning and we allow access to the global state during training, that is, centralized training for decentralized execution.
2.1 Independent Q-learning (IQL)
Independent Q-learning (Tan, 1993) approaches this goal by defining the state-action value function of agent as the expectation of the return, following policy from an observed trajectory , that is, . As in Q-learning (Watkins & Dayan, 1992), IQL assumes that the greedy policy, which chooses the action with the largest corresponding value, maximizes the expected return in each state. Note that this is not true, as the expected return also depends on the other agents’ behavior, which can introduce non-stationarity. That being said, IQL appears to be stable in practice and works quite well in most tasks.
We use a neural network with one head for each discrete action to approximate the value function(as in DQN, Mnih et al., 2015). For IQL, we learn a function111To condition on trajectories , we follow Hausknecht & Stone (2015) and use a recurrent network of GRU units (Chung et al., 2014), which condition on their hidden state, the last action , the current observation and the agent id . , parameterized by , with gradient-descend on the expected squared Bellman error
where are the parameters of a target network, which are replaced with a copy of the current parameters from time to time to improve stability. The expectation is approximated by drawing mini-batches of trajectories from an experience replay buffer (Lin, 1992). We also use double Q-learning (Hasselt et al., 2016) to further improve stability and share the parameters of all agents’ value functions for better generalization (similar to QMIX, Rashid et al., 2018).
2.2 Intrinsic Reward
We employ a local uncertainty measure introduced by O’Donoghue et al. (2018)
. The variance of a linear regression, i.e., fitting a linear functionto a fixed set of state-action pairs and random labels
with a Gaussian distribution, is
As each head of the IQL value function is a linear function of the last network layer, that is, the hidden state of a GRU, O’Donoghue et al. (2018) suggest to use as a measure of local uncertainty. This choice of intrinsic reward is somewhat justified, as for one-hot coded , the intrinsic reward is , which corresponds to the tabular case (e.g., in Jin et al., 2018) with scaling factor .
Intrinsic reward based on estimated uncertainties rarely reflects the precise visitation counts. We investigate how such unreliable intrinsic reward can still reliably improve exploration of decentralized agents. The main idea is to introduce a second controller during training that can be discarded afterwards. This joint agent is intrinsically rewarded and can thus explore the environment in a directed fashion. In principle, the agent’s policy could be learned by many algorithms. As it is only active during training, though, we propose a centralized agent, which conditions on the more informative global state. Most importantly, we train simultaneously the decentralized agents, which will be later deployed for execution, on the same replay buffer. These can utilize any decentralized off-policy learning algorithm, but we focus in the following on IQL for simplicity.
3.1 A Central MARL Agent
The large action spaces in MARL make individual heads for each joint action on value functions infeasible in the face of many agents. Maximizing a value function that conditions on all agents’ actions, on the other hand, has to be evaluated for all as well, which can be prohibitively expensive. Instead, we use the architecture of a COMA critic, which Foerster et al. (2018) introduced in the context of a policy gradient method. They define an agent-specific joint-value function , parameterized by , which has a head for each of ’s actions and conditions on the global state, all other agents’ actions and agent ’s the previous action :
Instead of maximizing this function w.r.t. the joint action , we propose here to approximate a local maximum by iteratively choosing the that maximizes each individual agent’s , and using it for in the next iteration of the maximization. In practice, we initialize with the greedy actions of the decentralized IQL agents and then perform this iterative local maximization (denoted ) for a small number of iterations. As in IQL, agents share parameters .
During exploration it is important that the sampling policy changes in response to newly discovered states. This change is imposed by intrinsic reward, which must therefore be transported quickly to earlier states in the episode. We use a implementation (Watkins, 1989) to accelerate this transport, but do not cut the traces after exploratory steps. This improves transport, but also introduces non-stationary targets. Our training procedure performs a parameter update after each episode and we compute the targets backwards:
where denotes the target network and . The loss
is minimized by gradient descent, where the expectation is approximated with the same mini-batches as in IQL and the same stabilization techniques are used as well.
3.2 Intrinsic Reward Revisited
Intrinsic rewards, as defined in Section 2.2, induce three challenges for collaborative MARL: dependence on joint actions, collaborative rewards, and evolving parameters.
First, estimating (2) is infeasible for MARL, as we would have to estimate one correlation matrix for each joint action . Instead of estimating a measure that depends on counting , we propose here to estimate one based on the count
. Although only a heuristic, this approach works in arbitrary large action spaces.
Second, in collaborative tasks all agents should receive the same reward to avoid diverging incentives. However, in MARL each agent estimates a different uncertainty, based on ’s observations and/or other inputs. As the interaction of all agents could have contributed to each agent’s uncertainty, it is unclear how to reconcile diverging estimates. We propose to use the largest uncertainty as intrinsic reward for all agents, to consider this potential interaction.
Third, the agents’ value function parameters continually change, particularly in the beginning of training when exploration is most important. The same inputs yield different representations at different times , and estimates with (2
) therefore become outdated after a while. To reflect this change, we propose to use an exponentially decaying average of the inverted matrix, where denotes the value function’s inputs at time and is a small decay constant. As the resulting uncertainty never decays to 0, we also introduce a bias and discard negative intrinsic rewards. The bias can be constant or an average over past uncertainties to only reward states with above average novelty.
The resulting collaborative intrinsic reward is:
3.3 Independent Centrally-assisted Q-learning (ICQL)
The intrinsically rewarded central agent samples in our training framework episodes, while the decentralized (here IQL) agents are simultaneously trained on the shared replay buffer. This improves exploration in two ways: (i) although the decentralized agents still benefit from the exploration that is induced by (potentially unreliable) intrinsic reward, their final policies are exclusively based on environmental rewards, and (ii) the central agent conditions on the true state of the system, which includes information that may not be observable by the agents.
However, sampling only with the central agent yields an out-of-distribution problem for the IQL agents: a single deviating decision can induce a trajectory that has never been seen during training. We therefore share the sampling process by deciding randomly at the start of an episode whether the IQL agents or the central agent takes control. The resulting behavior appears quite stable for different probabilities of that choice and we chose 50% for simplicity.
We extend a common partially observable predator-prey task in a grid-world
to make it challenging from an exploration perspective,
as preliminary experiments have shown that the
original task does not require directed
exploration of the state space.222 In the original task, the prey moves randomly
and the states in which the agents meet it
are almost uniformly distributed.
This provides sufficient exploration to find an optimal policy
and directed exploration is unnecessary.
In the original task, the prey moves randomly and the states in which the agents meet it are almost uniformly distributed. This provides sufficient exploration to find an optimal policy and directed exploration is unnecessary.We train 4 agents, with agent-centric observations, to hunt a mountain and a valley prey. Prey moves randomly in a bounded grid-world of height 41 and width 10. To simulate a mountain, both agents and valley prey do not execute 50% of all ‘up’ actions, and mountain prey does not execute 50% of all ‘down’ actions. Valley prey spawn randomly on the lowest row, mountain prey on the highest row, and agents on middle row. An episode ends either when one of the prey is caught, that is, when agents/boundaries surround it on all sides, or after 100 steps. Only capturing yields reward, 5 for the valley and 10 for the mountain prey. Exploring the state space helps to find the mountain prey without getting distracted by the valley prey.
Figure 1 shows training and test performance333We implemented all algorithms in the PyMarl framework (Samvelyan et al., 2019). Details can be found in Appendix A. for three algorithms: the original IQL (red, Section 2.1), IQL with intrinsic reward based on the agents’ last layers (green, Section 3.2) and our novel centrally-assisted exploration framework ICQL (blue, Section 3.3), where the intrinsic reward is based on the last layer of the central agents’ value functions.
The destabilizing effect of unreliable intrinsic reward on IQL can be seen in the green IQL (=1) curve: it speeds up learning, but also prevents the agents from finding the optimal policy (visible both in training and test plots). The bonus provides incentives for exploration, but also appears to distract the agents when their policy should converge.
Our ICQL framework (blue) learns even faster, but demonstrates different behavior during training and testing. On the one hand, one can see the same sub-optimal behavior during training (left plot), which executes 50% of the episodes with the intrinsically-rewarded central agent and 50% with decentralized agents trained simultaneously. On the other hand, the test performance (right plot) of the decentralized agents shows the same improved learning, but none of the instabilities once the mountain prey has been found.
We conclude that intrinsic reward is both a blessing and a curse for MARL settings. We have shown that even unreliable reward can improve directed exploration, but also introduces detracting incentives. Our novel ICQL framework for centrally-assisted exploration appears to stabilize learning and further speeds up training, most likely by exploiting access to the true state.
In future work, we will further evaluate how the framework performs with different decentralized learning algorithms, like VDN and QMIX, and employ other uncertainty estimates for intrinsic rewards. We also want to investigate the effect of adaptive biases and apply our method to StarCraft micromanagement tasks (SMAC, Samvelyan et al., 2019).
The authors would like to thank Jakob Förster, Gregory Farquhar and Christian Schroeder de Witt for fruitful discussions about decentralization and exploration in MARL. This project has received funding from the European Research Council (ERC), under the European Union’s Horizon 2020 research and innovation programme (grant agreement number 637713), and a grant of the EPSRC (EP/M508111/1, EP/N509711/1). The experiments were made possible by a generous equipment grant from NVIDIA.
- Bellemare et al. (2016) Bellemare, M. G., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., and Munos, R. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems (NIPS) 29, pp. 1471–1479, 2016.
- Burda et al. (2019) Burda, Y., Edwards, H., Pathak, D., Storkey, A., Darrell, T., and Efros, A. A. Large-scale study of curiosity-driven learning. In International Conference on Learning Representations (ICLR), 2019.
- Chung et al. (2014) Chung, J., Gulcehre, C., Cho, K., and Bengio, Y. Empirical evaluation of gated recurrent neural networks on sequence modeling. In NIPS Workshop on Deep Learning, 2014. URL http://arxiv.org/abs/1412.3555.
- Foerster et al. (2016) Foerster, J., Assael, I. A., de Freitas, N., and Whiteson, S. Learning to communicate with deep multi-agent reinforcement learning. In Advances in Neural Information Processing Systems (NIPS) 29, pp. 2137–2145. 2016.
Foerster et al. (2018)
Foerster, J. N., Farquhar, G., Afouras, T., Nardelli, N., and Whiteson, S.
Counterfactual multi-agent policy gradients.
Proceedings of the 15th AAAI Conference on Artificial Intelligence, pp. 2974–2982, 2018.
- Fortunato et al. (2018) Fortunato, M., Azar, M. G., Piot, B., Menick, J., Hessel, M., Osband, I., Graves, A., Mnih, V., Munos, R., Hassabis, D., Pietquin, O., Blundell, C., and Legg, S. Noisy networks for exploration. In International Conference on Learning Representations (ICLR), 2018.
- Gal et al. (2017) Gal, Y., Hron, J., and Kendall, A. Concrete dropout. In Advances in Neural Information Processing Systems (NIPS), pp. 3584–3593, 2017.
- Hasselt et al. (2016) Hasselt, H. v., Guez, A., and Silver, D. Deep reinforcement learning with double q-learning. In Proceedings of the 13th AAAI Conference on Artificial Intelligence, pp. 2094–2100, 2016.
- Hausknecht & Stone (2015) Hausknecht, M. J. and Stone, P. Deep recurrent q-learning for partially observable mdps. In 2015 AAAI Fall Symposia, pp. 29–37, 2015. URL http://www.aaai.org/ocs/index.php/FSS/FSS15/paper/view/11673.
- Jaques et al. (2018) Jaques, N., Lazaridou, A., Hughes, E., Gülçehre, Ç., Ortega, P. A., Strouse, D., Leibo, J. Z., and de Freitas, N. Intrinsic social motivation via causal influence in multi-agent RL. CoRR, abs/1810.08647, 2018. URL https://arxiv.org/abs/1810.08647.
- Jin et al. (2018) Jin, C., Allen-Zhu, Z., Bubeck, S., and Jordan, M. I. Is Q-learning provably efficient? In Advances in Neural Information Processing Systems (NeurIPS) 31, pp. 4863–4873. 2018.
- Leibo et al. (2019) Leibo, J. Z., Hughes, E., Lanctot, M., and Graepel, T. Autocurricula and the emergence of innovation from social interaction: A manifesto for multi-agent intelligence research. CoRR, abs/1903.00742, 2019. URL http://arxiv.org/abs/1903.00742.
- Lin (1992) Lin, L.-J. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine Learning, 8(3):293–321, 1992.
- Mnih et al. (2015) Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., and Hassabis, D. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, February 2015.
- O’Donoghue et al. (2018) O’Donoghue, B., Osband, I., Munos, R., and Mnih, V. The uncertainty Bellman equation and exploration. In Proceedings of the 35th International Conference on Machine Learning (ICML), pp. 3836–3845, 2018.
- Oliehoek & Amato (2016) Oliehoek, F. A. and Amato, C. A concise introduction to decentralized POMDPs. Springer Publishing Company, Incorporated, 1st edition, 2016. ISBN 3319289276, 9783319289274.
- Osband et al. (2016) Osband, I., Van Roy, B., and Wen, Z. Generalization and exploration via randomized value functions. In Proceedings of the 33rd International Conference on International Conference on Machine Learning (ICML), pp. 2377–2386, 2016.
- Osband et al. (2018) Osband, I., Aslanides, J., and Cassirer, A. Randomized prior functions for deep reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS) 31, pp. 8617–8629. 2018.
- Ostrovski et al. (2017) Ostrovski, G., Bellemare, M. G., van den Oord, A., and Munos, R. Count-based exploration with neural density models. In Proceedings of the 34th International Conference on Machine Learning (ICML), pp. 2721–2730, 2017.
- Pathak et al. (2017) Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T. Curiosity-driven exploration by self-supervised prediction. In Proceedings of the 34th International Conference on Machine Learning (ICML), 2017.
- Plappert et al. (2018) Plappert, M., Houthooft, R., Dhariwal, P., Sidor, S., Chen, R. Y., Chen, X., Asfour, T., Abbeel, P., and Andrychowicz, M. Parameter space noise for exploration. In International Conference on Learning Representations (ICLR), 2018.
- Rashid et al. (2018) Rashid, T., Samvelyan, M., de Witt, C. S., Farquhar, G., Foerster, J. N., and Whiteson, S. QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning (ICML), pp. 4292–4301, 2018.
- Roderick et al. (2018) Roderick, M., Grimm, C., and Tellex, S. Deep abstract Q-networks. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS), pp. 131–138, 2018.
- Samvelyan et al. (2019) Samvelyan, M., Rashid, T., de Witt, C. S., Farquhar, G., Nardelli, N., Rudner, T. G. J., Hung, C., Torr, P. H. S., Foerster, J. N., and Whiteson, S. The StarCraft multi-agent challenge. CoRR, abs/1902.04043, 2019. URL https://arxiv.org/abs/1902.04043.
- Sunehag et al. (2018) Sunehag, P., Lever, G., Gruslys, A., Czarnecki, W. M., Zambaldi, V., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, J. Z., Tuyls, K., and Graepel, T. Value-decomposition networks for cooperative multi-agent learning based on team reward. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS), pp. 2085–2087, 2018.
- Tan (1993) Tan, M. Multi-agent reinforcement learning: Independent vs. cooperative agents. In In Proceedings of the Tenth International Conference on Machine Learning (ICML), pp. 330–337, 1993.
- Tang et al. (2017) Tang, H., Houthooft, R., Foote, D., Stooke, A., Xi Chen, O., Duan, Y., Schulman, J., DeTurck, F., and Abbeel, P. #Exploration: A study of count-based exploration for deep reinforcement learning. In Advances in Neural Information Processing Systems (NIPS) 30, pp. 2753–2762. 2017.
- van den Oord et al. (2016) van den Oord, A., Kalchbrenner, N., Espeholt, L., kavukcuoglu, k., Vinyals, O., and Graves, A. Conditional image generation with PixelCNN decoders. In Advances in Neural Information Processing Systems (NIPS) 29, pp. 4790–4798. 2016.
Vinyals et al. (2019)
Vinyals, O., Babuschkin, I., Chung, J., Mathieu, M., Jaderberg, M., Czarnecki,
W., Dudzik, A., Huang, A., Georgiev, P., ichard Powell, Ewalds, T., Horgan,
D., Kroiss, M., Danihelka, I., Agapiou, J., Oh, J., Dalibard, V., Choi, D.,
Sifre, L., Sulsky, Y., Vezhnevets, S., Molloy, J., Cai, T., Budden, D.,
Paine, T., Gulcehre, C., Wang, Z., Pfaff, T., Pohlen, T., Wu, Y., Yogatama,
D., Cohen, J., McKinney, K., Smith, O., Schaul, T., Lillicrap, T., Apps, C.,
Kavukcuoglu, K., Hassabis, D., and Silver, D.
AlphaStar: Mastering the real-time strategy game StarCraft
Deepmind blog, accessed 04/16/2019,
- Watkins & Dayan (1992) Watkins, C. and Dayan, P. Q-learning. Machine Learning, 8:279–292, 1992.
- Watkins (1989) Watkins, C. J. C. H. Learning from delayed rewards. PhD thesis, Cambridge University, 1989.
- Zheng & Yue (2018) Zheng, S. and Yue, Y. Structured exploration via hierarchical variational policy networks, 2018. URL https://openreview.net/forum?id=HyunpgbR-.
Appendix A Training details
We implemented all algorithms in the PyMARL framework (Samvelyan et al., 2019)
, where we used RMSprop with learning rate 0.0005,
, batch size 32 and a replay buffer holding the last 200 episodes. Decentralized agents had a hidden layer of 64 GRU cells, sandwiched between 2 feed-forward layers, and central agents had 3 feed-forward layers with 128 hidden neurons each. The target network was updated every 200 episodes and we used-greedy exploration, which decayed within 20,000 steps. Intrinsic reward had magnitude , a decay constant and constant bias . ICQL approximated the local maximum with one iteration.
Appendix B Related Work
In this paper we focus on intrinsic reward for exploration (in difference to pure curiosity, Burda et al., 2019). Here the uncertainty is often derived from to the prediction quality after training on past trajectories. For example, pseudo-counts are based on the reconstruction probability of visual observations (Bellemare et al., 2016; Ostrovski et al., 2017), using a PixelCNN (van den Oord et al., 2016). Alternatively, Tang et al. (2017) count visitations using a hash function on a random linear projection. Furthermore, Pathak et al. (2017) use the predictability of the observed transition as intrinsic reward signal, and Roderick et al. (2018) reduce uncertainty with prior knowledge over state abstractions.
In the context of Bayesian posteriors, the uncertainty has been estimated from an ensemble of value functions, with optional bootstrapping techniques (Osband et al., 2016, 2018). Alternatively, Noisy Nets (Fortunato et al., 2018; Plappert et al., 2018) sample a value function for each episode from a diagonal Gaussian posterior over the parameters of the neural network. Similarly, Gal et al. (2017) suggested to use Concrete Dropout to estimate the posterior for model-based RL. To include the uncertainty of future reward, O’Donoghue et al. (2018) proposed the Uncertainty Bellman Equation (UBE), which propagates the ‘local uncertainty’ of future decisions with a Bellman operator.
For MARL, Zheng & Yue (2018) proposed to coordinate exploration by sharing latent variables, drawn from a learned distribution. Jaques et al. (2018) focuses on social motivations of competitive agents and Leibo et al. (2019) describes exploration as an auto-curriculum generated by competing species of agents.