Hierarchical abstraction and curiosity-driven exploration are two common paradigms in current reinforcement learning approaches to break down difficult problems into a sequence of simpler ones and to overcome reward sparsity. However, there is a lack of approaches that combine these paradigms, and it is currently unknown whether curiosity also helps to perform the hierarchical abstraction. As a novelty and scientific contribution, we tackle this issue and develop a method that combines hierarchical reinforcement learning with curiosity. Herein, we extend a contemporary hierarchical actor-critic approach with a forward model to develop a hierarchical notion of curiosity. We demonstrate in several continuous-space environments that curiosity approximately doubles the learning performance and success rates for most of the investigated benchmarking problems.READ FULL TEXT VIEW PDF
A general problem for reinforcement learning is sparse rewards. For example, tasks as simple as drinking water involve a complex sequence of motor commands, and only upon completion of this complex sequence, a reward is provided, which destabilizes the learning of value functions. Hierarchical Reinforcement Learning (HRL) partially alleviates this issue by decomposing difficult tasks into simpler subtasks, providing additional intrinsic rewards upon completion of the subtasks. Therefore, HRL is a major step towards human-like cognition  and decision-making . There exists a considerable body of research demonstrating that hierarchical architectures provide a significant performance gain compared to non-hierarchical architectures by performing such abstractions [6, 16, 26].
However, HRL does not completely eliminate the problem of reward sparsity. By adding intrinsic rewards for achieving subtasks, it rather transforms the problem of reward sparsity into the problem of selecting the appropriate subgoals or subtasks. Learning the subgoal or subtask-selection still suffers from reward sparsity. So how can we improve the learning of subtask selection under sparse rewards?
Current reinforcement literature offers two commonly used methods for overcoming rewards sparsity that we will investigate to address this question. The first method is hindsight experience replay (HER) . The idea behind HER is to pretend in hindsight that the final state of a rollout was the goal of the rollout, regardless of whether it was actually the original one. This way, unsuccessful rollouts get rewarded by considering in hindsight that they were successful. In recent work, Levy et al.  have successfully combined HER with a hierarchical actor-critic reinforcement learning approach, demonstrating a significant performance gain for several continuous-space environments. The second method to densify rewards is curiosity. Existing curiosity-based approaches in non-hierarchical reinforcement learning (e.g. [10, 20]) provide additional rewards when the agent is surprised. Following research around Friston et al. , the notion of surprise is based on the prediction error of an agent’s internal forward model. That is, the agent is surprised when its internal prediction of the world dynamics does not coincide with its actual dynamics.
There exists a significant amount of recent approaches on hierarchical reinforcement learning (e.g. [2, 12, 13, 15, 16, 19, 26]). We are also aware of significant recent improvements in curiosity-driven non-hierarchical reinforcement learning (e.g. [5, 7, 10, 11, 20, 27]). However, despite significant evidence from Cognitive Sciences, suggesting that curiosity is a hierarchical phenomenon , there exist no functional computational models to verify this hypothesis.
In this paper, we address this lack and ask the following central research question: To what extent can we alleviate reward-sparsity and improve the learning performance of hierarchical actor-critic reinforcement learning with a hierarchical curiosity mechanism?
We address this question by extending the hierarchical actor-critic approach by Levy et al.  with a reward signal that fosters the agent’s curiosity. We extend the approach with Friston et al.’s proposal to model surprise based on prediction errors  and provide the agent with intrinsic rewards if it surprised (see Figure 1). As a novelty and scientific contribution, we are the first to present a computational model that combines curiosity with hierarchical reinforcement learning, and that considers also hindsight experience replay as an additional method to overcome reward sparsity. We refer to our method as Curious Hierarchical Actor-Critic (CHAC) and evaluate our approach in several continuous-space benchmark environments.
Our research integrates hierarchical reinforcement learning with a curiosity and surprise mechanism inspired by the principle of active inference . In the following, we provide the background of these mechanisms and methods.
Reinforcement learning (RL) involves a Markov Decision Process(MDP) to maximize the long-term expected reward. An MDP is defined as a tuple,, where is a set of states, is a set of actions, is a reward function,
is a transition probability of reaching statefrom the current state when executing action , and
is a discount factor, indicating how much the agent prefers short-term to long-term rewards. In our setting, the agent takes actions drawn from a probability distribution over action, a policy, denoted. The goal of the agent is to take actions that maximize long term expected reward. In this work, we employ the Deep Deterministic Policy Gradient (DDPG) algorithm  for the policy learning. DDPG is a model-free off-policy actor-critic algorithm, which combines the Deterministic Policy Gradient (DPG) algorithm  with Deep Q-network (DQN) . This enables agent with DDPG to work in continuous space while learning with large, non-linear function approximators more stably and efficiently. In Section 3 we define how this non-hierarchical notion of reinforcement learning is extended to the hierarchical actor-critic case.
Friston et al.  describe surprise as “the improbability of sampling some signals, under a generative model of how those signals were caused.”. Hence, curiosity can be achieved by maximizing surprise, i.e., by maximizing the probability of sampling signals that do not coincide with the predictions by the generative model [4, 8].111Note that curiosity is a broad term and there exist other rich notions of curiosity . However, for this paper we focus on the well-defined and established notion of curiosity as maximizing prediction errors.
A common method realizing this in practical reinforcement learning applications is to define a generative forward model that maps states and actions to successive states. One can then use the forward model to implement surprise as a normalized function of the error between the successive states predicted by the model and the actual successive states. This strategy has been successfully employed in several non-hierarchical reinforcement learning approaches [4, 7, 10, 11, 20, 23, 24, 27].
For example, Pathak et al.  propose an Intrinsic Curiosity Module, introducing an additional internal reward that is defined as the squared error of the predictions generated by a forward model. Similarly, Hafez et al.  implement surprise as the absolute error of a set of forward models, and Watters et al.  use the squared error as a reward signal.
The hierarchical actor-critic (HAC) approach by Levy et al.  has shown great potential in continuous-space experimentation environments. At the same time, there exists extensive research [10, 20] showing how curious agents that strive to maximize their surprise improve their learning performance. In the following, we describe how we combine both paradigms.
Hierarchical actor-critic (HAC)  is a framework that enables agents to learn a nested hierarchy of polices. It uses hindsight experience replay (HER)  to alleviate reward-sparsity. Each layer of the hierarchy learns to solve a subproblem defined by the spaces and a transition function of its layers below. The highest layer receives as input the current state and the overall extrinsic goal. High-level layers produce actions that are subgoals for the next lower level. The lowest layer produces motor commands that are executable by the agent in the environment.
Formally, we define a hierarchy of layers with each containing an actor-critic network and a replay buffer to store experiences. Here we further expand the RL setting (cf. Section 2.1) for hierarchical agents. A layer is described as a Universal Markov Decision Process (UMDP) with an additional set of goals . The UMDP, an extension of MDP with universal value function approximator (UVFA) , is a tuple containing the state space , the goal space , the action space , the transition probability function , the reward function , and the discount rate for each layer . The state space of each layer is identical to the original, namely . The produced subgoals by the policy of each layer are within , and therefore . The action space is equal to the goal space of the next lower layer, except the lowest one, thus . Only in the lowest layer, we execute the so-called primitive actions of the agent within the environment and therefore have .
HAC involves the following three kinds of state transitions that implement hindsight experience replay (HER)  in a hierarchical setting.
These are the same transitions as in the non-hierarchical HER method: after a rollout has completed, the agent pretends in hindsight that the actually achieved state was the goal state. Computationally, this is implemented by adding state transition samples with modified goals to the replay buffer. It enables the critic function to encounter at least one sparse reward after a sequence of actions. Hindsight goal transitions generalize the approximation to other regions of the state-action space.
When a high-level layer sends a subgoal to a low-level layer, it frequently happens that the low-level layer fails to achieve the subgoal. Once the low-level layer is trained better, it achieves a subgoal more often. This dynamic process slows down the learning of the high-level layer because it constantly needs to adapt to the dynamics of the low-level layer. To alleviate this issue, HAC adds additional state transitions to the replay buffer that simulate an optimal lower-level policy and, therefore, a stable low-level transition function. These additional state transitions are generated by pretending in hindsight, that that subgoal provided as action to the low-level layer has been achieved. Technically, this is implemented by replacing the successor state of the state transition with the goal. With this procedure, HAC can learn multiple levels of policies in parallel, even if the lower-level policies are not yet fully trained.
Some subgoals may just be impossible or too hard to achieve for a low-level policy. To foster the generation of subgoals that are actually achievable by the low-level layer, HAC frequently tests whether subgoals can be achieved. During these testing phases, exploration is disabled, and a penalty is given when a subgoal could not be reached. Since difficult subgoals are penalized in the beginning of the training, but not anymore when the agent is trained better, subgoal testing provides HAC with a method to automatically generate a curriculum.
To combine HAC with curiosity-based rewards, we implement a forward model based on a multi-layered perceptron that learns to predict the successive stategiven the current state and an action at time . Formally, this mapping is given as follows:
with the models parameters . An action produced by a policy of the layer (except the bottom layer, where ) at time is a subgoal for the subsequent level. We implement one forward model per layer. That is, we define a forward model not only for the primitive action in the lowest layer but also for the subgoal action in the higher layers. The learning objective for training the forward model is to minimize the prediction loss, defined as:
Similar to the approach by Pathak et al. , the forward model’s error of the layer is used to realize the curiosity-based bonus, denoted as . We calculate the mean-squared-error as follows:
The regular extrinsic rewards (from the environment) are defined in the range of , hence we need to normalize the curiosity reward resulted of Equation 3. The normalization of the curiosity reward is conducted with respect to the maximum and minimum values of the curiosity level in the whole history (stored in a buffer), and respectively, as follows:
In other words, if the prediction error is high, corresponding to high curiosity, the normalized value will be close to , otherwise, it is close to .
The total reward at time that layer receive given the extrinsic reward and the curiosity reward is controlled by the hyper-parameter as follows:
This part is crucial in determining the balance of changing the reward, since if , which is identical to HAC. We further elaborate on the different values of in Section 4.
We implement the forward model (of each hierarchical layer
) as a multilayer perceptron (MLP), receiving the concatenated current stateand action and generating a prediction for the successor state as output (cf. Equation 1). For most experiments in this paper (see Section 4), we use an MLP with 3 hidden layers of size 2048 to learn the forward model from the agent’s experiences. Experimentally, we found that this setting yields the best performance results, except for the UR5 reacher environment where we use a hidden size of 128 (see Figure 1(d)). Following Levy et al. , we also realize the actor and critic networks with MLPs of 3 hidden layers of size . Both the forward model and actor-critic are trained concurrently with the learning rate of using ADAM optimizer . After each interaction episode, samples are randomly drawn from the replay buffer for training the network parameters of all components, including the forward model. The hyper-parameters were either adapted from HAC  or fine-tuned with preliminary experiments.
We compare the performance of our framework in several goal-based environments with continuous state and action spaces. All environments provide a sparse extrinsic reward when the goal is reached. To evaluate our approach, we record the learning performance in terms of successful rollouts in relation to training rollouts. Therefore, we alternate training (with exploration using -greedy) and testing rollouts (without exploration) and measure the success rate as the average number of successful testing rollouts withing a testing batch.
To evaluate our proposal, we run experiments in following simulated environments:
Ant reacher: The ant reacher environment (see Figure 1(a)) consists of a four-legged robotic agent that must learn to walk to reach a target location. The action space is based on the joint angles of the limbs, and the observation space consists of the Cartesian locations and velocities of the body parts of the agent. The target location is random Cartesian coordinates of the agent’s torso. The yellow and pink spheres in the figure indicate the end-goal and subgoal respectively.
Ant four rooms: This environment is the same as Ant reacher, except that there are walls in the environments that the agent cannot pass (see Figure 1(b)). The walls form four rooms that are connected by passages to transition from one room to another, increasing the difficulty compared to Ant reacher.
Fetch robot reacher: This reacher environment (see Figure 1(c)) is based on an inverse kinematics model that provides a 3D continuous action space. The task of the robot is to move the gripper to a target position (indicated in the figure by the black sphere), defined in terms of Cartesian coordinates.
UR5 reacher: This environment consists of the first three DoFs (two shoulder joints and one elbow joint) of a UR5 robotic arm that must reach (feasible) random joint configurations indicated by yellow boxes in Figure 1(d). The action space is determined by the angles of the joints, and the state space consists of joint velocities angles.
Results from Figure 3 reveal significant performance gains in terms of the learning progress for all investigated environments. We first used the ant environments to determine a reasonable value for and observe that provides better results than . Hence, we used this value also for all reacher environments, i.e. Fetch and UR5.
For the ant environments, the success rates peak at around 0.4 (reacher) and 0.28 (four rooms) when using plain HAC with two layers. Adding the curiosity-based reward leads to success rates that are more than twice as high, namely around 0.95 for reacher and 0.7 for four rooms.222Note that the performance of our HAC implementation in the ant environments is lower as in the original paper by Levy et al.  because we used a smaller replay buffer due to memory limitations. Similar performance gains are achieved in the reacher environments: The Fetch reacher remains at a relatively low success rate when using plain HAC, as it cannot achieve success rates higher than 0.05. Using CHAC with improves the performance towards success rates between 0.2 and 0.3. HAC-based UR5 achieves similarly low success rates (less than 0.05), while the CHAC-based UR5 achieves a better performance, up to 0.12.
Curiosity and the ability to perform problem-solving in a hierarchical manner are two important features of human-level problem-solving and learning. As a novelty and scientific contribution, this paper presents the first computational approach that combines both features by extending hierarchical actor-critic reinforcement learning with a curiosity-enabled reward function. The level of curiosity is modeled by the prediction error of learnable forward models included in all hierarchical layers. Our experimental results provide significant evidence that curiosity improves hierarchical problem-solving. Specifically, using the success rate as evaluation metrics, we show that curiosity more than doubles the learning performance for the proposed hierarchical architecture and benchmark problems.
Thirty-First AAAI Conference on Artificial Intelligence, pp. 1726–1734. External Links: Cited by: §1.
International Conference on Machine Learning (ICML), External Links: Cited by: §1.