I Introduction††Correspondence to email@example.com
Automated learning of robot control from high-dimensional sensor data – such as camera images – is a long standing goal of robotics research. In recent years, a lot of progress towards this challenge has been made by employing machine learning methods for real robot learning. Examples of such successes include: learning grasp success predictors from raw image data and the growing list of successful real-world applications of Reinforcement Learning (RL) such as learning to flip a pancake , learning to fly a model helicopter , in-hand manipulation of objects using a Shadow Robotics hand  and the Ball-in-a-Cup task with a robot arm 
. However, most of these approaches have either used extensive pre-training in simulation, used expert demonstrations to speed-up learning, or did not consider learning from raw sensory input (e.g. images). Most of these limitations are due to the fact that only a limited amount of data is available when learning from a real robot system, prohibiting the naive application of expressive control policies (such as deep neural networks learned from vision).
One common strategy for minimizing the need for large amounts of data is to carefully curate the information used to represent the state of the robotic system (from which an optimal action should be predicted). Defining the state of the robot via expressive features – e.g. positions, orientations and velocities of the robotic system and considered objects in a scene – can drastically simplify the learning problem since: i) the learned control policies require much fewer parameters to represent a solution, and ii) the learned policies do not need to learn to recognize objects of interest from images (and conversely learn to explicitly ignore spurious signals such as distractors in the visual scene).
While the above makes a compelling case for using feature based representations when learning in the real world, feature-extraction incurs a significant overhead in the experimental setup; as computing meaningful features from real-world sensor data can be difficult, computationally expensive and requires extensive sensor arrays and pre-processing pipelines. Furthermore, even when such a feature extraction pipeline is available during learning, it is often desirable to obtain a controller that – once learned successfully – can be deployed with a minimal setup (e.g. in our experiments below an object tracking system is required during training but not afterwards).
In this paper, we develop a method that can simultaneously learn control policies from both feature-based representation and raw vision input in the real-world – resulting in controllers that can afterwards be deployed on a real robot using two off-the-shelf cameras. To achieve this we extend the multi-task RL framework of Scheduled Auxiliary Control (SAC-X)  to scenarios where the state-space for different tasks is described by different modalities. We base our approach on the following assumptions:
An extended set of observations is available to the agent during training time, which includes raw sensor data (such as images), proprioceptive features (such as joint angles), and “expensive” auxiliary features given via an object tracking system.
At test time, the “expensive” auxiliary features are unavailable, and the policy must rely only on raw sensors data and proprioceptive information.
Control policies learned with features will converge faster than policies learned directly from raw-sensor data, as shown in Figure 6. And hence provide a useful guiding signal for learning vision based policies.
A subset of the observations provided to the learning system forms a state description for the MDP of the RL problem we want to tackle (i.e. given this subset and a control action it is possible to predict the next state).
Starting from these assumptions we develop a method with the following properties i) like SAC-X our approach allows us to jointly learn a control policy for one main task and multiple auxiliary tasks, ii) our method allows for joint learning of vision and feature based policies for all tasks, iii) through a special neural-network architecture, all learned policies internally share a set of features (i.e. they share hidden layers in the network) enabling accelerated learning, and iv) by executing feature based skills during training the convergence of vision based control policies can be improved.
While the presented approach is general, we focus on one concrete application: solving the Ball-in-a-Cup task using a robotic arm. This task requires a robot arm to swing-up a ball attached to a string and catch it in a cup (c.f. Figures 2 and 5 for a visualization). We show, in simulation, that our method significantly accelerates learning when compared to learning from raw image data without auxiliary features like object positions. Furthermore, we demonstrate that our method is capable of learning the task on real hardware. Compared to previous solutions to the Ball-in-a-Cup task , our method does neither require imitation data nor a specialized policy class and can learn the task in approximately 28 hours of training111Training time ignores the time taken for the robot to automatically reset the environment. Observed overhead of 20-30% due to resets and entanglement. on a single robot.
Ii Related Work
Our method can be seen as an extension of a long line of work on aiding exploration in Reinforcement Learning with auxiliary tasks (defined via different reward functions). Among these is the idea of General value functions, as coined in work on learning with pseudo-rewards associated with random goals defined based on sensorimotor datastreams . This idea was further extended to a Deep RL setting in work on Universal Value Function Approximators (UVFA) , which aims to learn a goal conditioned action-value function. In a further extension, Hindsight Experience Replay (HER)  introduced the idea of computing rewards for different goals “in hindsight” by treating the end of a recorded trajectory as the intended goal. Finally, Scheduled Auxiliary Control (SAC-X)  – the method on which we base our paper – aims to simultaneously learn so called intention-policies for several, semantically grounded, auxiliary tasks and use each intention to drive exploration for all other tasks.
Just like SAC-X, our approach can also be related to curriculum learning approaches (see e.g. Bengio et al.  for a description and Heess et al.  for an RL application). In particular, our experiments utilize multiple reward functions with varying amounts of shaping and difficulty of success. While many curriculum learning approaches invent new tasks based on internal metrics such as current task performance [27, 8, 14] we here restrict ourselves to a fixed set of pre-specified tasks. These algorithms also often employ methods for intelligently scheduling tasks to learn based on learning progress. We here use simple uniform random selection of tasks. Each task is executed for a fixed number of steps before a new task is chosen.
While numerous works exist that consider RL in multi-task settings [13, 21, 7] these typically focus on varying rewards and environment behaviors but expect a fully shared observation space. Work on learning with varying observations across tasks is more scarce (we refer to Espeholt et al.  for a recent success). Among these the asymmetric actor-critic technique [22, 20] is closest to our approach. There the authors experiment with the idea of providing additional state-information (that is only available at training time) to a learned action-value function (while withholding said information from the policy) to ease credit assignment. Our approach generalizes this idea to using a different definition of state not only for the actor and critic, but also across different tasks – while still learning a single policy network. We further expand on this interesting relation in Section III-C.
Our work also has loose connections to ideas from behavioral cloning [1, 24], a simple form of imitation learning where an agent attempts to replicate another agent’s policy. Since we duplicate tasks for feature-based and vision-based policies (i.e. the tasks obtain the same reward and differ only in their state-spaces), the feature-based policies can be seen as generating demonstration data for the vision based policies (or vice versa) in our experiments. While no direct behavior cloning is performed, transitions generated from one policy are used for off-policy learning of other policies.
In this paper we test and demonstrate our technique on the dynamic Ball-in-a-Cup task. Robot learning frameworks that utilize RL have been applied to many different robots and many different tasks, including recent successes such as picking up simple objects , manipulating objects in hand , and manipulating a valve . However, many of these recent works focus on mostly static tasks, such as object manipulation. Dynamic tasks have additional difficulties compared to static tasks, e.g. the importance of timing and the difficulty of reaching (and staying in) unstable regimes of the robots configuration-space. Nonetheless, robot learning for dynamic tasks has been investigated in the past including pancake flipping , walking on unstable surfaces , and even the Ball-in-a-Cup task [6, 15]. Chiappa et al.  complete the Ball-in-a-Cup task with a simulated robot arm using motion templates generated from human demonstration data. More directly related, Kober and Peters  were able to solve the Ball-in-a-Cup task on a real robot, but their technique required imitation learning through kinesthetic teaching, reward shaping, a hand-designed policy class (in the form of Dynamic Movement Primitives (DMP)) and expressive features to form the state-space. In this work, we are able to learn a final policy that can solve the same task from raw image data and proprioceptive information.
Finally, like SAC-X, our method learns a policy using an off-policy RL algorithm. We specifically utilize SVG-0 with retrace [11, 19]. However, other off-policy RL methods such as MPO or soft-actor critic could be utilized [3, 9, 10]. The only requirement is that the algorithm is off-policy so that the policy for one task can learn from data generated by other behaviour policies.
Iii Simultaneous Reinforcement Learning from Features and Pixels
Iii-a Background and Notation
In this work we consider the standard Reinforcement Learning (RL) setting as described by a Markov Decision Process (MDP). The MDPconsists of continuous states and actions , together with an associated transition distribution
which specifies the probability of transitioning from stateto under action . Given a reward function and the discount factor the goal of learning is to optimize for the parametric policy with parameters (which specifies a distribution over actions) that maximizes the cumulative discounted reward , where is the initial state distribution. This maximization problem can alternatively be defined as
where we define the action-value function as
We are, however, not only interested in learning a single policy for the main task MDP but, alongside, want to learn policies for a given set of auxiliary rewards . Following the notation from the Scheduled Auxiliary Control (SAC-X) framework  paper we thus define a set of auxiliary MDPs , that share the state-action space and the transition dynamics with but differ in their reward definition (e.g. MDP uses reward respectively). We then aim to learn a policy for each such auxiliary task – denoting them with
respectively. We refer to these per-task policies as intentions in the following – as their goal is to entice the agent to intently explore interesting regions of the state-action space. To collect data for this learning process we randomly choose among one of the intentions in each episode and execute it on the robot. During execution a reward vectoris collected (with each component denoting one of the tasks). Such a random execution of intentions was referred to as SAC-U in . The data collected during execution is then sent to a replay buffer from which policies can be learned via off-policy learning – we defer a description of this learning process to the next section, in which we extend the SAC-X setting to varying state spaces.
Iii-B Scheduled Auxiliary Control with Varying State Spaces
As described above the SAC-X paper introduced the concept of training simultaneously on multiple tasks, with each task having its own reward function. We now describe how this idea can be extended to a scenario in which each task is potentially associated with a different state-space. Such a learning setup is useful if one wants to learn a final policy that uses only a subset of the available observations but wants to make use of all available information at training time. In our experiments, for example, we aim to learn a policy for the Ball-in-a-Cup task from vision and proprioceptive information only (i.e. should be conditioned only on images from two cameras as well as the robot joint positions and velocities and internal state necessary to make the task Markovian). During training, however, we want to exploit the fact that additional state information is available (e.g. we need to be able to track the ball in the Ball-in-a-Cup setup to calculate a reward signal).
The idea, then, is to simultaneously train both feature-based and vision-based intention policies – instead of training from scratch from raw-image data only. Ideally, the feature-based intentions will learn to solve the task quickly – due to the lower dimensionality of the state-space and the more informative features – and then generate meaningful experience data for the vision based intentions. This can be seen as a form of curriculum learning.
Crucially, the additional feature-based intentions can be removed after training, leaving us with the “raw-sensor” based policy that we desired – eliminating the need for the expensive and carefully calibrated feature detection setup at execution time.
To describe this process more formally: Let be the set of all measurable signals at time that make up the (potentially over specified) full state during training, i.e.:
where are the observed measurements at . For example, for the Ball-in-a-Cup task the measurements include: proprioceptive features consisting of joint positions and velocities, features computed via a Vicon tracking system  (such as ball position and velocity), and finally “raw” sensor data (e.g. camera images). In a standard RL setup a minimal sub-set of the components of would be used to construct the MDP state . For example, for the Ball-in-a-Cup task such a sub-set could consist of proprioceptive features together with Vicon features (disregarding vision information). Instead of committing to a single state-description we consider the case in which can be decomposed into multiple, non-overlapping subsets of observations:
In our experiments we use three such separate feature sets that we call (containing robot joint position, velocities, previous actions, and additional internal robot state data), (containing information about the ball position and velocity and cup position, orientation, and velocity as computed based on Vicon tracking) and (the images collected from two cameras setup in the workspace), respectively.
We then associate a fixed masking or filter-vector with each of these newly constructed sets given by the index vector
where the indicator function is defined on a per-task basis and returns 1 if this state set should be enabled for task and 0 otherwise. To simplify the presentation (but without loss of generality) we will restrict the following derivation to the concrete state-feature set – as mentioned above and further described in Table I. Given these definitions, we can perform learning using the following policy evaluation and policy improvement steps. We first define two filter vectors for each task, one used for learning an action-value critic – referred to as – and one for optimizing the policy – referred to as . For the rest of the paper we assume that for all tasks both filter masks enable sufficient observations to form a Markov state (i.e. they enable together with either or ).
We then construct the learned action-value functions for all with parameters
as the following feed-forward neural network. Let
where consists of the concatenated output of three function approximators (here feed-forward neural networks with parameters and respectively) applied to the state-features from each feature group, denotes element wise multiplication, subscripts denote element (in case of ) and row-access (in case of ) respectively. denotes the final output layer and the full parameters of the network are given by . That is, depending on whether or not a given set of features is activated for the Q-function of task the output of the corresponding -network is passed forward. This way, the g-networks can be learned jointly across multiple tasks, while the output layers (given by the function ) specialize on a per task basis, where a task is a combination of a reward function and a state-space. Specializing outputs per task, rather than per reward, allows the network to adjust the policy to differences in the observation types such as parts of the environment that are noisy for one observation type but not another. Learning of all action-value functions is performed jointly (analogously to the procedure from Riedmiller et al. ) by regressing the retrace  targets:
where are the policy probabilities under which the trajectory was recorded and denote the parameters of a target network that are copied from the current parameters every optimization steps to stabilize learning .
Analogously to the learned action-value functions we define the policy networks as:
denotes the multivariate normal distribution with meanand diagonal covariance matrix and the parameters of the policy are optimized by following the reparameterized gradient of based on the learned Q-functions:
where the second term denotes additional entropy regularization of the policy (with weight ). We refer to Riedmiller et al.  for details on the calculation of this gradient.
depicts the complete network architecture of a policy network for an exemplary feature selection. The input layers all use the previously described input gating architecture. The output layers use the gated output heads introduced in the SAC-X paper. In this particular example, only the proprioception and feature groups are enabled for task 0 – and are hence allowed to flow into the hidden layers of thenetwork – while the image network output is zeroed out. The network used in our experiments assumes the outputs of all the functions are the same shape, and then sums the inputs element-wise. This reduces the amount of zeros flowing into the network layers, but requires the assumption that each of the state-spaces can match the expected internal representation of each other.
Iii-C Asymmetric Actor-Critic through Different State Spaces
The idea of using different observations for different tasks that we described above has similarities to the asymmetric actor-critic technique from Pinto et al.  that are worth expanding on. In asymmetric actor-critic training, the state-space of the critic (in our case the Q-function network) is enriched with additional information (which may not be available at test time). This additional information (e.g. extracted, hand-designed features) can be used by the critic to simplify the problem of credit-assignment (identifying the cause of reward for a given state-action pair). Once a policy has finished training the critic is no longer needed, and can be discarded – akin to how auxiliary tasks can be discarded after training with our method. When viewed from this perspective, we can interpret our method as an extension of asymmetric actor-critic to a multi-task setting (with varying state-definitions across task-policies).
Furthermore, we can observe that the ideas of auxiliary tasks with state-based features (to aid fast learning from raw images) and asymmetric learning in actor and critic as complimentary. This opens up further possibilities in our experiments: in addition to learning both feature and vision based policies we can restrict the critic state-filters () to provide only feature observations for all tasks, potentially leading to an additional speed-up in learning time (we refer to this as the asymmetric actor-critic setting below).
Iv Ball-in-a-Cup Task Setup
The following section describes the experimental setup for both simulation and the real-robot. Table I shows the observations contained in the previously defined state-groups (i.e. , , ), as well as the source of the observation and its shape.
|Cup Orientation (Quaternation)||4|
|Cup Linear Velocity||3|
|Cup Angular Velocity (Euler Angles)||3|
|Ball Linear Velocity||3|
|Action Filter State||4|
|Stacked Color Front Image|
|Stacked Color Side Image|
We use a Sawyer robot arm from Rethink Robotics 222http://mfg.rethinkrobotics.com/intera/Sawyer_Hardware
, which has 7 degrees of freedom (DoF). The agent controls velocities of 4 out of 7 DoFs (joints J0, J1, J5 and J6) – which are sufficient to solve the task. The unused DoFs are commanded to remain at a fixed position throughout the episode.
On the real system, the position and orientation features of the ball and cup are determined by an external Vicon motion capture system. IR reflective markers are placed on the cup and ball. The Vicon system tracks the position and orientation of the marker clusters. The position and orientation estimates are accurate to sub-millimeter precision, have low latency and low noise.
We also include two external cameras roughly positioned at orthogonal views (see Figure 2). RGB frames of
pixels are captured at 20 frames per second, and down-sampled via linear interpolation topixels – and subsequently fed to the neural networks. Figure 3 shows pictures of the real camera images seen by the robot during training. We deliberately make no effort to further crop the images, tune the white balance or correct for lens distortions – relying on the adaptability of our approach instead. To enable estimation of ball velocities, when learning from raw images, we feed a history of the last three images captured by the cameras into the neural networks.
Data from proprioception (100Hz), Vicon (100Hz) and cameras (20Hz) is received asynchronously. State observations are sampled from this data at 20Hz with no buffering (i.e., most recent data is used). The agent controls the robot synchronously with the observation sampling at 20 Hz. The control rate is chosen to allow for the robot to dynamically react to the current ball position and velocity, while not being so high-rate as to overwhelm the learning algorithm.
Iv-a Experimental setup
The ball has a diameter of 5 cm and is made of foam to prevent damage to the robot while swinging. A Kevlar string of 40 cm was used to attach the ball to a ball bearing fixed at the robot’s wrist. The bearing is in line with the robot’s wrist axis of rotation and helps prevent winding of the string while swinging the ball – when the ball goes around the wrist the string rotates freely via the bearing.
The cup is 3D printed, and a woven net is attached to it to ensure visibility of the ball even when inside the cup. The cup is attached to the wrist right behind the bearing. We perform the experiment with two different cup sizes: a large cup with 20 cm diameter and 16 cm height and a small cup with 13 cm diameter and 17 cm height. Experiments are performed with the larger cup size unless otherwise stated.
Iv-B Robot Safety
When training on the real robot, the safety of the robot and its environment must be considered. An agent sending random, large, velocity actions to the robot may cause it to hit objects or itself. Additionally, quickly switching between minimum and maximum joint velocity commands may incur wear to the robot’s motors. We hence utilize an action filter as well as a number of hard-coded safety checks to ensure safe operation.
In more detail: The Sawyer robot has built in self-collision avoidance, which is enabled for the task. To prevent collisions with the workspace we simply limit the joint positions to stay within a safe region. Additionally, the limits help keep the real robot, cup and ball within the field of view of the Vicon system and cameras.
To prevent high frequency oscillations in the velocities we pass the network output through a low-pass first order linear filter with cutoff frequency of 0.5 Hz – which was chosen qualitatively so that the robot is able to build up large velocities before hitting the joint limits but prevents switches between minimum and maximum velocities. Such a filter has an internal state which depends on the history of policy actions. If unobserved, this would make the learning problem non-Markovian. We hence, provide the agent with the internal state of the action filter as part of the proprioceptive observations.
Iv-C Episode Resets
Training is episodic, with a finite horizon of 500 steps per episode, resulting in a total wall clock time of 25 seconds per episode. We randomly switch between intention policies every 100 steps (dividing the episodes into 5 subsets) – this is done to increase diversity in the starting positions for each policy. In simulation, episodic training is easy, as the simulator can simply be reinitialized before each episode. In the real world, resets for the Ball-in-a-Cup task can be tricky – due to the string wrapping and tangling around the robot and cup. In order to minimize human intervention during training, the robot uses a simple hand-coded reset policy that allows for mostly unattended training. This procedure simply commands the robot joints to drive to positions from a set of predefined positions at random (that were chosen to maximize the number of twists the arm does in order to untangle the string). For more details on the reset behaviour we refer to the supplementary material.
We use the MuJoCo physics engine to perform simulation experiments. We use a model of the robot using kinematic and dynamic parameters provided by the manufacturer. Joint torques and maximum velocity values are also taken from a data-sheet. We do not attempt to create a highly accurate simulation (i.e. via system identification), as we mainly use the simulation to gauge asymptotic learning behaviour. When training in simulation, observations come directly computed from the simulation state and hence are noise free – e.g. images are rendered with a plain background and do not contain artifacts that real-cameras suffer from such as distortion or motion blur. Figure 4 shows the simulation setup as well as samples of the raw camera images provided to the agent.
V-a Task Descriptions
Table II shows the reward functions used in the various simulation and real-robot experiments, here the ball position is always w.r.t. the cup coordinate frame. Note that is the main sparse reward function for the Ball-in-a-Cup task.
Figure 5 illustrates the different cup reference frames and positions used for the reward computation, such as the base, rim, etc.. The formula for is defined according to Equation 6, which was tuned to return based on the minimum and maximum ball height, respectively. The reward shaping of uses a 2D Gaussian with (cup base center) and for both X and Y axes, as per Equation 7. In this case, is defined so that values within the interval would coincide with the cup diameter and return zero when the ball is below the cup base;
We use two main state-spaces for the different training task which we refer to as: feature state-space and pixel state-space. Feature state-space tasks have both and enabled. Pixel state-space tasks have both and enabled.
Tasks are defined by a combination of state space type and reward function. We refer to a specific combination by the reward id followed by either “F” for feature state space or “P” for pixel state space. For example, 1F refers to a task with reward function 1 and a feature state-space.
|Reward ID||Reward Name||Reward Function|
|1||Ball Above Cup Base||+1 if ball height is above the base of cup in the cup frame|
|2||Ball Above Cup Rim||+1 if ball height is above rim of cup in cup frame|
|3||Ball Near Max Height||+1 if ball height is near the maximum possible height above cup in cup frame|
|4||Ball Near Opening||Shaped distance of ball to center of cup opening|
|5||Sparse Catch||+1 if ball in cup|
|6||Shaped Ball Above Cup Base||See equation 6|
|7||Shaped Swing up||See equation 7|
|8||Do nothing (distractor task)||Negative reward proportional to joint velocities|
V-B Learning with Different State-Spaces in Simulation
To measure the effectiveness of our proposed approach we first perform an ablation study in simulation – so that training runs can be parallelized and meaningful statistics can be easily gathered. Our evaluation protocol is as follows: after each episode of training we freeze the weights of the main task policy and run an evaluation episode for the sparse catch task (using the appropriate state space for this task) and report the cumulative reward.
Figure 6 shows the results of the ablation experiment. Each curve in the plot depicts the mean of 10 independent runs of the average reward of the evaluated task. The yellow curve shows the evaluation of task 5F when trained with tasks 1F, 2F, 3F, 4F, and 5F. As hypothesized earlier, training based on feature-only observations converges fastest – presumably because it is easier to learn within the feature state-space given its low dimensionality and expressiveness. With an episode time of 20 seconds, learning succeeds in time equivalent to approximately 5.5 hours of real-time training. The purple curve shows the evaluation of task 5P when trained with tasks 1P, 2P, 3P, 4P, and 5P. Using only image based state-space definitions results in approximately 8 times slower convergence of the main task policy. For comparison, this corresponds to an estimated 1.6 days of continuous training time on a real robotic system. In practice, the total time to train would likely be even higher as the episode resets takes time in the real world (and our simulation is noise-free).
Finally, the red and green curves show the results of our method. The red curve depicts the evaluation of task 5P when training is performed with tasks 1F, 2F, 3F, 4F, 5F, 1P, 2P, 3P, 4P and 5P, with . We can observe that this results in a significant speed-up in learning time, when compared to training from image data alone – with training time being slowed down only by a factor of 2, compared to training from features. The green curve shows the evaluation of task 5P when trained with tasks 1F, 2F, 3F, 4F, 5F, 1P, 2P, 3P, 4P and 5P, but this time using the asymmetric actor-critic technique (i.e. enabled in all critic filter lists and disabled in all critic filter lists). We see that there is a slight improvement in the early training regime, likely due to faster convergence of the critic network. This shows that faster learning of a good value estimate can lead to additional speed-ups.
V-C Learning from Features on a Real Robot
The simulation results from the previous section suggest that our approach can result in a significant speed-up when compared to learning from raw images only. However, there clearly exist differences between simulation and the real-robot due to modeling inaccuracies: Firstly, the simulated parameters of the robot (e.g. actuator torques, joint frictions, etc.) likely not closely matching the real robot. Secondly, the sensors in simulation are perfect with no noise and no missing observations unlike their real world counterparts. To gauge whether learning on the real robot performs similarly we hence first verified that we can learn a feature-based policy on the real robot.
Fort this we used different combinations of tasks 1F, 2F, 3F, 4F, 5F and 8F together with the larger cup. Figure 7 shows the resulting average reward of task 5F as training proceeds on the real robot. Note that only a single robot is used. Training takes place with minimal human supervision (a human operator only intervenes to reset the environment when the robot is unable to reset itself after multiple tries).
The yellow curve shows the performance of the policy for task 5F when only task 5F is used for training. This demonstrates that it is possible to learn Ball-in-a-Cup on the real robot with just the sparse reward function. However, the other curves in the plot show that training speed can be improved if good auxiliary tasks are included during training. This finding is in agreement with the prior SAC-X work . More interestingly, the purple curve shows the result of training with a purposefully useless “distractor” auxiliary task (8F), that is added to the auxiliary tasks 1F, 2F, 3F. Even with this distractor task, the robot learns a good sparse catch policy from features – using approximately the same training time as when only the useful tasks are included. This suggests that, while it may be difficult to find a minimal subset of auxiliary tasks which speed-up training, it may be safe to just add any additional auxiliary tasks one thinks to be useful, with minimal effect on the training time and the final policy performance.
Qualitatively, the learned policies exhibit good behavior. The swing-up is smooth and the robot recovers from failed catches. With a brief evaluation of 20 runs, each trial running for 10 seconds, we measured 100% catch rate. The shortest catch time being 2 seconds.
We repeated the same experiments with the smaller cup, to increase the difficulty of the task and assess achievable precision and control. There was a slight slow-down in learning and a small drop in catch rate to 80%, still with a shortest time to catch of 2 seconds.
V-D Learning with Different State-Spaces on a Real Robot
Finally, we applied our full method to obtain an image-based policy for the Ball-in-a-Cup task. Due to the long experiment times on a real robot, we did not re-run all baselines from Figure 6. Instead we opted to only use the approach that worked best in simulation: learning with different state-spaces combined with asymmetric actor-critic. We initially used tasks 1F, 2F, 3F, 4F, 5F, 1P, 2P, 3P, 4P, and 5P as in the simulation experiments. While all of the task policies showed some improvement in this experiment learning was slower than in simulation. To minimize experimentation time we hence decided to change the auxiliary rewards into shaped rewards, with the intention of speeding-up training time for these auxiliary tasks in order to more quickly generate good data for the sparse catch policy. A description of the changed reward structure can be found in the supplementary material.
Figure 8 shows the learning curve for task 5F and 5P when trained with 5F, 6F, 7F, 5P, 6P, and 7P. The critic filter vectors enabled only and for all tasks. We can observe that the policies for both state-spaces converge to their final performance after around 5,000 episodes. Thus, the training time is longer than in the equivalent simulation experiment (c.f. Figure 6). This increase in training time is likely due to the differences in the quality of the simulated vs real-world camera images. I.e. the simulation images are noise-free, have no motion blur and are consistently lit with proper colors. Furthermore, the robot joint sensors have noise and delays which are not modelled in simulation. Despite the inherent challenges of the real world, we are still able to learn a sparse catch policy (operating on image and proprioception data only) from scratch in about 28 hours – ignoring time used for episode resets. This amounts to using about twice as many episodes as in our successful simulation experiments.
We conducted an evaluation of the final policy saved after 8,360 episodes. During the evaluation the exploration noise was set to zero. We ran a total of 300 episodes, achieving catch rate (catch in all 300 episodes). Results of this evaluation (total reward and catch time) are presented in Table III.
|Catch time (seconds)||2.39||1.85||11.80|
|Total reward (maximum of 500)||409.69||14.0||462.0|
In this work we introduced an extension to the Scheduled Auxiliary Control (SAC-X) framework which allows simultaneously learning of tasks that differ not only in their reward functions but also in their state-spaces. We show that learning speed can be improved by simultaneously training a policy in a low-dimensional feature-space as well as a high-dimensional raw-sensor space. On the experimental side, we demonstrated that our method is capable of learning dynamic real robot policies from scratch. To this end, we learned the Ball-in-a-Cup task, from scratch, using a single robot. Unlike previous work we can learn this task without any imitation learning, or restrictions on the policy class. Once learned, our policy can operate using raw camera images and proprioceptive features only.
- Abbeel and Ng  Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, page 1. ACM, 2004.
- Abbeel et al.  Pieter Abbeel, Adam Coates, Morgan Quigley, and Andrew Y Ng. An application of reinforcement learning to aerobatic helicopter flight. In Advances in neural information processing systems, pages 1–8, 2007.
- Abdolmaleki et al.  Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. arXiv preprint arXiv:1806.06920, 2018.
- Andrychowicz et al.  Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems, pages 5048–5058, 2017.
- Bengio et al.  Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41–48. ACM, 2009.
- Chiappa et al.  Silvia Chiappa, Jens Kober, and Jan R Peters. Using bayesian dynamical systems for motion template libraries. In Advances in Neural Information Processing Systems, pages 297–304, 2009.
- Espeholt et al.  Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint arXiv:1802.01561, 2018.
- Forestier et al.  Sébastien Forestier, Yoan Mollard, and Pierre-Yves Oudeyer. Intrinsically motivated goal exploration processes with automatic curriculum learning. arXiv preprint arXiv:1708.02190, 2017.
- Haarnoja et al. [2018a] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290, 2018a.
- Haarnoja et al. [2018b] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905, 2018b.
- Heess et al.  Nicolas Heess, Gregory Wayne, David Silver, Timothy Lillicrap, Tom Erez, and Yuval Tassa. Learning continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems, pages 2944–2952, 2015.
- Heess et al.  Nicolas Heess, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, Ali Eslami, Martin Riedmiller, et al. Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv:1707.02286, 2017.
- Hessel et al.  Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojciech Czarnecki, Simon Schmitt, and Hado van Hasselt. Multi-task deep reinforcement learning with popart. arXiv preprint arXiv:1809.04474, 2018.
- Jiang et al.  Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International Conference on Machine Learning, pages 2309–2318, 2018.
- Kober and Peters  Jens Kober and Jan Peters. Learning motor primitives for robotics. In Robotics and Automation, 2009. ICRA’09. IEEE International Conference on, pages 2112–2118. IEEE, 2009.
- Kormushev et al.  Petar Kormushev, Sylvain Calinon, and Darwin G Caldwell. Robot motor skill coordination with em-based reinforcement learning. In Intelligent Robots and Systems (IROS), 2010 IEEE/RSJ International Conference on, pages 3232–3237. IEEE, 2010.
Levine et al. 
Sergey Levine, Peter Pastor, Alex Krizhevsky, Julian Ibarz, and Deirdre
Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection.The International Journal of Robotics Research, 37(4-5):421–436, 2018.
- Mnih et al.  Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
- Munos et al.  Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Marc Bellemare. Safe and efficient off-policy reinforcement learning. In Advances in Neural Information Processing Systems, pages 1054–1062, 2016.
- OpenAI et al.  OpenAI, Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. arXiv preprint arXiv:1808.00177, 2018.
- Parisotto et al.  Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning. arXiv preprint arXiv:1511.06342, 2015.
- Pinto et al.  Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojciech Zaremba, and Pieter Abbeel. Asymmetric actor critic for image-based robot learning. arXiv preprint arXiv:1710.06542, 2017.
- Riedmiller et al.  Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van de Wiele, Volodymyr Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing-solving sparse reward tasks from scratch. arXiv preprint arXiv:1802.10567, 2018.
Ross et al. 
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell.
A reduction of imitation learning and structured prediction to
no-regret online learning.
Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627–635, 2011.
- Rusu et al.  Andrei A Rusu, Matej Vecerik, Thomas Rothörl, Nicolas Heess, Razvan Pascanu, and Raia Hadsell. Sim-to-real robot learning from pixels with progressive nets. arXiv preprint arXiv:1610.04286, 2016.
- Schaul et al.  Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International Conference on Machine Learning, pages 1312–1320, 2015.
- Schmidhuber  Jürgen Schmidhuber. Powerplay: Training an increasingly general problem solver by continually searching for the simplest still unsolvable problem. Frontiers in psychology, 4:313, 2013.
- Sutton et al.  Richard S Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick M Pilarski, Adam White, and Doina Precup. Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction. In The 10th International Conference on Autonomous Agents and Multiagent Systems-Volume 2, pages 761–768. International Foundation for Autonomous Agents and Multiagent Systems, 2011.
- U.K.  Vicon Motion Systems Ltd. U.K. Tracker motion capture software for vr object tracking, 2019. URL https://www.vicon.com/products/software/tracker. [Online; accessed 1-February-2019].
Appendix A Network Architecture and Hyperparameters
In this section we outline the details on the hyper-parameters used for our algorithm and baselines. Each intention policy is given by a Gaussian distribution with a diagonal covariance matrix, i.e.
The neural network outputs the mean and diagonal Cholesky factors , such that . The diagonal factor has positive diagonal elements enforced by the softplus transform to enforce positive definiteness of the diagonal covariance matrix.
The general network architecture we use is described in Table IV. The image inputs are first processed by two convolutional layers followed by a fully-connected layer (see state-group heads size) followed by layer normalization. The other input modalities (features and proprioception) go through a fully connected layer (see state-group heads size), followed by layer normalization. The output of each of these three network blocks are then multiplied with 1 or 0 (depending on
). Because all the input layers output the same shape, and we assume that the different state-groups can extract the same internal hidden representation, we sum the state-group output layers elementwise. This reduces the number of dead inputs and number of parameters needed in the first shared hidden layer. This summed output is passed to a set of fully-connected, shared hidden layers (shared layer sizes in the table). The shared hidden layer output is passed to the final output layer. In the actor, the output size is the number of means and diagonal Cholesky factors. In the critic, the output size is 1, corresponding to the Q-value of the state-action pair.
|2D Conv layer features (layer 1/ layer 2)|
|2D Conv shapes (layer 1/ layer 2)||,|
2D Conv strides (layer 1/ layer 2)
|Actor net shared layer sizes||200, 200|
|Actor net state-group heads size||100|
|Critic net shared layer sizes||400, 400|
|Critic net state-group heads size||200|
|Discount factor ()||0.99|
|Adam learning rate||0.0001|
|Replay buffer size||1,000,000|
|Target network update period||1,000|
|Maximum transition use||2,500|
|Tanh on networks input||No|
|Tanh on output of layer norm||Yes|
|Tanh on Gaussian mean||Yes|
Appendix B Real World Episode Reset Procedure
As described in the main paper we use a hand-coded reset strategy at the beginning of each episode (if the string was detected to be tangled around the arm). This procedure works as follows. Before starting the episode the robot arm is driven to a preset starting position. If the string is untangled and the ball is hanging freely, then the ball will be in the expected starting area. If the ball is not detected in this area, then the robot begins its untangle procedure. Ideally, we would know the state of the string and be able to either plan or learn a sequence of actions to untangle the string. However, the string is unobservable in features and given the low-resolution of the images, largely unobservable to the cameras. Instead, to untangle, the robot picks a random sequence of positions from a pre-set list. This list was chosen to maximize the number of twists and flips the arm does in order to maximize the chance that the string is unwrapped. After the sequence is completed the robot returns to the check position. If the ball is untangled then the next episode can begin. If the ball is still not in the expected area the entire procedure is repeated until a successful untangle.
Appendix C Robot workspace
|min. position (rad)||-0.4||0.3||0.5||2.6|
|max. position (rad)||0.4||0.8||1.34||4.0|
|min. velocity (rad/s)||-2.0||-2.0||-2.0||-2.0|
|max. velocity (rad/s)||2.0||2.0||2.0||2.0|
C-a Joint limits
The Sawyer robot arm has a very large workspace333http://mfg.rethinkrobotics.com/intera/Sawyer_Hardware and can reach joint configurations which are not required to solve the Ball-in-a-Cup task or are outside of the motion capture system volume. In order to reduce the robot workspace, throughout all experiments we fixed J2, J3 and J4, and constrained position and velocity limits of the other joints, as detailed in Table V.
C-B Starting position
At the beginning of every episode, the robot joints J0-J6 are commanded to .
C-C Untangling positions
The pre-set list of untangling positions is detailed in Table VI.
Appendix D Changes to Experiment Reward Functions
As mentioned in the main text, the experiment in section V-D used a set of modified reward functions. The initial real robot training used tasks 1F, 2F, 3F, 4F, 5F, 1P, 2P, 3P, 4P, and 5P. As discussed, these rewards were switched to shaped rewards to speed-up training time.
The actual tasks used in section V-D are 5F, 6F, 7F, 5P, 6P, and 7P. Tasks 1F, 2F and 3F have been replaced by shaped reward 6F. Task 4F was replaced by shaped reward 7F. Similarly, tasks 1P, 2P and 3P are replaced by shaped reward 6p. Task 4P was replaced by shaped reward 7P.