Recent progress in both supervised learning and deep reinforcement learning techniques has provided promising results towards achieving near human-level control in robotics for specific tasks. In the case of robotic manipulation tasks, large-scale data needs to be collected to take advantage of deep learning techniques, as demonstrated by Levineet al. , who used 14 real robots in parallel for 2 months to collect 800,000 grasp attempts. Although such methods can perform impressively, their real-world scalability and cost-to-performance ratio are questionable.
On the other hand, robotic learning using physical simulators provides a faster, more scalable, and safer training methodology than learning directly with physical robots. This has been showcased with impressive performances in simulation driven reinforcement learning . However, equivalent performance in physical robots is still lacking due to the apparent gap in environment domains between simulations and the real world.
Transfer learning with a domain adaptation between simulation environments and the real world  provides promising approaches to overcome this “reality gap.” The ability to generalize and adapt between images seen by a robot in simulation environment and those in the real world is crucial for such case of vision-based robotics.
. We train these neural network models entirely in simulation environments and then transfer them to the real world (Fig.1(b)). Especially, we propose a transfer learning method based on two variational autoencoders (VAE)  that generate common pseudo-synthetic images from synthetic and real images for detecting object positions. Our method successfully overcomes the gap between images synthesized in a simulation and images captured in the real world in order to predict object positions in the real world with less than
approximation error. Using this precisely detected object position, we can transfer pick-and-place or reaching tasks learned with a simulated 6-degrees-of-freedom (DOF) robotic arm to an actual physical robot without re-training in the real world.
Learning from synthetic images generally does not enable robots to perform as desired in real environments due to the gap between synthetic and real images. Shrivastava et al.  proposed a method to generate realistic images by refining synthesized images to overcome the gap between synthetic and real images. Santana and Hotz  combined a VAE and generative adversarial networks (GANs)  to generate realistic road images. However, these approaches need many unlabeled real images for the adversarial training during the refinement. Our method enables us to use a very limited dataset of real images, which are typically costly to collect, while leveraging a large dataset of synthetic images that can be easily generated in a simulation environment, for training the neural networks. Furthermore, it remains invariant to changes in lighting conditions or the presence of other distractor objects.
The rest of paper is organized as follows. Section II explains the formulation of the transfer learning problem. Section III describes details of our proposed method. Section IV quantitatively analyzes the method, and Section V presents two robotic applications utilizing our method. Finally, we conclude the paper in Section VI with directions for future work.
Ii Problem Statement
When we have two labeled image datasets (one synthesized in a simulation environment, and the other captured in the real world), we can assume image instances as and with representing synthetic image data and representing real image data, respectively. Since it is easy to synthesize many images for expected labels in a simulation environment and expensive to capture many images for expected labels in the real world, typically . We thus aim to extract meaningful information, , from the real world images that we can use for subsequent tasks of interest.
However, due to time and cost constraints, it is difficult to collect sufficiently large amounts of real world images to guarantee asymptotic convergence of the function of our interest, i.e. . We take an approach of modeling a given scene within a simulation environment in order to learn the function mapping, . This is done based on a large amount of corresponding synthetic images which can be collected easily in simulation. Given this setting, we want to learn a conditional distribution of synthetic images given the real world images, , by minimizing the following error,
where the expectation with respect to the conditional distribution minimizes the distance between the feature maps obtained from the real images and the feature maps of the corresponding reconstructed synthetic images obtained from the real images based on the conditional distribution .
In this paper, we focus on detecting real object positions from raw RGB-D (red, green, blue, depth) image data using this formulation as our target task for evaluating transfer learning. First, we train deep neural networks (DNN) with large number of synthetic image data as well as a small number of real image data along with their corresponding object position information. In order to prepare the data for training, we assume the object is put at a grid position during the training phase to achieve a uniform distribution of object position as shown in Fig.2(a). At the time of inference, we perform domain transfer from a synthetic environment to a real one by using the above trained DNN, in order to detect objects placed at random positions.
Iii Detecting Object Positions Using Vae
depicts the concept of our proposed method. The core idea of this work is that the distribution of image features may vary between simulated and real environments, but the output labels, like object position, should remain invariant for the same scene. We use two VAEs for generating similar common images from synthetic and real image data and use this common data distribution to train a convolutional neural network (CNN) to predict the object position with improved accuracy. Note that although we use two VAEs with distinct encoder layers as generative models for images, they have the same decoder, which is used to train the CNN. Thus, even if the VAE generates blurry images, the ensuing CNN will learn to predict from this skewed but common image distribution. Since the CNN can be trained with many generated images from the synthetic domain, we can achieve improved object position estimation from a very limited set of labeled real images.
Our proposed method consists of three steps as shown in Fig. 4:
Prepare two VAEs that output pseudo-synthetic images from both synthetic and real images.
Train a CNN to detect object positions using the output of trained VAE in (a).
Detect object positions by transforming domains using the trained VAE and CNN.
First, we prepare two VAEs to generate similar images from synthetic and real images. We set up a simulation environment that looks similar to the real world and capture large-scale synthetic images along with corresponding ground-truth object position labels, . We train VAE1, which encodes and decodes from a synthetic image to the same synthetic image as shown in Fig. 4(a-1).
The encoder compresses the input image to the latent representations , and the decoder reconstructs the image back from this latent space. However, using the encoder-decoder results in intractable posterior distribution , so we optimize the encoder parameters by variational inference and decoder parameters by minimizing the negative log likelihood of the data. Using this method, we obtain the optimal parameters by minimizing the lower bound given as,
is the prior distribution of the latent representation, which is typically the Gaussian with zero mean and unit variance.
We copy the weights of VAE1 to a VAE that has the same structure (VAE2) and then train VAE2, which encodes and decodes from a real image to the corresponding synthetic image as shown in Fig. 4(a-2). During the training, we fix the decoder layers and adapt only the parameters for the encoder part, which receives the real images as input, corresponding to the conditional distribution with encoder parameters . This is equivalent to forcing the latent space obtained from the synthetic and real images to be identical. Similar work has recently been done by Chaudhury et al. , who explicitly force the latent representation of multi-modal data to be identical during the optimization process. In this paper, we achieve similar effects for the latent space sharing using only a small dataset of real images by these steps. We obtain the optimal parameter by minimizing the following lower bound,
In the above optimization, note that are matching pairs of corresponding synthetic and real images. The learned encoder, , and decoder, , can be combined to obtain the desired conditional distribution , which can generate pseudo-synthetic images as output from the corresponding real image as input. VAE2 outputs can be subsequently used to obtain accurate object positions from a CNN trained purely in the synthetic image domain.
Next, we train a CNN for detecting object positions as shown in Fig. 4(b). Due to the availability of a large training dataset synthesized in a simulation environment, we can obtain a good prediction by a trained CNN for detecting object positions. To overcome the gap between synthetic and real images, we use the outputs of the trained VAEs in the above step, instead of using synthetic images directly.
Finally, we can detect object positions in the real world as shown in Fig. 4(c). VAE2 outputs blurry pseudo-synthetic common images, and the CNN trained with the similar common images outputs object position.
There can be alternate strategies in steps (b) and (c). In the above description, we train the CNN by using VAE1 output, but we can also train a multilayer perceptron (MLP) by using the latent representations obtained from the encoder of VAE1 in step (b). Since VAE1 and VAE2 have similar latent space structures, we can use latent space output from VAE2 combined with the above trained MLP to detect object position.
Our proposed method for detecting object positions was evaluated in four experiments:
Naive object position detection using a CNN without any generative model. We use this as a baseline.
Detection of object positions by the proposed method.
Applicability of the proposed method for different shapes.
Robustness of the proposed method in different lighting conditions and with other objects present.
For all experiments, we used a Gazebo®  simulation environment and Kinect®  as shown in Fig. 5. In Gazebo, an object model is loaded on white Styrofoam ( ) at an intended position as shown in Fig. 5(a). A corresponding Kinect model is also loaded in Gazebo to capture images of the work space scene. Fig. 5(c) shows a synthetic image captured by the Kinect in Gazebo. Additionally, the same object is manually put on the white Styrofoam at an intended position and its image is captured by the physical Kinect as shown in Fig. 5(b). Fig. 5(d) shows an image captured by the physical Kinect.
The images captured in both Gazebo and the real world were cropped to a smaller region, as shown by the yellow dotted lines in Figs. 5(c) and 5(d). The cropped RGB-D image sizes were pixels for experiments (A)(B) and for experiments (C)(D). We used five objects shown in Table I for experiments (A)-(D).
|No.||Color||Shape||Size ()||Weight ()||Experiment|
|(4)||blue||triangular prism||radius height||(C)(D)|
For all the experiments, we used the neural network configuration for VAE, CNN and MLP as shown in Fig. 6.
Iv-a Naive object position detection using CNN
Initially, we detect the red cube (1) position from images captured by Kinect using a CNN. The red cube is put at an uniform grid position during the training phase and then at a random position during the test phase as shown in Fig. 2. Making the grid smaller during the training phase enables us to prepare a larger training dataset, where we can expect the cube position to be detected more precisely. Table II shows the relationship between the grid size and the number of samples for the red cube on the white Styrofoam. If we want to obtain grid data, we have to capture images. While it is easy to capture images in Gazebo, it is highly resource intensive to capture such a large number of images in the real world. Also, we prepared synthetic images at random positions for the purpose of evaluation during the later experiments.
|Number of samples|
Fig. 7 shows the prediction errors: the difference between the true position and position detected by the trained CNN. The dark blue and dark green bars show the average position estimation error values for and coordinates, respectively. The light blue and light green bars show the maximum value of error for and coordinates, respectively. In Gazebo, the errors can be reduced by using a more fine-pitch grid during the training phase as shown in Figs. 7(a)-(e). When we use grid synthetic data for the training, we can obtain about errors on average and errors at worst in Gazebo.
In order to detect real object positions, we captured real images at grid positions for the training data. Moreover, we prepared real images at random positions for the later evaluation. If we can obtain cube images at more fine-pitch grid positions in the real world as well as in Gazebo, we can expect better precision. However, in practice it is very challenging to capture numerous images in the real world. The bar charts in Fig. 7(f) show the prediction errors by the CNN trained with the real images. As observed, these results are similar to those for the CNN trained with synthetic images in Gazebo. We set this result as the baseline for the later comparison.
We tried to feed the real test images to the CNN trained with images synthesized in Gazebo, and the corresponding prediction errors are shown as the bar charts in Fig. 7(g). As expected, the result for this naive usage of a model trained in Gazebo is worse than the baseline result in Fig. 7(f). This is primarily due to the “reality gap” between the synthetic and the real images.
Iv-B Detection of object positions by proposed method
We applied our proposed method for the same red cube (1) used in experiment A. We used synthetic images at grid positions for training VAE1 and real images and synthetic images at grid positions for training VAE2. Note that no data augmentation of these image data were performed .
Fig. 8(a) shows VAE1 output from a synthetic image input, and Fig. 8(b) shows VAE2 output from a real image input. Fig. 8 shows that we can obtain similar images that look like blurry synthetic images by using these two VAEs as we expected. The mean squared error (MSE) was on average for two input images and on average for two output images. This serves as a quantitative metric to show that our proposed VAEs improve the similarity between output images from real and synthetic domains. Although the white Styrofoam is put on a white desk and there is a wall on the left-hand side in the real world image, the white Styrofoam in the VAE output is put on a gray desk as shown in Fig. 8(b). VAE2’s output does not contain the real world wall but does contain a lower part of a robot arm and a similar shadow of the red cube.
After successfully learning to generate images with the VAE, we train the CNN/MLP for detecting cube positions using synthetic images generated in Gazebo. Fig. 9 shows the prediction errors of the red cube (1) in the real world.
As observed in Fig. 9(b), our proposed method of using the generated images of VAE2 to train a CNN results in a considerable reduction (nearly times) of the prediction error, as compared to the the baseline results of Fig. 9(a). Furthermore as observed by the results in Fig. 9(c), the alternative method of using the output of VAE2 encoder to train a MLP also results in similar good performance. These results suggest that the performance benefit clearly stems from the domain adaptation obtained by our VAE training method and it does not depend much on the choice of CNN or MLP.
Iv-C Applicability of proposed method for different shapes
We applied our proposed method to other shapes and colors as well. At this time, we put the white Styrofoam on a table in front of a real robot arm. For these experiments, we used three of the objects in Table I: green cube (2), black cylinder (3), and blue triangular prism (4). We prepared two VAEs in our proposed method for each object, and the corresponding outputs of VAE2 are shown in Fig. 10. In all three cases, VAE2 learns to successfully generate pseudo-synthetic images from the corresponding real images.
Iv-D Robustness of proposed method against different lighting conditions and distractor objects
In addition, we assessed the robustness of the proposed method in different lighting conditions and with other objects present. We usually kept the room light of our experimental space turned on as shown in Fig. 11(a). We turned off the room light and turned on a table light instead for a different lighting condition as shown in Fig. 11(b)
Fig. 12 shows the VAE2 outputs under two lighting conditions. VAE2 is trained with captured images under the default lighting condition shown in Fig. 11(a) and applied to the captured images under the other lighting condition without any re-training. The images on the left-hand side are raw images captured by the physical Kinect. Although the lighting condition in Fig. 11(b) is actually quite darker, the brightness levels of captured images are made similar due to Kinect’s auto-brightness functionality. The green color of the object is much lighter and the shadow is cast longer to the right in Fig. 12(b) than in the default lighting condition. As observed from the right side of Figs. 12(a) and 12(b), in both cases the VAE2 learns to generate very similar pseudo-synthetic images. Although the MSE between the real images under different lighting conditions was , the generated images result in a much smaller MSE of on average. This shows that using our proposed method we can effectively deal with differences in the scene caused by varying lighting conditions.
In the second set of experiments, we evaluated the robustness of the proposed method against the presence of multiple distractor objects in the same scene. Fig. 13 shows the generated output images from VAE2 when fed with real images containing multiple other objects. In this case, the VAE2 was trained with a single green cube object and it was subjected to the newly captured images with multiple objects without any further re-training. As shown in the right side of Fig. 13, the VAE2 continues to robustly generate pseudo-synthetic images with only the green cube while completely ignoring the other objects in the same scene. Therefore, this selectivity is quite useful for detecting the position of target objects even in the presence of numerous distractor objects of varying colors and shapes.
V Robotic Applications
In this section, we present the results of robotic tasks using the object position obtained by the proposed transfer learning method. Specifically, we perform two robot arm manipulations: (a) a pick-and-place task using finite state machines (FSM) trained in a supervised learning manner and (b) a reaching task using model-free control trained by reinforcement learning. The position of the desired object was estimated using our proposed VAE encoder with MLP method in both cases. Furthermore, in both cases the entire training were performed with a simulated robot arm and the learned policies were directly transferred to the real robot, generating trajectories in the real world without additional retraining.
We used Gazebo for the simulation environment and Kinect for the RGB-D camera. We used a UR5® robot arm from Universal Robots® and a two-finger gripper from Robotiq® for robotic manipulation in the real world. The robot arm consists of 6 joints, and the gripper can be controlled with an analog value between (open) and (close). In Gazebo, we used a corresponding simulated UR5 with simulated 2-finger gripper and it was controlled with a position based controller.
V-a Pick and Place Task Using Learned FSM
Rahmatizadeh et al. 
propose an approach where the user demonstrates a manipulation task in a virtual environment. They use the collected demonstrations to train a long short-term memory (LSTM) neural network to control the robots movements. When applied to the real world, Rahmatizadehet al. use objects annotated with 2D QR code markers to detect their positions. However, here we applied our proposed method to detect object positions directly from RGB-D images captured by Kinect.
We defined a pick-and-place task as transitions of an FSM that a green cube is picked up and put on a red cube. The same as De Magistris et al. , we executed a program for the transitions of the FSM repeatedly and recorded six robot arm joint angles and one gripper position at in each state and the initial positions of red and green cubes obtained from Gazebo for supervised training data. The training dataset was collected from over random cubes positions in an accelerated simulation environment with a real-time factor of .
Fig. 15 shows the diagram of the LSTM neural network fed with the estimated object positions directly from real image data based on our VAE-MLP pipeline. The LSTM neural network has 11 inputs (six robot arm joint angles, one gripper position at the current state, and four initial positions of two cubes) and seven outputs (six robot arm joint angles and one gripper position at the next state).
We trained the LSTM based robot controller by using these supervised data and then it could complete the sequence of the task in Gazebo with a success rate in trials. Finally the learned policy was transferred to the real robot without any retraining. We obtain the estimated positions of the red and green cubes based on our proposed method and fed these position data into the LSTM that was trained entirely in Gazebo. This network was then able to predict the sequence of actions needed for the task in the real world with success rate, across 10 trials. The successful experimental results for the pick and place task are shown in the video (see https://youtu.be/Wd-1WU8emkw).
V-B Reaching Task Using Model-Free Deep Deterministic Policy Gradient Learning
To train the robotic arm for reaching a 2D point on the table top, we performed model-free training of the simulated robotic arm in Gazebo. Angular position, angular velocity, and target position were used as state information for the robotic arm. During training, the target position was set randomly on the table top, whereas for testing, the target position was obtained by the proposed method. The height was fixed as the table top height. Continuous angular acceleration values for each joint were used as six-dimensional action space.
In order to do reinforcement learning with continuous actions, we used the deep deterministic policy gradient (DDPG) algorithm  that learns to compute continuous actions from state information input at each time step. We used the following reward function at each time-step,
where and are the current position of the robotic arm end-effector and the desired position of the object to be reached. The terms and represent the roll, pitch, and yaw angles of the current and desired pose of reaching, respectively. The reward function was designed so as to ensure that the robotic arm reaches the target position at a viable pose to enable grasping. The first term in the reward function minimizes the Euclidean distance between current 3D location of the robot’s end-effector and target 3D location. The second term minimizes the difference between the Euler angles of the current and desired poses as a soft constraint. We annealed the value of from a high to low value during the training period to ensure that the agent learns to reach with an accurate pose during early phases of training. Similarly, was varied from a low to high value to ensure accurate reaching of the end-effector to the object.
The learned policy for reaching a point on the table top using the DDPG algorithm in Gazebo was directly transferred to the real robot. The real environment was set up, and the object location on the table was computed using the proposed method from real images captured by Kinect.
After training, the learned policy for reaching an object on the table top using DDPG in Gazebo was directly transferred to the real robot, without any further reinforcement learning with the physical robot arm. Finally, in the real environment the object location on the table was computed directly from real raw RGB-D images captured by Kinect using our VAE-MLP pipeline. The successful experimental results for the object reaching task are shown in the video (see https://youtu.be/Wd-1WU8emkw).
Vi Conclusion and Future Work
We present a transfer learning method for transferring the ability of detecting object positions learned in a simulation environment to the real world. The proposed method can detect object positions more precisely than the baseline method of training a CNN or MLP directly from real image data. Our method performs robustly in different lighting conditions and with other objects present.
In addition, we applied the proposed method to robotic applications with robot controllers learned in the simulation environment and executed them in the real world combined with our proposed object position detection method. We show that a robotic arm can complete both a pick-and-place task and a reaching task in a real environment without any additional training. Furthermore it works independent of the type of learning mechanism i.e. supervised or reinforcement learning.
In this paper, we use only simple shapes with fixed poses and different colors for multiple object cases. For the next step, we plan to enhance our method for various type of shapes with varied poses and demonstrate object selectivity among same color objects. Transfer learning on complex natural images with arbitrary camera orientation is also a prominent direction for future research. In addition, we will tackle transfer learning of more complex robotic applications like contact based precise assembly tasks .
-  S. Levine, P. Pastor, A. Krizhevsky, and D. Quillen, “Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection,” International Symposium on Experimental Robotics (ISER), 2016.
-  V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing Atari with Deep Reinforcement Learning,” NIPS Deep Learning Workshop, 2013.
J. Schulman, S. Levine, P. Moritz, M. Jordan, and P. Abbeel, “Trust Region Policy Optimization,” International Conference on Machine Learning (ICML), 2015.
-  A. A. Rusu, M. Vecerik, T. Rothorl, N. Heess, R. Pascanu, and R. Hadsell, “Sim-to-Real Roboto Learning from Pixels with Progressive Nets,” arXiv 1610.04286, 2016.
-  I. Higgins, A. Pal, A. A. Rusu, L. Matthey, C. P. Burgess, A. Pritzel, M. Botvinick, C. Blundell, and A. Lerchner, “DARLA: Improving Zero-Shot Transfer in Reinforcement Learning,” International Conference on Machine Learning (ICML), 2017.
-  S. James, A. J. Davison, and E. Johns, “Transferring End-to-End Visiomotor Control from Simulation to Real World for a Multi-Stage Task,” 1st Annual Conference on Robot Learning (CoRL), 2017.
-  C. Finn, X. Y. Tan, Y. Duan, T. Darrell, S. Levine, P. Abbeel, “Deep Spatial Autoencoders for Visuomotor Learning,” International Conference on Robotics and Automation (ICRA), 2016
-  D.P. Kingma and M. Welling, “Auto-Encoding Variational Bayes,” The International Conference on Learning Representations (ICLR), 2014
A. Shrivastava, T. Pfister, O. Tuzel, J. Susskind, W. Wang, and R. Webb, “Learning from Simulated and Unsupervised Images through Adversarial Training,” Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
-  E. Santana and G. Hotz, “Learning a Driving Simulator,” arXiv:1608.01230, 2016.
-  I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. OzairA. Courville, and Y. Bengio, “Generative adversarial nets,” Advances in neural information processing systems (NIPS), 2014.
-  S. Chaudhury, S. Dasgupta, A. Munawar, M. A. S. Khan, and R. Tachibana, “Conditional generation of multi-modal data using constrained embedding space mapping,” arXiv preprint arXiv:1707.00860, 2017.
-  Open Source Robotics Foundation, Gazebo simulator, “http://gazebosim.org”.
-  Microsoft Kinect for Xbox One, “http://www.xbox.com/en-US/xbox-one/accessories/kinect”.
-  Q. Bateux, E. Marchand, J. Leitner, F. Chaumette, and P. Corke “Visual Servoing from Deep Neural Networks,” arXiv:1705.08940, 2017.
-  R. Rahmatizadeh, P. Abolghasemi, A. Behal, and L. Boloni, “Learning real manipulation tasks from virtual demonstrations using LSTM, ” arXiv 1603.03833, 2016.
G. De Magistris, A. Munawar, and P. Vinayavekhin, “Teaching a Robot Pick and Place Task using Recurrent Neural Network,” Vision Engineering Workshop (ViEW), 2016.
-  T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” arXiv:1509.02971, 2015.
-  T. Inoue, G. D. Magistris, A. Munawar, T. Yokoya, and R. Tachibana, “Deep Reinforcement Learning for High Precision Assembly Tasks,” International Conference on Intelligent Robots and Systems (IROS), 2017.