Robots that operate in human environments have potential for making our lives easier. Unlike industrial environments, fixed robot trajectories are not usable in human environments since the objects and situations that the robot can encounter are virtually unbounded. Robots should be able to adapt to ever-changing situations and learn new skills to operate in unstructured environments. Utilizing recent machine learning techniques is a promising avenue for robotics.
Robotics researchers have increasingly been using deep learning, which emerged as a powerful statistical tool for processing high-dimensional inputs. Data for robot learning can be gathered in a multitude of ways including trial-and-error, from human experts and by observations 
. Reinforcement Learning (RL) is the most popular trial-and-error method where the robot explores the state space itself. Using reinforcement learning on real robots is impractical because the robot would need thousands of trials and it is challenging to explore the state space in a safe manner. Transferring RL policies from simulation to real robot is a solution proposed for this problem . Human expert methods include Learning from Demonstration (LfD) in which the robot learns the task by observing how humans behave in the same task . Observation methods  often involve learning by watching either an expert or another agent execute the task. How much human demonstration data a learning algorithm needs is an important criteria.
In this work, we use end-to-end learning from human demonstration data for data efficient behavior learning. The contribution of this work is a practical pipeline which takes a short amount of demonstration time to teach new behaviors to a robot. This approach provides a data-efficient alternative to reinforcement learning for robots. It is valuable to teach skills to robot in a short amount of time so that the outcome policies can be tested quickly and more data can be collected if necessary.
Ii End-to-End Training
We use an eye-in-hand setup where a RGB camera is attached to the end-effector. Our approach is based on mapping the input image to the end-effector velocity. A Convolutional Neural Network (CNN) is used to represent mapping function. End-to-end training shows promise in learning visuomotor policies.
The data collection is achieved by the human providing demonstrations of the intended behavior using a joystick. The operator uses start/stop recording buttons to initiate and pause recording of demonstration data. The human teacher aims for demonstrations with different initial states since it helps explore the state space. The recorded data consists of the raw image and the end-effector velocity (in the end-effector coordinate frame) that the operator applied.
The CNN represents the mapping from the input image to the end-effector velocity. The input images are re-sized to 224 by 224 pixels. We use a modified version of the ResNet architecture 
where the global average pooling and fully connected layers are removed and replaced by a new 1x1 convolutional layer for channel reduction, and new fully connected layers. Instead of outputting class probabilities as the original ResNet, we output 6 floating point numbers which represent the twist of the end-effector in the end-effector coordinate frame:. We use pretrained weights for the original ResNet layers and randomly initialised weights for the new layers. All weights are fine-tuned during training. These modifications makes the neural net training much faster (less than 30 minutes for the servoing task), which makes it feasible to learn robot behaviors practically.
During robot execution, the camera image is resized to 224 by 224 pixels and fed to the network for a feed-forward pass-through. To ensure that the robot does not mistakenly collide with itself or the table, we compute a safety twist at every frame. We apply the aggregated twist to the robot using the velocity control feature of the UR robot drivers. To compute , we construct a virtual workspace volume from spheres, cylinders and planes. If the end-effector position is inside this work area, then . Otherwise, we compute the twist term that will direct the end-effector back inside the work area with a magnitude that is proportional to the distance to the work area boundary. We were able to execute the approach in real-time, at the rate camera images are received (30 Hz).
The evaluation of our approach is focused on how much human demonstration effort is required for the robot to learn a new task. We chose a visual servoing task in order to validate the soundness and practicality of the approach. A human teacher (one of the authors) demonstrated the task for 20 minutes with different initial object and robot arm positions, ensuring that the robot would point directly above the cup with the camera looking straight into it. We measure the demonstration time in wall-clock time: we think it represents a realistic estimate of how much time it would take a robot teacher to demonstrate a task. We consider five checkpoints in this dataset: 4,8,12,16 and 20 minute marks as seen in TableI. For example, for the 4 minutes dataset we use the first four minutes of demonstrations only. Note that the frame count is not an exact multiple of demonstration time. This is because some of the demonstration time is not recorded while manually moving the robot and or the cup.
The success metric for the experiments is whether the robot can get near to the goal position (above the cup) under 1 minute. For each dataset, we kept the cup position constant and varied the initial robot position. We used 9 initial positions where 5 of them were close to the cup and 4 were relatively further. For each initial position, we run two trials. A total of 90 trials were executed where we recorded the success rate. The results are shown in Figure 2.
The success rate was the lowest for 4-minute dataset and steadily increased as demonstration time went up, as expected. 16 and 20 minute datasets achieved 100% success. From this result we can claim that 16 minutes of human demonstrations were enough to learn this task. We did not observe catastrophic forgetting for the 20-minute dataset, which is common in neural nets for multi-task learning .
Iv Conclusion and Future Work
This study shows that the robot can learn a simple task in under 1 hour including demonstration, model training and deployment time. Our approach can also be used for quickly learning an initial policy for data-inefficient reinforcement learning approaches .
Future work includes extending the approach to contact-rich and multi-step tasks. This would require utilizing multi-modal sensory input including images from different cameras and force/torque data.
-  Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (NIPS), 2012.
-  Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):21, 2017.
-  Fangyi Zhang, Jürgen Leitner, Michael Milford, Ben Upcroft, and Peter Corke. Towards vision-based deep reinforcement learning for robotic motion control. arXiv preprint arXiv:1511.03791, 2015.
-  Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. arXiv preprint arXiv:1808.00177, 2018.
-  Baris Akgun, Maya Cakmak, Karl Jiang, and Andrea L Thomaz. Keyframe-based learning from demonstration. International Journal of Social Robotics, 2012.
-  Adithyavairavan Murali, Siddarth Sen, Ben Kehoe, Animesh Garg, Seth McFarland, Sachin Patil, W Douglas Boyd, Susan Lim, Pieter Abbeel, and Ken Goldberg. Learning by observation for surgical subtasks: Multilateral cutting of 3d viscoelastic and 2d orthotropic tissue phantoms. In 2015 IEEE International Conference on Robotics and Automation (ICRA).
-  Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334–1373, 2016.
-  Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In , 2016.
David Isele and Akansel Cosgun.
Selective experience replay for lifelong learning.
Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
-  Tim Brys, Anna Harutyunyan, Halit Bener Suay, Sonia Chernova, Matthew E Taylor, and Ann Nowé. Reinforcement learning from demonstration through shaping. In Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015.