There has been a recent push in robotics to move from analytical reasoning to more data-driven and self-supervised learning of planning and control. Specifically, the paradigm of end-to-end learning has gained a lot of prominence. In an end-to-end learning framework, the input is the perceived image and the output is the action primitives or torques themselves. Amazingly, it has been shown that given a specific task, if enough amount of data is collected, these frameworks can outperform manually designed mathematical models[1, 2, 3, 4].
However, the critics of self-supervised learning often argue that the amount of data required to learn these models effectively is huge and presents a big bottleneck. For example, in our recent paper , we collect more than 50K examples to learn a grasping model. Furthermore, since most end-to-end learning approaches deal with large amounts of data, they use self-supervision from other sensors. This more often leads to weaker notion of success. For example,  uses a force sensor in the gripper to see if the robot was successful in grasping the object. This often leads to an unstable grasp (see figure 2
for examples) being classified as a good grasp in the self-supervised framework.
To address the data scalability concerns, researchers at Google developed an arm farm with between 6-14 robots to collect 800K examples for grasping in parallel . This effort demonstrated how exploration can be parallelized and used to collect data at scales like never before. However, apart from scaling up data collection, is there another way to improve performance? In fact, it is known from the rich history of ML that “all data is not equal” [5, 6]. It has been often shown that mining hard examples leads to faster convergence and better performance. Similarly, obtaining better labels for training tends to make learning more efficient and robust. So, when using multiple robots, is there a better way to use multiple robots for exploration leading to faster learning and robust task performance?
Inspired by the recent work in adversarial learning [7, 8], we propose a novel physical “adversarial” framework to collect data for manipulation tasks such as grasping. Instead of using robots in a collaborative fashion, we propose to create robot adversaries for self-supervised learning. Specifically, we have an original learner that learns how to perform the task such as grasping; and the adversarial learner tries to learn a task/policy which makes the original learner fail on its task. For example, if the first learner is trying to learn how to grasp the object; the adversarial learner attempts to learn how to steal the object via snatching. This in turn forces the original learner to learn to grasp in a robust manner such that it cannot be snatched by the adversary. In this paper, we create two such adversarial frameworks for grasping. We show how adversarial tasks can help us provide supervision/labeling which rejects weak notions of success leading to faster and better learning.
More importantly, we demonstrate quantitatively that using an adversarial framework leads to a significantly better grasping policy as compared to having two robots collect data in parallel for the same task.
Ii Related Work
The learning framework we propose intersects with numerous areas of research. In particular, our method employs adversarial learning in physical robots. We then demonstrate the significant improvements this brings to training a robot to grasp.
Self-supervised deep learning9]10]
. In robotics, to solve data requirements, deep learning methods have been coupled with self-supervision to generate end-to-end learning solutions[1, 2, 11, 3, 4].
. Active learning is often employed in domains where data is cheap but labeling is expensive. This is the case in self-supervised paradigms, where the cost of labeling corresponds in part to the time it takes to run the experiment, so there is often a severe limit on the practical amount of data that can be generated. Active learning is motivated by the idea that fewer examples can result in better accuracy if the right data is chosen (refer to). In simplified, linear classification problems, it has been shown that exponential decrease is achievable in expectation with no efficiency gains in the worst case . We will show that our technique, which operates over much more complex decision surfaces still achieves significant performance boost over naive sampling.
Our work is also related to Hard-negative/example mining [5, 6], formerly referred to as bootstrap learning. Recent methods in computer vision [14, 15, 16, 17] and sample efficiency in reinforcement learning  apply hard example mining to train deep neural networks. These methods, however, base their filter of examples by the loss of each sample. In our case, adversarial/antagonist network guides the training of the original network to select harder training set.
The value of more efficient training in robotics has long been investigated due to the high cost of data. Use of active learning has ranged from curriculum learning , selectively filtered uncommon or interesting data , to directed exploration in reinforcement learning (see ). Much of the recent work in improving exploration for deep learned models has focused reducing uncertainty [22, 23] or is novelty seeking [24, 25]. Unlike reinforcement learning, our approach is a repeated game of a single time-step. Moreover, to our knowledge, no previous work in robotics or reinforcement learning has trained an adversary to guide exploration or for sample efficiency.
Game Theory. We formulate our adversarial training as a two player zero sum repeated game (refer to 
for an overview of game theory). Game theory has been explored in many robotics applications. Multi-agent systems[27, 28] are often formulated as a cooperative game. In pursuit evasion  and robust control methods , nature acts as the opponent against the system forcing the system to chose safe actions. Our approach is analogous, as the adversary forces the system to choose more stable actions. Unlike existing robotic game theoretic approaches, our technique trains two differing neural networks to act as both players.
Adversarial methods. Generative adversarial methods [7, 8] are similar in that they train two neural network agents in a game theoretic settings. However, the objective is to train a model capable of generating examples indistinguishable from training examples. More recent work leverages adversarial examples, similar to hard-negative mining approaches discussed above, to train a more robust classifiers . This method generates new samples based on analysis of a single trained network. Our formulation relies on two discriminative models that adapts to the current behavior of the opposing player.
Grasping. We test out our technique on the grasping problem, an intensely researched problem which is a complex task requiring a robust solution. Refer to [32, 33] for surveys of prior work. Significant progress has been made recently using data-driven solutions for training deep learned models on both corporeal robots [1, 2, 3, 4] and in simulation [34, 35].
Early work sought to define stable grasp metrics using analytic methods . However, later work found such metrics unstable under shaking  and pose uncertainty [38, 39]. Our approach demonstrates that adversarial agent in a data-driven context leverage these destabilizing factors to train a more effective solution. While we demonstrate the core algorithm on stable grasping, we believe the approach is applicable to other domains.
The goal of this paper is to explore how we can improve self-supervised systems with respect to (a) the quality of supervision and (b) robustness of the trained models. Current self-supervised systems use combinations of sensors to define success/failure on tasks [1, 2, 3, 4]. For example, in grasping, a force sensor in the gripper can be used to see if the robot was successful in grasping. However, such supervision is weak and noisy at best.
In this paper, we argue that learning to defeat an adversarial robot might provide significantly better supervision. Specifically, we learn models for two tasks simultaneously: one model is learned for the original-task such as grasping; the other model learns an adversarial task to defeat the original-task model. As an example, for grasping one adversarial task is to snatch the grasped object from the original model. An important feature of this joint learning is that while the adversarial is learning how to defeat original model; the original model adapts itself to defeat the adversarial model. This leads to greater robustness in the task performance compared to the original model itself.
Specifically, in this paper, we explore the use of adversarial robots for the task of grasping through two adversarial mechanisms. The first shakes the object in the gripper to break the grasp. The second adversarial mechanism is snatching. Given an object grasped by one arm of a robot, the adversary attempts to snatch the object using a second arm (in our experiment we use Baxter robot, which has two arms). Therefore, the original GraspNet tries to learn the grasping task such that it is robust to either a shaking or snatching adversary, respectively. Experimentally, we demonstrate that learning via competition from an adversarial model helps improve the performance on the test data.
Iv Adversarial Learning Framework
For purposes of explanation, the agent is the protagonist being trained to perform the task and the adversary is the antagonist attempting to defeat the agent. Our goal is to learn a non-linear function (we use ConvNets and represent the parameters) which given the current state of the environment (represented as ) predicts the action parameters . Therefore, . At the same time, we also try to learn an adversarial task , which given some state representation() after the action , predicts the adversarial action parameters . Therefore, . Note that the state of the world after protagonist action, , depends on the action and the world.
The optimization goal is to learn according to the following criteria:
where is the original task loss and is the adversary loss. Therefore, if the protagonist was successful on the original task and otherwise. Similarly, if the adversary was successful in defeating the protagonist. Note that is the weighting factor for learning of the protagonist.
The second term in the objective function pushes the adversary to learn parameters such that adversary’s action leads to agent’s failure (minimizing adversary’s loss). However, the original protagonist is trying to learn the parameters such: (a) the agent is able to minimize its own loss (the first term) and (b) maximize adversary loss, which implies the original task is performed so robustly that the adversary cannot defeat it.
Initializing : We first learn original task, without any adversary in play, only using the weak notion of success. Therefore, we collect the data with random actions and see which actions lead to success in the task using the sensors. This data is used to learn an original ConvNet. Note that the ConvNet
gives probability of success for all possible actions and therefore the policy is to select the action in greedy manner.
Initializing : Given the initial learned ConvNet for the original task, we use it to perform the task followed by an adversary action . Initially, we use a random adversary policy and collect data to observe effect of random adversary actions on the original task. We use the collected data to learn ConvNet for initial adversary task. Again, the ConvNet gives probability of success for all possible adversary actions.
Joint Training: Once we have initialized both protagonist and adversary task networks, we use the ConvNets to collect joint data. The original task network () defines the probability of success for all actions, we select the action with the highest probability of success and perform the action . If the is unsuccessful, we update parameters of original task network . If the is successful, then the adversary network () is used to select an adversary action
. Based on the success or failure of adversary action, both the adversary and protagonist network parameters are updated. This procedure is repeated by generating a series of batch updates for each iteration. Training repeated multiple epochs until a minimum accuracy threshold on the train set was achieved.
V Experimental Framework
We demonstrate the effectiveness of the proposed adversarial framework for the task of grasping using two different adversaries: shaking the hand holding the object; snatching the object using the other hand.
V-a Grasp prediction formulation
We formulate our problem to perform planar grasping , which defines action space/grasp configuration using three parameters: as position of grasp point on the surface of table and angle of grasp, respectively. This reduces the task of finding successful grasps to finding a successful configuration, in a given image . Examples of planar grasps are shown in Figure 6. However, as mentioned in 
, this formulation is problematic due to the presence of multiple grasp locations for each object. Therefore, we sample image patches from the input image and evaluate them by the ConvNet to give probability of success for all the grasp angles with grasp location being center of patch. Given an image patch, we output an 18-dimensional likelihood vector where each dimension represents the likelihood of whether the center of the patch is graspable at, , …. Therefore, the grasping problem can be thought of as 18 binary classification problems.
V-B Grasping as the protagonist agent
Figure 5 with describes the network architecture for our protagonist agent policy. Given an input image , multiple () patches are sampled. For each of these patches the agent network predicts the probabilities of successful grasping in the 18 different grasp angles. This gives a grasp probability matrix. Depending on the exploration strategy, an element is chosen from this matrix which corresponds to the grasp configuration . This grasp is then executed. The agent policy hence uses the state representation as the input image, i.e. and outputs the grasp action .
V-C Shake formulation
Shaking is used as one of the adversarial mechanisms (Figure 7). After the protagonist grasping agent grasps an object, the antagonist attempts to destabilize the grasp by vigorously shaking the end effector. The shake action space is discrete with 15 possible options. Each shake action corresponds to pair of end effector orientation and direction of linear shake. There are 5 end effector orientations used in this work as shown in Figure 4 and 3 possible directions of linear shake for each of these orientations. A sample end effector shake motion can be seen in Figure 7. Note that the frequency and amplitude of the shake is held constant for all the shake actions.
V-D Snatch formulation
The other adversarial mechanism we explore is snatching (Figure 8). After the protagonist grasping agent grasps an object, the antagonist attempts to destabilize the grasp by attempting first to push grasp  followed by pulling the object. Note that here the antagonist does not have control over the protagonist’s arm but has control over a second arm that attempts this push grasp. The snatch action space is discrete with 36 possible options. Each snatch action corresponds to a grasp point in the plane of the object (see Figure 9). There are 9 translational configurations and 4 rotational configurations. A sample end effector pull motion can be seen in Figure 8.
V-E Antagonist adversary
Given the original state and the executed action , we represent the current state as a function of and (say ). In this work is the image patch of centered at an rotated by . Hence, is the grasped patch of the object rotated by the angle of grasp so that it is visible in canonical viewpoint. This effectively encodes the image patch after the agent’s fingers grasp the object. This means that the adversary’s state representation is also the image .
The adversary policy now has to predict the adversarial action . For this we again use the deep network () architecture in Figure 5. For shaking, there are 15 actions, so and . Snatching has 36 actions, so and . Given the state , the network produces the probabilities of the adversary succeeding for the different actions. Based on the adversary’s exploration strategy, the action
is sampled from the probability distribution and executed. The adversary policyhence uses the state representation as a rotated patch and outputs the adversarial action .
V-F Network architecture
. The only difference being the fewer neurons in fc7 (1024 vs 4096). The output of the network is scaled to (0,1) using a sigmoidal response function. The convolutional weights are initialized from ImageNet pretrained weights, while the fully connected weights are initialized randomly.
V-G Learning to grasp with shake adversary
In our formulation to jointly train the grasping agent and the shaking adversary, we would first initialize the policy of the agent and the adversary. For the grasping task, we first collect random samples  and train an initial model on this. Let this dataset of random grasps be called . Each element of contains a patch grasped , the discrete angle at which it was grasped and if the grasp succeeded or failed. The training target is given by
Using this, the grasping network is trained using the sigmoidal outputs of the network in as predictions . The loss for this training is the binary cross entropy loss between and
. This network is trained using RMSProp.
Now given the initial grasping policy , we grasp objects by sampling from the probability distribution generated by the network in . Once this grasp is executed random shakes are applied on the grasped objects. We call this dataset , where is the shake data (success or failure of the shaking adversary).
Given a successful grasp in , first is rotated with the angle represented by to get . The target for the network in is now
Using this, the shaking network is trained using the sigmoidal outputs of the network in as predictions . The loss for this training is the binary cross entropy loss between and . This network is trained using RMSProp. Hence using these random shakes, the initial shaking policy is learnt.
To train , we again use the dataset . The training target for , is given by
Here, is the maximum probability the network in believes it can destabilize the grasping agent. is a factor that controls how strong the adversary controls the learning of the agent. Using these labels, network in is trained.
Once we have our initial models and , we now follow the iterative training process. For iteration , we grasp objects using with importance sampling on grasp probabilities. For successful grasps shakes are applied according to with greedy sampling on shake probabilities. The data collected is then used to train first followed by . The data collection procedure is described in Figure 10.
Vi-a Hardware Setup
All our experiments were run on a Baxter robot. The protagonist arm is employed with parallel jaw grippers which can apply up to 35N of gripping force with maximum payload of 2.2Kg. However, for the collection of adversarial data, the gripping force was reduced to 7N to bias the framework towards the adversary. However for final testing we report results with the maximum gripping force as well as with the train time gripping force. For experiments with the snatching adversary, the other arm of Baxter is used to snatch grasped objects from the protagonist arm.
Vi-B Data Collected
The grasping model was initialized using 40K grasping tries from .
For the shaking task, a total of 9k grasps are attempted over 3 iterations of batch adversarial learning. Out of these, the protagonist successfully grasped 2.4k. This low grasping rate can be attributed to our exploration strategy where we use importance sampling to sample grasps. Out of these successful grasps 0.5k were dislodged by the adversary.
For the snatching task, a total of 2k grasps are attempted over 2 iterations of batch adversarial training. The grasp network in this case was initialized with the grasp network trained by the shake adversary. Out of the 2k grasps attempted, the protagonist successfully grasped 0.7k. Of these, 0.2k adversarial snatches were successful in dislodging the protagonist.
For the baseline, we trained a grasping model using 56K training examples. Note that the baseline is trained with more data points assuming two robots can collect data faster in parallel (collaborative setting rather than adversarial).
Vi-C Grasping Results
For demonstrating the improvements of our adversarial framework, we use the objects shown in Figure 11. These objects are novel and have not been seen by the robot before. As a baseline we train a grasping model with the same amount of data as the adversarial model but without adversarial supervision. This is similar to . The results for grasping with low gripping force (20% of maximum gripping force), without additional grip and by sampling only 128 patches on the input image can be seen in Table I. For a successful grasp to be reported, the object must be grasped and lifted 30cm from the table.
We can clearly see the improvement by the adversarial framework among these objects. After 3 iterations of training with shaking adversary, our grasp rate improves from 43% to 58%. Note that our baseline network that does not perform adversarial training has a grasp rate of only 47%. This clearly indicates that having extra supervision from adversarial agent is significantly more useful than just collecting more grasping data. What is interesting is the fact that 6K adversary examples lead to 52% grasp rate (iteration 1) where as 16K extra grasp examples only have 47% grasp rate. This clearly shows that in case of multiple robots, training via adversarial setting is a more beneficial strategy.
Finally, we used the model trained after competing with shaking adversary to further train it by competing against a snatching adversary. Adding iterations of adversarial training with the snatching adversary further improves the performance to 65%.
To stay closer to the baseline , we increase the grasping force to the maximum possible and attach a rubber grip to the fingers similar to the one in . Instead of sampling 128 patches, 10 times more patches are sampled to ensure better coverage of the image space. The results for this can be seen in Table II. Once again, the shaking adversarial framework (80%) beats the baseline (68%) handily. In the case of adding snatching adversary (82%), we see a small 2% improvement.
|Object||Grasping without adversary||Grasping with adversary|
|Shake Adversary||Snatch Adversary|
|Water Gun 1||4||3||5||8||7||8|
|Water Gun 2||4||4||4||7||8||9|
|Object||Grasping without adversary||Grasping with adversary|
|Water Gun 1||5||8||8|
|Water Gun 2||3||8||8|
Data-driven approaches in robotics have demonstrated significant improvements in recent years [1, 2, 11, 3, 4]. However, these recent approaches are data hungry, which often limits their applicability. We presented an adversarial self-supervised learning framework to help overcome this data issue. By pitting an adversary against an agent attempting to learn the core task, an agent is forced to learn robust solutions resulting in better and more efficient learning. We tested our approach on grasping problems to validate and evaluate its benefits. The result is a significant improvement over baseline in grasping of novel objects: an increase in overall grasp success rate to 82% (compared to 68% if no adversarial training is used). Even more dramatically, if we handicapped the grasping by reducing maximum force and contact friction, the method achieved 65% success rate (as compared to 47% if no adversarial training was used).
This work was supported by ONR MURI N000141612007, NSF IIS-1320083 and gift from Google.
-  I. Lenz, H. Lee, and A. Saxena, “Deep learning for detecting robotic grasps,” International Journal of Robotics Research, 2015.
-  L. Pinto and A. Gupta, “Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours,” IEEE International Conference on Robotics and Automation (ICRA), 2016.
S. Levine, C. Finn, T. Darrell, and P. Abbeel, “End-to-end training of deep
Journal of Machine Learning Research, 2016.
-  S. Levine, P. Pastor, A. Krizhevsky, and D. Quillen, “Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection,” ISER, 2016.
-  K.-K. Sung, “Learning and example selection for object and pattern detection,” MIT A.I. Memo No. 1521, 1994.
H. A. Rowley, S. Baluja, and T. Kanade, “Neural network-based face detection,”IIEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 1998.
-  I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Neural Information Processing Systems (NIPS), 2014.
-  V. Dumoulin, I. Belghazi, B. Poole, A. Lamb, M. Arjovsky, O. Mastropietro, and A. Courville, “Adversarially learned inference,” arXiv preprintarXiv:1606.00704, 2016.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” inNeural Information Processing Systems (NIPS), 2012.
-  V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski et al., “Human-level control through deep reinforcement learning,” Nature, 2015.
-  L. Pinto, D. Gandhi, Y. Han, Y.-L. Park, and A. Gupta, “The curious robot: Learning visual representations via physical interactions,” European Conference on Computer Vision (ECCV), 2016.
-  B. Settles, “Active learning literature survey,” University of Wisconsin–Madison, Computer Sciences Technical Report 1648, 2009.
-  S. Dasgupta, “Analysis of a greedy active learning strategy,” in Neural Information Processing Systems (NIPS), 2004.
-  I. Loshchilov and F. Hutter, “Online batch selection for faster training of neural networks,” arXiv preprint arXiv:1511.06343, 2015.
-  E. Simo-Serra, E. Trulls, L. Ferraz, I. Kokkinos, and F. Moreno-Noguer, “Fracking deep convolutional image descriptors,” arXiv preprint arXiv:1412.6537, 2014.
X. Wang and A. Gupta, “Unsupervised learning of visual representations using videos,” inInternational Conference on Computer Vision (ICCV), 2015.
-  A. Shrivastava, A. Gupta, and R. Girshick, “Training region-based object detectors with online hard example mining,” arXiv preprint arXiv:1604.03540, 2016.
-  T. Schaul, J. Quan, I. Antonoglou, and D. Silver, “Prioritized experience replay,” arXiv preprint arXiv:1511.05952, 2015.
-  T. D. Sanger, “Neural network learning control of robot manipulators using gradually increasing task difficulty,” IEEE transactions on Robotics and Automation, 1994.
-  C. Dima, M. Hebert, and A. Stentz, “Enabling learning from large datasets: Applying active learning to mobile robotics,” in IEEE International Conference on Robotics and Automation (ICRA), 2004.
-  S. B. Thrun, “Efficient exploration in reinforcement learning,” Technical Report CS-CMU-92-102, School of Computer Science, Carnegie Mellon University, 1992.
-  R. Houthooft, X. Chen, Y. Duan, J. Schulman, F. D. Turck, and P. Abbeel, “Variational information maximizing exploration,” arXiv preprintarXiv:1605.09674, 2016.
-  I. Osband, C. Blundell, A. Pritzel, and B. Van Roy, “Deep exploration via bootstrapped dqn,” arXiv preprint arXiv:1602.04621, 2016.
-  M. G. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos, “Unifying count-based exploration and intrinsic motivation,” arXiv preprintarXiv:1606.01868, 2016.
-  D. Abel, A. Agarwal, F. Diaz, A. Krishnamurthy, and R. E. Schapire, “Exploratory gradient boosting for reinforcement learning in complex domains,” arXiv preprintarXiv:1603.04119, 2016.
-  T. Basar and G. J. Olsder, Dynamic noncooperative game theory. Siam, 1999, vol. 23.
-  P. Stone and M. Veloso, “Multiagent systems: A survey from a machine learning perspective,” Autonomous Robots, 2000.
-  L. Panait and S. Luke, “Cooperative multi-agent learning: The state of the art,” Autonomous agents and multi-agent systems (AAMAS), 2005.
-  S. M. LaValle, “Robot motion planning: A game-theoretic foundation,” Algorithmica, vol. 26, no. 3-4, pp. 430–465, 2000.
-  T. Başar and P. Bernhard, H-infinity optimal control and related minimax design problems: a dynamic game approach. Springer Science & Business Media, 2008.
-  I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” International Conference on Learning Representations (ICLR), 2015.
-  A. Bicchi and V. Kumar, “Robotic grasping and contact: a review,” in IEEE International Conference on Robotics and Automation (ICRA), 2000.
-  J. Bohg, A. Morales, T. Asfour, and D. Kragic, “Data-driven grasp synthesis—a survey,” IEEE Transactions on Robotics, 2014.
-  D. Kappler, J. Bohg, and S. Schaal, “Leveraging big data for grasp planning,” in IEEE International Conference on Robotics and Automation (ICRA), 2015.
-  J. Mahler, F. T. Pokorny, B. Hou, M. Roderick, M. Laskey, M. Aubry, K. Kohlhoff, T. Kröger, J. Kuffner, and K. Goldberg, “Dex-net 1.0: A cloud-based network of 3d objects for robust grasp planning using a multi-armed bandit model with correlated rewards,” in IEEE International Conference on Robotics and Automation (ICRA), 2016.
-  C. Ferrari and J. Canny, “Planning optimal grasps,” in IEEE International Conference on Robotics and Automation (ICRA), 1992.
-  R. Balasubramanian, L. Xu, P. D. Brook, J. R. Smith, and Y. Matsuoka, “Physical human interactive guidance: Identifying grasping principles from human-planned grasps,” IEEE Transactions on Robotics, 2012.
-  J. Weisz and P. K. Allen, “Pose error robust grasping from contact wrench space metrics,” in IEEE International Conference on Robotics and Automation (ICRA), 2012.
-  J. Kim, K. Iwamoto, J. J. Kuffner, Y. Ota, and N. S. Pollard, “Physically based grasp quality evaluation under pose uncertainty,” IEEE Transactions on Robotics, 2013.
-  M. Dogar and S. Srinivasa, “A framework for push-grasping in clutter,” Robotics: Science and Systems (RSS), 2011.
-  T. Tieleman and G. Hinton, “Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude,” COURSERA: Neural Networks for Machine Learning, vol. 4, no. 2, 2012.