Log In Sign Up

Punyo-1: Soft tactile-sensing upper-body robot for large object manipulation and physical human interaction

by   Aimee Goncalves, et al.
Toyota Research Institute

The manipulation of large objects and the ability to safely operate in the vicinity of humans are key capabilities of a general purpose domestic robotic assistant. We present the design of a soft, tactile-sensing humanoid upper-body robot and demonstrate whole-body rich-contact manipulation strategies for handling large objects. We demonstrate our hardware design philosophy for outfitting off-the-shelf hard robot arms and other upper-body components with soft tactile-sensing modules, including: (i) low-cost, cut-resistant, contact pressure localizing coverings for the arms, (ii) paws based on TRI's Soft-bubble sensors for the end effectors, and (iii) compliant force/geometry sensors for the coarse geometry-sensing surface/chest. We leverage the mechanical intelligence and tactile sensing of these modules to develop and demonstrate motion primitives for whole-body grasping control. We evaluate the hardware's effectiveness in achieving grasps of varying strengths over a variety of large domestic objects. Our results demonstrate the importance of exploiting softness and tactile sensing in contact-rich manipulation strategies, as well as a path forward for whole-body force-controlled interactions with the world.


page 1

page 2

page 3

page 4

page 5

page 6

page 7

page 8


Soft-bubble: A highly compliant dense geometry tactile sensor for robot manipulation

Incorporating effective tactile sensing and mechanical compliance is key...

Comparing Piezoresistive Substrates for Tactile Sensing in Dexterous Hands

While tactile skins have been shown to be useful for detecting collision...

The robot skin placement problem: a new technique to place triangular modules inside polygons

Providing robots with large-scale robot skin is a challenging goal, espe...

Extrinsic Contact Sensing with Relative-Motion Tracking from Distributed Tactile Measurements

This paper addresses the localization of contacts of an unknown grasped ...

Navigation with Tactile Sensor for Natural Human-Robot Interaction

Tactile sensors have been introduced to a wide range of robotic tasks su...

Variable compliance and geometry regulation of Soft-Bubble grippers with active pressure control

While compliant grippers have become increasingly commonplace in robot m...

I Introduction

Figure 1: (top) Punyo-1 upper-body humanoid robot for whole-body manipulation with soft sensing arms, end effectors, and chest. (bottom) Side-by-side view of human and Punyo-1 grasping a large domestic object.

There are many difficult problems that must be solved before robust robotic manipulation finds its way into our homes and daily lives. Many avenues of manipulation research rely on gripper-only grasps and interactions, but humans often manipulate larger objects using their whole bodies with natural, contact-rich actions. Arms, chests, and other body parts are frequently used (Fig. 2) to carry and stabilize large or heavy objects, piles of items, or delicate items that require a gentle distribution of pressure. Additionally, our research has shown that older adults need and want assistance with large and heavy objects – a type of assistance that may allow them to live independently longer. Co-developing effective hardware and control strategies for whole-body manipulation tasks greatly expands the capabilities of robots, especially for aiding people in their homes.

Whole-body manipulation requires innovative solutions in both hardware and control. Only a few examples of whole-body manipulation have been reported in the literature (see Sec. II) and most current research focuses on the planning aspect of the problem. Control of whole-body interaction remains challenging, especially in the field of physical Human-Robot Interaction (pHRI). Recent innovations in manipulator design [1] point towards a mechanical design philosophy of mechanical intelligence, i.e., taking advantage of passive mechanical and material properties to solve problems [2]. Another related development in feedback-driven manipulation control uses tactile sensing [3]

to better estimate contact state, but this is typically limited to small regions like finger tips. Extending these ideas, we present a mechanical design and control strategy that explicitly leverages robot surface compliance, rich-contact, and tactile sensing to achieve effective whole-body grasping.

In this paper, we present a design philosophy for outfitting traditional hard robots with durable, highly compliant, tactile-sensing surfaces to enable more capable, contact-rich manipulation and minimal reliance on advanced control strategies. Punyo-1, the first in a series of bimanual humanoid robots covered in our soft, sensing modules embodies this design philosophy. The word “punyo” (ぷにょ) is a Japanese word used to describe something that is chubby and cute, yet resilient.

We also present a set of grasping experiments that exemplify the advantages of softness and tactile-sensing for whole-body grasping and lifting, object manipulation, and contact-rich pHRI.

Ii Background

In this section, we draw motivation for the domestic whole-body manipulation problem based on user needs, we survey robotic whole-body manipulation, and we overview the concepts of mechanical intelligence, compliance, and tactile sensing.

Ii-a Domestic user needs

Since 2016, we have observed and interviewed almost 300 older adults (age 65+) and surveyed over 1,600 in the United States and Japan (unpublished results). A dominant theme is that older adults have difficulty with lifting and carrying large, bulky and/or heavy objects; “heavy” can be quantified as 10 lbs/5 kg in the PROMIS [4] or as “groceries” in the SF-36 Physical Function [5]. As people age, they experience a decrease in their mobility and stability, which are key to being able to lift and carry these objects. Older adults are reluctant to ask for or accept help from a person as relying on someone else (e.g., familial caregiver or home health aide) can decrease their sense of independence. Some older adults even discontinue the use or purchase of such objects, which maintains their independence with a slightly reduced quality of life. Through our user needs finding research, we have found that some older adults are open to the idea of robotic technology physically assisting them, especially in Japan where the ratio of older adults to young adults is 1:2 [6].

Ii-B Robotic large object manipulation

The field of manipulation has primarily focused on end-effector planning and control – the need for whole-body manipulation arises out of a need to also grasp objects too large or heavy for the end effector alone. Early work in this domain [7, 8] identified kinematic requirements and force-based control methods for maintaining contact and attempted to account for the target geometry in the dynamics and control [9]. In the bimanual case, the planning problem has been analyzed from a geometric perspective  [10] on the basis of the finite nature of pairs of geometries that can be in contact and the resulting motion constraints induced.

In contrast with planning-oriented methods, whole-body control methods that have been reported mostly depend on joint-torque measurements and either treat it as a Cartesian impedance control problem [11] or explicitly aim to regulate the contact forces  [12]

. These methods rely on point-contact assumptions and derive model-based control laws to regulate the state. More recent works have also explored utilizing reinforcement learning to synthesize policies for whole-arm manipulation 


Figure 2: Humans, including older adults, must employ the whole body for household tasks.

While the kinematic/planning oriented approaches do not address robustness to disturbances (e.g., unexpected contact), both planning and model-based control methods require explicit object models, and in either case, grasp-generation control is not explicitly addressed. Moreover, none of these methods adequately address the challenging nature of the contact-dynamics between robot and object. Some challenges of contact-rich whole-body manipulation are similar to that of dexterous gripper grasp planning and control [14]:

  • accounting for closed-kinematic chains of multiple arms making and maintaining contact with a target geometry,

  • enumerating and maintaining rich contact (more than two independent contact regions) with the target object,

  • accounting for the frictional and compliance properties of the contact, and

  • observing the state of the target object and the contact area.

While numerically intensive planning with precise control or carefully trained machine learning can yield functional whole-body grasps, the field of soft robotics

[2] has shown that incorporating mechanical intelligence into the hardware designs could help leapfrog many of these underlying challenges. Compliant materials on robot surfaces can alleviate the burden of active compliance on grasping controllers and offer large, high-friction contact patches for stable form and force closure. Furthermore, these large contact patches provide a wide opportunity for tactile sensing-based state estimation.

Ii-C Compliance and tactile sensing

When it comes to maintaining contact and regulating object state during grasping and manipulation, it is increasingly recognized that incorporating under-actuated mechanisms, joint and surface compliance, and friction offers tremendous gains in manipulation robustness [1]. Effectively using compliant contacts all over the body was demonstrated by Alspach et al. [15], in which pressure-based feedback informed grasping of unmodeled objects. Inspired by these ideas, a gripper developed by Toyota Research Institute (TRI), known as the Soft-bubble (or Punyo) gripper [16, 17], incorporates highly compliant tactile sensing fingers [18] and not only maintains passively stable grasps but leverages multi-modal sensing to detect contact and manipuland state.

Various tactile sensing skins have been developed for both robot grippers [3] and for the body of the robot [19, 20]. These potentially enable whole-body interaction with objects, humans and the environment [21], and are a promising solutions for multi-contact control, as evidenced by results on monitoring contact patch geometries [22], and in developing manipulation primitives [23].

Building upon these ideas, we demonstrate an approach for augmenting off-the-shelf hard robot platforms with surface compliance and tactile sensing. We also develop a simple grasping controller that relies on compliance and tactile feedback. Our approach is perhaps best compared against the tactile feedback approach shown by Mittendorfer and colleagues [21], although their methods have only been demonstrated on a limited range of object weights and sizes. Moreover, in comparison, we demonstrate robust grasping with sensing that is much lower in spatial resolution but much more compliant; this allows for a large degree of passive surface behavior to maintain the grasp. With a strong focus on exploiting mechanical intelligence, we enable robust and effective whole-body manipulation for large, unmodeled domestic objects with comparatively simpler tactile feedback-based control strategies.

Iii Upper-Body Mechanical Design

We designed and fabricated a soft, bimanual upper-body platform, depicted in Fig. 1. This experimental platform is comprised of off-the-shelf robot arms augmented with compliant sensors on the links, end effectors, and chest. The upper-body assembly is mounted to a vertical lift with passive wheels so that it can be moved up, down, and around for grasping experiments.

Iii-a Kinematic exploration

The upper-body consists of two Kinova Jaco Gen2 7DOF robot arms [24] mounted vertically on either side of a compliant chest. The two arms and chest are mounted in-line on aluminum rail so that the shoulder width can be adjusted. The bottom of the chest aligns with the base of the arms, extends upward 42.8 cm and is 28.5 cm wide. The convex chest extends forward beyond the central axis of each arm’s first link by 8 cm.

The configuration of the arms and chest relative to one another, especially the shoulder angles, significantly impacts the whole-body manipulation workspace. To determine the most qualitatively capable starting configuration for grasping experiments, manipulator reachability, and ability to perform bimanual upper-body grasps, the arms were manually driven to explore grasping a set of large domestic objects (e.g., hamper, pasta pot) while varying the shoulder angle from -45 to 90 (Fig. 3). To ensure stable grasps, passively compliant, non-sensing structures were added to the contact locations (e.g., thin, high-friction rubber stickers and off-the-shelf inflated structures like inflatable armbands. From experimentation, a shoulder angle of 90 provided the best reachability.

With the shoulder angle set, the upper-body was designed to allow for adjustability for grasping experiments. The robot arms are mounted on horizontal sliders so that shoulder width can be adjusted between 45.8 and 55 cm (measured from the arms’ first link central axis); for the experiments herein, we settled on a shoulder width of 52 cm. The chest support structure provides mounting locations for quickly iterating the mechanical and sensing properties of the chest. The adaptability of the upper-body structure enables us to converge, through experimentation and adjustments, on an upper-body anatomy appropriate for manipulating a wide range of domestic objects.

Figure 3: right: Test rig used for exploration in -45 deg configuration, left: Schematic of the tested shoulder angles with respect to chest support structure.
Figure 4: High-friction, anti-rotation foam structures and air-tube/cable routing for pressure-sensing chambers.

Iii-B Vertical lift and passive mobility

The entire upper-body is secured to a manually-operated vertical lift shown in Fig. 1 (top). The vertical lift allows for the adjustment of the upper body to a specific grasping height. The base of the chest can travel vertically from the floor to a height of 140 cm. The upper body and vertical lift assembly sit on a platform with locking, leveling passive casters. This platform also supports the control computer and other electronics. With a power and networking cable bundle running off the platform, the system can be wheeled around to transfer grasped objects from one location to another.

The choice to build upper-body hardware atop a human operated vertical lift mechanism and a passive wheeled base is two fold: (1) The system represents a mobile manipulation platform, either wheeled or legged, without distracting development focus from the core manipulation goals being explored, and (2) the human element prevents a reliance on ground truth knowledge of the manipuland’s state, forcing an embrace of tactile-driven feedback control.

Iv Compliance and tactile sensing

The three types of modules used to cover the arms and chest combine the benefits of highly-deformable passive compliance with the richness of tactile sensing. The individual sensing module capabilities described here are stand-alone and can be used to augment more surfaces of future Punyo revisions, as well as other robotic platforms.

Iv-a Module i: Pressure-Sensing Chambers on Arm Links

Each arm is outfitted with four large, compliant, single chamber, air-filled pressure sensors, hereafter referred to as pressure-sensing chambers, that cover each of the main links of the robot arm. The pressure-sensing chambers consist of off-the-shelf inflatable armbands equipped with barbed tube reducers (McMaster 3913N16) in the inflation ports (Fig. 4(a)) with tubes connecting each to central pressure sensing electronics. Inflated armbands were chosen as the basis for our pressure-sensing chambers based on their passively stable performance during the kinematic exploration (Sec. III), availability, and ease of pressure sensing augmentation. The pressure-sensing chambers are held in place on the robot arm by high-friction, compliant foam structures which both prevent rotation around the link and form channels for wires and air tubes to run underneath the chambers (Fig. 4).

The pressure-sensing chambers are covered in cut-resistant Dyneema® fabric covers (McMaster 7847T13) (Fig. 4(c)), providing protection from sharp features that may be present in uncontrolled, cluttered environments. Each cut-proof cover is sealed using hook and loop and can be opened for armband repair and servicing, as well as for installing additional sensing layers (Fig. 4(b)).

Figure 5: Pressure-sensing chamber assembly: (a) Inflated armband with reduced port, (b) Partially opened Dyneema® fabric cover, and (c) robot-ready pressure-sensing chamber.

Each pressure-sensing chamber provides a scalar pressure value from a high accuracy Honeywell MPRLS0025PA00001A sensor. To localize contact on the chamber’s surface, surface contact position sensors can be placed underneath the chamber covers; (Fig. 6). The surface contact position sensors consist of an array of linear contact potentiometer sensors used to detect positions of contact. A small strip of stiffened rubber over each sensor serves as a force concentrator, increasing the sensor’s sensitivity to contact, even through the outer protective covering.

A flexible cartridge aligns four of these position sensors over the primary contact surface (i.e., the surface used for grasping) of a pressure chamber. These locations were chosen to sense intentional grasping contact over incidental contact, e.g., link-to-link self-contact, shown in Fig. 5(b). This architecture allows sensing of up to four separate contact points, each with position information. The additional information gained from these surface contact sensors can augment the sensed pressure to infer a pressure/force location for better estimate of contact state. However, for the scope of this paper, the control approach (described in Sec. V) only uses pressure as a feedback signal.

Figure 6: Surface contact position sensors: (a) four linear position sensors in flexible cartridge, (b) applied to a link’s pressure-sensing chamber demonstrating self-contact.

Iv-B Module ii: Soft-bubble “Paw” End Effectors

The bimanual system includes two tactile sensing “paws” as end effectors that add no additional degrees of freedom (Fig. 

7). The compliant paws use Soft-bubble gripper modules [16] and are able to sense internal pressure, dense contact geometry depth images, contact patches, and shear forces. The paws are used in both whole-body enveloping grasps like those shown in Fig. 11 as well as palm-only grasps.

Figure 7: Soft-bubble “paw” module end effectors on our system.

Iv-C Module iii: Force/Geometry Sensors on Chest

The soft, sensorized chest of our upper body system consists of a 45 array of custom designed force/geometry sensor modules (Fig. 8).

While our pressure-sensing chambers provide a single scalar pressure output, an individual force/geometry sensor module produces a directional force vector estimate during contact. This measurement is done with a sensory board containing four coils that induces an electromagnetic field and measures the eddy currents introduced and varied by the conductive target in the closed system (in this design, the target is a metal plate). These eddy currents are mapped directly to the distance and pose of the target, and, further, with the foam separation layer, can be mapped to an implied force that has created this displacement of the conductive target. With these sensors patterned across the contact surface of the robot chest, we are able to sense contact as the modules’ sensing surface conform to the outer contact surface of an object.

Mechanically, the assembly is as follows (Fig. 8 (right)): Each force/geometry sensor houses a board with four coils and accompanying electronics in a 5 cm square by 13.5 cm tall ABS case, which mounts to the robot’s chest frame. A 12.7 mm thick piece of soft polyurethane foam is glued to this ABS case, and on top of that there is a thin metal plate sandwiching the foam between itself and the ABS case. Unlike the pressure-sensing chambers, this design does not rely on inflated structures to achieve its softness. This opens up the door for many types of compressible materials to replace the current foam, such as anisotropic 3D printed lattices, to obtain different properties and force vs. displacement profiles. Overall, this design allows us to tune the softness and sensing of the force/geometry sensor modular chest structure.

The array of force/geometry sensors sits on a reconfigurable chest frame structure comprised of two crossbars with five vertical slats where the sensor modules are mounted. The crossbars can be changed out to modify the curve of the chest; its current configuration has a slight convex curve with the surface of each slat (and thus the base of each force/geometry sensor ) angled 170 from those horizontally adjacent. This curve could be updated to have a more or less extreme angle between the slats, or to even be concave, in order to support different types of grasps.

The chest is outfitted in a neoprene cover striped in 3M high-friction tape (3M TB641) (Fig. 8 (left)). The neoprene material creates a smooth, uniform surface over the entire force/geometry sensor array, providing a protective layer for the sensors and bridging gaps between the modules. Meanwhile, the 3M tape provides a high friction surface for objects to interface with during manipulation, reducing the load carried by the arms during a grasp.

Figure 8: Force/geometry chest array: protective neoprene surface covered (top-left) and uncovered (bottom-left), individual force/geometry sensor module exploded view (top-right), individual force/geometry sensor module assembled (bottom-right).

V Grasp planning and control

Figure 9: Modeling the Punyo-1 robot: depiction of the frames and joints relevant to planar whole-body grasping.

Exploiting mechanical intelligence often leads to simplifications in the control strategy for intelligent systems. In our case, taking advantage of our compliant hardware enables simple tactile feedback-based control schemes for large object grasping. The tactile feedback comprises of the sensed pressure at each of the pressure-based sensors (pressure-sensing chambers and Soft-bubble paws on Punyo-1). For the scope of this paper, we make an assumption that the change in sensed pressure post-contact is strictly positively correlated to external contact forces and therefore developed a grasp primitive based on a switching controller that is regulated by pressure changes. The Kinova Jaco SDK natively provides an interface for commanding joint velocities, and thus, our grasp primitive is designed in velocity space.

# Per arm pressure sensor readings
# Initial condition of arm
# Grasp pressure change threshold
# Joints involved in the grasp
# Current state of policy
# Desired joint velocities for each grasp stage
while  do:
     for each # Repeat for each arm          
     command on joint and on others # Move joint at a time
     if any(then
         #Transition to next stage of grasp
     end if
end while
Algorithm 1 Whole-body grasping primitive

The grasp primitive algorithm can be seen in Algorithm  1. For the work presented in this paper, we command each of the 7DOF arms independently; a low-level controller that monitors for self-collisions and feasible motions runs at 100 Hz. The joints relevant to both the primitives and the world, as well as the end effector reference frames, are depicted in Fig. 9. Our grasps are designed primarily to activate joints , , and .

An example motion policy for whole-arm grasping for the right arm is defined by first deciding a convenient pre-grasp configuration ; for instance, the arm oriented to be planar and curved inwards towards the target object with the soft face of the bubble palm facing towards the expected contact direction. All of the results presented in this paper used the same desired velocities parameters, which for each stage of the grasp (denoted by in Algorithm 1), is given by . The terminal pressure threshold starts at - this magnitude corresponds to a gentle contact useful for human-robot interactions (e.g., hugging, Fig. 10(d)), though we used higher thresholds to achieve more stable grasps of larger objects. We also developed a scripting interface to allow for parametric tuning of the pre-grasp configuration, grasp velocities, and the grasp pressure change threshold to enable grasping of the target objects. In practice, this joint-space method works well for arm/body grasps, but palm grasps were easier to implement in Cartesian space. This flexible and scriptable approach allows us to adapt to several new objects in a matter of minutes.

Vi Results

We tested the Punyo-1 hardware on a variety of domestic objects including a hamper, bins, a water jug, a stock pot, and a fish tank. In each case, we were able to generate contact-rich, stable grasps. This was all done without external perception and without prior models of manipuland shape.

Vi-a Importance of softness and friction

In order to test the hypothesis that softness leads to improvements in whole-body control and interaction with objects, we compared the performance of a hard surface (unmodified) Jaco arm against a soft Punyo-1 arm grasping a cylindrical pot (Fig. 10). We used an upside-down pot to eliminate the handle and lip affordances. The pot was loaded to 3.8 kg. In order to offer a fair comparison, we developed a torque limited switching controller (no tactile sensing).

The soft robot outperforms the hard robot in obtaining a robust grasp that can resist perturbation of the manipuland. Starting from the same pre-grasp state (Fig. 9(a)-9(b)), the weighted pot slipped out of the unmodified Jaco’s arm when the pot’s supporting surface was removed (Fig. 9(c)) while the compliant arm was able to continue holding the load (Fig. 9(d)). Further analysis of the comparative merits of a soft vs hard robot in performing whole-body manipulation is still under active investigation.

Figure 10: Comparative grasps of a weighted cylindrical pot (3.8 kg) between an unmodified Jaco arm and a Punyo-1 arm. (a-b) shows the pre-grasp state, (c) shows the slipping of the unmodified arm’s grasp while (d) shows a successful grasp in the compliant Punyo-1 arm.

Vi-B Whole-body grasping

In order to demonstrate the whole-body manipulation capabilities of Punyo-1, we tested and qualitatively analyzed the robustness of the resulting grasps on a set of objects (Fig. 11). We show that the system can handle more forceful tasks like carrying a large water bottle (Fig. 10(a)) or grasping two bins simultaneously (Fig. 10(c)). Initial testing shows that the upper-body can reliably achieve a stable grasp that is robust to antagonism. We can also use the same controller in a more delicate human-robot hugging interaction by lowering the grasp controller’s pressure threshold (Fig. 10(d)).

Figure 11: Robot grasping examples: (a) stable water bottle grasp and (b) corresponding sensor visualization, (c) dual waste bin grasp, and (d) human-robot hugging.

Vi-C Robustness of grasps to initial conditions

To test the robustness of the grasp to initial conditions, we varied the initial pose of the target manipuland (a laundry hamper) and qualitatively characterized the stability of the grasp. Fig. 12 shows the different stages of the grasp. Our platform and controller are able to handle a range of variability in initial object pose. The adaptability and stability of our grasps is due to the highly compliant nature of the robot and controller.

Vi-D System Durability

Punyo-1’s hardware proved durable when subjected to the repeated grasp testing detailed in this manuscript. The Dyneema® and neoprene fabric covers for the pressure-sensing chambers and force/geometry sensor chest array respectively provided confidence that the system could resiliently interact with objects. Collision avoidance requirements are also eased by the hardware because if the robot makes contact, whether intentional or accidental, the softness and ability to react on sensor input reduces the risk of damage to the robot and environment.

Figure 12: Two grasp series (top and bottom) with different initial laundry hamper conditions.

Vii Discussion

It is evident from our grasping experiments that the addition of highly compliant surfaces to hard robot manipulators and upper-body grasping surfaces enables stable whole-body grasps with a reduced reliance on object models, perception, planning, and control. The large, encompassing contact patches afforded by the surface deformation help solve typical grasping problems mechanically, including generating ample friction and obtaining force and form closure. These results solidify our commitment to continued development of a modular soft, sensing hardware and algorithmic toolbox for augmenting hard robots and reducing the barrier to entry for robust, whole-body grasping.

While our robot’s compliance is playing a large role in grasp stability, the tactile sensor feedback available is currently underutilized. Air pressure signals from the arm’s pressure-sensing chambers and end-effector paws are used to adapt each grasp to objects, but the chest’s force/geometry, arm’s surface contact position, and paw’s shear and geometry signals are currently unused. To move from basic object grasping to force-based manipulation, it is important to further understand the roles these and other sensing modalities play in contact patch estimation, object state estimation, and force control.

Even at this early stage of understanding, the breadth of potential capabilities enabled by rich tactile sensing is apparent. The ability for a robot to sense contact, both intentional and incidental, on all surfaces is important in the domestic environment where space is tight, cluttered, unmodeled, and shared with humans. Robots with whole-body tactile sensing offer many opportunities for compelling physical human-robot interaction, collaboration, and communication.

Viii Future Work

Continuing on our development of soft, sensing modules for hard robots, we will focus on techniques for localizing contact and estimating contact patches for sensors along the arms. We will also develop robot hardware agnostic mounting schemes for these modules. Rigorous mechanical and grasp testing will lead to a better understanding of mechanical failures and lead to increased module durability.

As the hardware team continues to explore and develop tactile-sensing capabilities, the control team must work to close the gap between sensing data available and which data is actually used for feedback control. More insight into which sensing modalities are useful and where will guide future hardware development. The large amount of data currently available, coupled with the durability of the system, may lend itself well to a machine learning based approach to grasp and other manipulation primitive generation.

Upon further understanding the grasping of large objects, we will move towards the grasping of heavy objects as well as force-controlled manipulation of these objects. These capabilities will require a better understanding of planning through contact, as well as the development of methods for more accurately making, controlling, and breaking contacts.

Lastly, our hardware platform is well suited for physical human-robot interaction research. We will use this system to explore collaborative manipulation in the context of physical assistance in the domestic environment.


Thank you Sam Creasey, Tristan Whiting, Phoebe Horgan, Richard Denitto, Liam Rondon, and Adeeb Abbas for engineering, experimental, filming and paper graphics support.


  • [1] J. Hughes, U. Culha, F. Giardina, F. Guenther, A. Rosendo, and F. Iida, “Soft manipulators and grippers: A review,” Frontiers in Robotics and AI, vol. 3, p. 69, 2016. [Online]. Available:
  • [2] S. Kim, C. Laschi, and B. Trimmer, “Soft robotics: a bioinspired evolution in robotics,” Trends in biotechnology, vol. 31, no. 5, pp. 287–294, 2013.
  • [3] Z. Kappassov, J.-A. Corrales, and V. Perdereau, “Tactile sensing in dexterous robot hands,” Robotics and Autonomous Systems, vol. 74, pp. 195–220, 2015.
  • [4] D. Cella, S. Yount, N. Rothrock, R. Gershon, K. Cook, B. Reeve, D. Ader, J. F. Fries, B. Bruce, and M. Rose, “The patient-reported outcomes measurement information system (PROMIS): progress of an NIH roadmap cooperative group during its first two years,” Medical care, vol. 45, no. 5 Suppl 1, p. S3, 2007.
  • [5] J. E. Brazier, R. Harper, N. Jones, A. O’cathain, K. Thomas, T. Usherwood, and L. Westlake, “Validating the SF-36 health survey questionnaire: new outcome measure for primary care.” British medical journal, vol. 305, no. 6846, pp. 160–164, 1992.
  • [6] World Bank Data, “Age dependency ratio, old (% of working-age population) - japan,” 2019. [Online]. Available:
  • [7] K. Salisbury, “Whole arm manipulation,” in Proceedings of the 4th international symposium on Robotics Research, 1988, pp. 183–189.
  • [8] B. S. Eberman, “Whole-arm manipulation: kinematics and control,” Ph.D. dissertation, Massachusetts Institute of Technology, 1989.
  • [9] P. Song, M. Yashima, and V. Kumar, “Dynamics and control of whole arm grasps,” in Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No. 01CH37164), vol. 3.   IEEE, 2001, pp. 2229–2234.
  • [10] J. Seo and V. Kumar, “Spatial, bimanual, whole-arm grasping,” in 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.   IEEE, 2012, pp. 257–264.
  • [11] M. Florek-Jasińska, T. Wimböck, and C. Ott, “Humanoid compliant whole arm dexterous manipulation: control design and experiments,” in 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.   IEEE, 2014, pp. 1616–1621.
  • [12] H.-C. Lin, J. Smith, K. K. Babarahmati, N. Dehio, and M. Mistry, “A projected inverse dynamics approach for multi-arm cartesian impedance control,” in 2018 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2018, pp. 5421–5428.
  • [13] W. Yuan, K. Hang, H. Song, D. Kragic, M. Y. Wang, and J. A. Stork, “Reinforcement learning in topology-based representation for human body movement with whole arm manipulation,” in 2019 International Conference on Robotics and Automation (ICRA), 2019, pp. 2153–2160.
  • [14] R. Ozawa and K. Tahara, “Grasp and dexterous manipulation of multi-fingered robotic hands: a review from a control view point,” Advanced Robotics, vol. 31, no. 19-20, pp. 1030–1050, 2017.
  • [15] A. Alspach, J. Kim, and K. Yamane, “Design of a soft upper body robot for physical human-robot interaction,” in Humanoids, Nov 2015, pp. 290–296.
  • [16] N. Kuppuswamy, A. Alspach, A. Uttamchandani, S. Creasey, T. Ikeda, and R. Tedrake, “Soft-Bubble grippers for robust and perceptive manipulation,” arXiv e-prints, p. arXiv:2004.03691, Apr. 2020.
  • [17] A. Alspach, S. Joonhigh, and Toyota Research Institute. (2021) Build a punyo gripper. [Online]. Available:
  • [18] A. Alspach, K. Hashimoto, N. Kuppuswamy, and R. Tedrake, “Soft-bubble: A highly compliant dense geometry tactile sensor for robot manipulation,” in 2019 2nd IEEE International Conference on Soft Robotics (RoboSoft).   IEEE, 2019, pp. 597–604.
  • [19] J. Kim, A. Alspach, and K. Yamane, “3d printed soft skin for safe human-robot interaction,” in IROS.   IEEE, 2015, pp. 2419–2425.
  • [20] G. Cannata, M. Maggiali, G. Metta, and G. Sandini, “An embedded artificial skin for humanoid robots,” in 2008 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems.   IEEE, 2008, pp. 434–438.
  • [21] P. Mittendorfer, E. Yoshida, and G. Cheng, “Realizing whole-body tactile interactions with a self-organizing, multi-modal artificial skin on a humanoid robot,” Advanced Robotics, vol. 29, no. 1, pp. 51–67, 2015.
  • [22]

    N. Kuppuswamy, A. Castro, C. Phillips-Grafflin, A. Alspach, and R. Tedrake, “Fast model-based contact patch and pose estimation for highly deformable dense-geometry tactile sensors,”

    IEEE Robotics and Automation Letters, 2019.
  • [23] F. R. Hogan, J. Ballester, S. Dong, and A. Rodriguez, “Tactile dexterity: Manipulation primitives with tactile feedback,” in 2020 IEEE international conference on robotics and automation (ICRA).   IEEE, 2020, pp. 8863–8869.
  • [24] Kinova Robotics. (2021) Jaco Gen2 7DOF. [Online]. Available: