Log In Sign Up

Online Modeling and Control of Soft Multi-fingered Grippers via Koopman Operator Theory

by   Lu Shi, et al.

Soft grippers are gaining momentum across applications due to their flexibility and dexterity. However, the infinite-dimensionality and non-linearity associated with soft robots challenge modeling and closed-loop control of soft grippers to perform grasping tasks. To solve this problem, data-driven methods have been proposed. Most data-driven methods rely on intensive model learning in simulation or offline, and as such it may be hard to generalize across different settings not explicitly trained upon and in physical robot testing where online control is required. In this paper, we propose an online modeling and control algorithm that utilizes Koopman operator theory to update an estimated model of the underlying dynamics at each time step in real-time. The learned and continuously updated models are then embedded into an online Model Predictive Control (MPC) structure and deployed onto soft multi-fingered robotic grippers. To evaluate the performance, the prediction accuracy of our approach is first compared against other model-extraction methods among different datasets. Next, the online modeling and control algorithm is tested experimentally with a soft 3-fingered gripper grasping objects of various shapes and weights unknown to the controller initially. Results indicate a high success ratio in grasping different objects using the proposed method. Sample trials can be viewed at


page 1

page 3

page 5


Modeling and Control of Soft Robots Using the Koopman Operator and Model Predictive Control

Controlling soft robots with precision is a challenge due in large part ...

Sample-efficient Model Predictive Control Design of Soft Robotics by Bayesian Optimization

This paper presents a sample-efficient data-driven method to design mode...

Enhancement for Robustness of Koopman Operator-based Data-driven Mobile Robotic Systems

Koopman operator theory has served as the basis to extract dynamics for ...

Koopman Operators for Modeling and Control of Soft Robotics

Purpose of review: We review recent advances in algorithmic development ...

Koopman-based Control of a Soft Continuum Manipulator Under Variable Loading Conditions

Controlling soft continuum manipulator arms is difficult due to their in...

Robust Data-Driven 3-D Shape Servoing of Unmodeled Continuum Robots Using FBG Sensors in Unstructured Environments

In this paper, we present a novel and generic data-driven method to serv...

I Introduction

Grasping and manipulation are fundamental functions of robotics. Traditional robotic grippers consist of a set of mostly rigid joints and links for a variety of tasks such as perching [backus2014design, liu2020adaptive], digging [tong2021dig, danfeng2010research], sorting [gupta2012using, wu2019coordinated], and others [mucchiani2021dynamic, kim2019shallow]. Soft grippers offer flexibility and dexterity, and have gained momentum recently (e.g., [shintake2018soft, giannaccini2014variable]). Various applications have been increasingly using soft grippers, such as minimally invasive surgery [rateni2015design], agriculture [navas2021soft], human rehabilitation [zhang2017design], as well as manufacturing techniques (3d printing [tawk20193d, wang20163d], silicone molding [liu2020two, zhou2017soft], surface and shape deposition [suresh2015surface]), and others [gao2021soft, roels2019multi, glick2018soft].

Soft robots (including soft grippers) have garnered significant attention and grown into an important research topic, one of the prevailing challenges relates to their modeling and control [hughes2016soft]

. Due to the infinite degrees of freedom of soft robots, accurate first-principles-based models are hard to derive. Moreover, in most cases, system non-linearities pose additional difficulties to control design. One way to resolve such constraints is by considering data-driven methods.

Various methods based on neural networks have been proposed for modeling and control of soft grippers. The authors in 


show the efficiency of recurrent neural networks in terms of grasp success and stability using a shape-memory soft gripper with integrated hybrid microelectronic sensing skin. Uniformly sampled x-y positions and z axis orientations are considered as candidates in the dataset while no closed-loop control is implemented. In 


, a highly compliant multi-fingered hand is controlled by inflating or deflating chambers to demonstrate tasks that human’s hand could do, e.g., turning a valve and grasping a bottle from a table. The control algorithm is designed based on imitation learning and reinforcement learning that utilize measurements and experience to update the policy. Another work 

[khin2021ML] tries to estimate objects’ instability and slippage due to acceleration of the robot or insufficient grasp strength using neural networks to improve grasp performance.

Fig. 1: Instance of the soft gripper considered herein grasping a soft ball using the proposed online learning-based method. The method is designed to be agnostic to the shape and/or weight of the object to be grasped.

An alternative data-driven direction approach, which is also pursued in this paper, is based on the Koopman operator theory. The Koopman operator theory can be used to map a complicated nonlinear system into a higher-dimensional linear system approximating the unknown nonlinear dynamics directly from data. Thus, it is appropriate for modeling and control of soft robots [kaiser2020Overview]. For example, the highly nonlinear dynamics of a soft robot consisting of three pneumatically-driven actuators is identified by Koopman operator [bruder2019Ksoft]. The method constructs the relationship between PWM signals corresponding to actuator pressure and position/velocity of the end-effector; learned model of this form can then be used directly for control design. In [haggerty2020Ksoft], Hankel Dynamic Mode Decomposition is used to estimate the Koopman operator. The learned model is then used in a Linear Quadratic Regulator (LQR) to control a soft robot arm to reach a target position. The Koopman operator theory is also implemented in the modeling of a soft robotic swimmer in simulation [castano2020Ksoft].

In this paper, we propose an online modeling and control approach for soft multi-fingered robotic grippers based on Koopman operator theory. First, dictionaries of lifting functions are designed based on the workspace of the soft gripper. Then, instead of implementing the meta-learning based Koopman identifier [mazouchi2021online], ACD-EDMD [shi2021ACD-EDMD] is used to learn and update the model at every time step in real-time without any complicated dictionary learning. Updated models, in turn, provide the dynamics constraints within an MPC structure to help achieve desired control objectives.

The paper’s contribution are twofold:

  • We propose an online modeling approach that utilizes the Koopman operator theory. An online control algorithm is designed that uses the learned and continuously-refined models in real-time and adapts control inputs for an MPC structure accordingly.

  • We evaluate and compare via physical experiments the prediction accuracy of different modeling methods among different datasets using two soft grippers. The proposed online modeling and control algorithm is implemented in the grippers to grasp different objects of varying shape and weight, without any explicit information provided to the algorithm a-priori.

Ii Preliminary Material

Ii-a Koopman Operator Theory and Extended Dynamic Mode Decomposition (EDMD)

The Koopman operator theory [koopman1931koopman] details how to construct an infinite-dimensional linear operator that governs the evolution of observables ; the latter is also called the “lifted” variables of original states . Then, the original nonlinear system

can be represented by Koopman modes, eigenvalues and eigenfunctions. Consider the nonlinear system

, with . The evolution of observables using Koopman operator can be written as Decomposition of the full state observable using Koopman modes , eigenvalues and eigenfunctions , yields [williams2015EDMD]


Given state history (commonly referred to as “snapshots”), can be expressed as a finite-dimensional approximation of the Koopman operator via EDMD. To do so, we need a dictionary of functions that lift state variables to a higher-dimensional space where observable dynamics is approximately linear.

Given a dictionary , the Koopman operator can be approximated by minimizing the total residual between snapshots,

. This can be solved by truncated singular value decomposition, yielding


with , and denoting pseudoinverse, transpose and conjugate transpose, respectively. With via (2), we obtain


where is the

-th eigenvector,

is the -th left eigenvector of scaled so , and is a weight matrix such that  [williams2015EDMD].

The evolution of the original nonlinear system using the estimated Koopman operator is described by replacing expressions (3) into (1). Control inputs can be readily incorporated to the definition of as an augmented state [proctor2018EDMDc].

Ii-B Acd-Edmd

One of the most critical steps that affects the performance of Koopman operator-based methods is how to choose a proper dictionary for lifting the original states, . In our previous work [shi2021ACD-EDMD], we have proposed ACD-EDMD as a general and analytical approach to formalize the construction of the dictionary of lifting functions based on fundamental dynamical system properties, namely the configuration space of a rigid robot and the workspace for a soft robot. Hermite polynomials are the basis functions for Euclidean spaces, whereas for non-Euclidean spaces, state variables are mapped into a higher Euclidean space. Then, the dictionary of states is expressed as the Kronecker product of the lifting functions of all these lower-dimensional spaces. For the control input, is computed by the zero- and first-order Hermite polynomials and Kronecker products thereof similarly to the cases for states. Finally, the complete dictionary is formed as the Kronecker product between lifting functions for states and inputs, i.e. . With the proper design of the lifting functions, we can then apply the algorithm introduced before in Section II-A to extract a model of the robotic system at hand using the Koopman operator theory.

Iii Soft Gripper System Design and Properties

In this work we consider soft multi-fingered robotic grippers that are pneumatically actuated. A rigid sub-system is used for gross transport and positioning of the wrist plate, which contains the soft fingers (actuators) that are responsible for object grasping (Fig. 1). Note that motion of the rigid sub-system is not included in the model we aim to learn. Instead, we only consider the grasping behavior of the soft fingers in this paper.

Iii-a Soft Gripper Design and Integration

The soft gripper comprises three identical bending actuators [kokkoni2020finger] (Fig. 2) which are retrofitted PneuNets [shepherd2011multigait]. These actuators use pneumatic pressurization and depressurization applied to in-series connected deformable chambers to create rotational motion. The soft actuators are made via silicone (Smooth-On Dragon Skin 30) casting in 3D-printed molds (with carbon-fiber reinforced Onyx filament), and are attached to 3D-printed plates. We consider two distinct plates for arranging the soft fingers, one where the three fingers are arranged in a symmetric manner and one that is asymmetric (Fig. 3). The soft gripper assembly is attached to the ReactorX 150, a rigid manipulator with four links and four joints.111 The original fifth joint of the robot is a rigid pincher which is removed for the purposes of this work. The plate acts as a wrist affixed to the manipulators endpoint.

The rigid sub-system serves a dual purpose: first to move the soft gripper to a desired position and then to lift the soft gripper after it has grasped an object. Note that the focus of this work is on the soft gripper, assessing how our proposed Koopman operator-based algorithm can enable the soft gripper to grasp objects of different shapes and weight without explicit knowledge of. To evaluate grasping success, we quantify if the gripper can hold the object as it is being lifted up by the rigid manipulator.

Fig. 2: Soft finger depressurized (left), idle (middle) and pressurized (right).
Fig. 3: CAD renderings of the rigid attachment plates considered herein that afford symmetric (left) and asymmetric (right) soft finger arrangements.

Iii-B Pneumatic Actuation

The soft fingers are pneumatically actuated using Programmable-Air boards (Fig. 4). The board is based on the Arduino nano (ATMega328P), and has a maximum pressure of and flow rate of . Pressurization is controlled by changing the PWM signal sent to pumps. Two such boards were considered for the experiments; the detailed configuration is discussed in Section V.

Fig. 4: The Programmable-Air Pneumatic Control Board.

Iii-C Workspace Construction

As discussed in [shi2021ACD-EDMD], the workspace of soft robots needs to be defined to obtain the dictionary of lifting functions, which is used in the Koopman operator-based modeling approach. The workspace of a robot is the set of positions that it can reach. For soft multi-fingered grippers, the position of the tips of the fingers can be considered to construct the workspace. The state of the soft gripper is thus defined by the concatenation of the positions of each tip as , , and , expressed in terms of an inertial frame set at the center of the plate (Fig. 5).222 Although this work uses a -fingered soft gripper, the proposed approach can be directly extended to account for general -fingered soft grippers.

We can further simplify the workspace used for operator learning by exploiting the physical constraints underlying the

state vectors. The types of actuators considered herein are constrained to create planar motion and can only bend about the normal vector to the plane they are constrained to create motion at (which is determined herein based on the arrangement of the rigid attachment plate). This way, only one of

for each finger suffices, thus giving rise to a workspace for the whole soft gripper of . Without loss of generality, here we choose as the representations of sub-workspace for each of the fingers, but any other state component would work as well. Besides the Kronecker product of sub-dictionaries, we also include in the library observations of other states and inputs for full-observability. We define . The dictionary is computed as . After eliminating redundant terms, the final dictionary contains terms.333 It is beneficial to seek to construct the smallest possible dictionary that best describes the system at hand for computational expediency. At times, doing so relies on exploiting engineering intuition and/or any system symmetries. In this work we take advantage of the bending-only motion afforded by the specific fingers we consider. However, our method is general and can directly apply to any type of soft fingers that afford unconstrained position (and orientation) control of their tips in full 3D space, albeit at the expense of increasing the size of the constructed dictionary.

Iv Online Modeling and Control

Unlike rigid robotic systems whereby it is known how to construct their dynamics, soft robotic systems possess complicated and, most often, infinite-dimensional dynamics. Thus, data-driven approaches may be a better fit for extracting models for soft robotic systems directly from data. However, models learned offline or in simulation (a technique often employed in relevant works) may be invalidated at runtime due to the presence of uncertainty [karydis2015IJRR], unless a very delicately designed experimental environment and operation process are imposed for the dynamics learned offline to remain valid for online model-based control. This process can get very tedious and is not scalable. Instead, online control should require limited data and limited time to enable online computation, which are key properties enabled via Koopman operator-based methods [mazouchi2021online, shi2020CCTA]. Thus, we propose an online modeling approach using Koopman operator theory to update dynamics constraints in real time and adjust inputs accordingly within an MPC structure.444 While we consider MPC in this work, the continuous refined learned model can be used in any other online model-based control algorithm.

Iv-a Online Model Predictive Control (MPC) Formulation

MPC determines control inputs at each time step by solving an optimization problem among a range of prediction horizon constrained by robot dynamics. The general optimization algorithm can be formulated as

In this work, the soft grippers’ dynamic constraints are obtained via ACD-EDMD in real time. In other words, at each time step, we solve an optimization problem using an updated learned model. The cost function is usually user-designed. Herein, we set a desired trajectory and minimize the difference between predicted states and the desired points. With these considerations in place, the MPC problem solved here at each time step is expressed as


Iv-B Online Modeling and Control Algorithm

We are now ready to present the proposed online modeling and control algorithm (Alg. 1). After initialization, at each time step , control signals are obtained from the MPC controller with the updated model constraints learned based on data from the previous time step. Then, the physical robot is driven by the control inputs and observations are collected to the online training dataset. A refined model estimated from the updated dataset via ACD-EDMD is then used to create updated dynamic constraints for the following step. The process is repeated until the control task (here grasping different objects) is achieved.

V Experimental Testing and Results

To test the efficiency of our online modeling and control algorithm, we evaluate separately modeling accuracy and control performance. First, we compare the performance of different model-estimation approaches in online and offline datasets to show the benefits of training with online observations. We further compare the ACD-EDMD based online modeling estimation accuracy with SINDy (Sparse Identification of Nonlinear Dynamical Systems (SINDy) [brunton2016SINDy]

) and LSTM (Long Short-Term Memory 

[hochreiter1997long]) using online training datasets. Then, we implement our online control algorithm in the soft gripper and evaluate it in grasping different objects.

Fig. 5: Coordinate systems and motion capture marker configuration.
1 initialize: Evolve the system with random valid inputs for the first time steps. Then, compute the estimated Koopman operator from the observations:
2 for  do
3       repeat
4             MPC Controller: Calculate the control inputs with the model constraints learned in the previous step by solving optimization (4).
5             Plant: Propagate the soft robotic gripper with : , where is the evolving law of real plant.
6             Model: Collect new data from observations and update the ACD-EDMD model.
8      until control task is finished;
Algorithm 1 Online Modeling and Control
Method Dataset MSE() MSE() MSE() (magnitude)
SINDy Offline (Unloaded)
ACD-EDMD Offline (Unloaded)
SINDy Offline (Loaded)
ACD-EDMD Offline (Loaded)
SINDy Online
LSTM Online
TABLE I: Prediction accuracy of different approaches

Motion capture system (12-camera Prime13 Optitrack) is used to collect position data (, , ) of the tip of each finger. To ensure consistency among all training data, we set the origin of the inertial frame at the geometric center of the attachment plate (Fig. 5). In both of the grippers, one control board controls soft fingers and , and the other control board controls finger . Thus, together with the PWM signals of two control boards, states and inputs of the robotic soft gripper are observed.

V-a Settings of Different Approaches to Compare Against


SINDy is a data-driven approach to extract governing equations of nonlinear systems. It builds on the assumption that only a few key nonlinear candidates governing the evolution of original system. Sparse regression determines weights of those nonlinear terms from observation data. Here we adopt the example in [brunton2016SINDy] as the library of candidate nonlinear functions, consisting of constant, polynomial and trigonometric terms, that is

The sparse regression problem is solved by least absolute shrinkage and selection operator (LASSO) [tibshirani1996LASSO].

V-A2 Lstm

LSTM is an artificial recurrent neural network that offers feedback connections and the ability to deal with entire datasets rather than single data points. LSTM networks have been used in robotics applications (e.g., [park2018multimodal]), notably in-hand robotic manipulation [andrychowicz2020learning]

. In this paper, we use the LSTM layer in MATLAB R2020a Deep Learning Toolbox to construct the network. A single layer of 20 hidden units is applied as designed in 


V-B Model Estimation Accuracy

Different datasets for training are collected as follows.

  1. Unloaded Offline training data: In offline training, constant PWM signals are given by the control board. Each signal lasts for sec with a sampling rate of Hz ( sec). Preliminary testing revealed that the finger can function effectively when the PWM is within the range of to . Thus, we collect the position and voltage signals data by either pressurizing or depressurizing all the fingers when the gripper is unloaded within that range. A total number of sets are collected as the ‘Unloaded Offline’ training data. A moving-average filter with window size of is applied to the offline training data.

  2. Loaded Offline training data: Data from four trials of grabbing plush oranges (object in Fig. 6), two trials of grabbing the smaller rectangular box (object in Fig. 6) and two trials of grabbing the larger square box (object in Fig. 6) are collected with length of points for each trial. The ‘Loaded Offline data’ case is constructed as a combination of these data sets.

  3. Online training data: Online observations are collected when the gripper attempts to grasp the plush orange. Let be the training data length; every observations are used to predict .

The three aforementioned datasets are used to predict for the whole online data trial of length . We calculate the Mean Squared Error (MSE) between predicted (superscript ) and true (superscript ) states per for the whole trajectory. Results are contained in Table I. For all algorithms, predictions computed from the offline loaded data are more accurate than those made based on unloaded data. Further, online approaches lead to significantly smaller error than all offline approaches.

We also compare the efficiency of all methods with online training data by evaluating repeated grasping of object for times to avoid bias. We tested all three approaches with different training data size, .

Results are depicted in Table II (training time) and Fig. 7 (prediction accuracy). Our algorithm can run and orders of magnitude faster than SINDy and LSTM, respectively, per each time step. This performance is attainable because ACD-EDMD leverages fundamental characteristics of the dynamical system at hand (herein the workspace of the soft gripper) which leads to a simultaneously smaller and more descriptive dictionary to use. Further, with reference to Fig. 7, our method can achieve similar accuracy with LSTM for smaller training data size, and comparable accuracy for larger training data size. (More specifically, our method can lead to more variable accuracy compared to LSTM for , but mean values are close.) The combination of comparable accuracy but substantially faster training time makes our proposed approach appropriate for online learning-based modeling and control tasks.

Fig. 6: Objects of different shape and/or weight considered in this work. Sample trials can be viewed at
Training Size \  Approach ACD-EDMD SINDy LSTM
s s s
s s s
s s s
TABLE II: Training times per each time step for different approaches

The bottom row panels of Fig. 7 show that the prediction error obtained by SINDy is much larger than the other two approaches.555 For error bars to be visible, we omitted those of SINDy in . SINDy’s worse than expected performance may be because of the small training data size used in the online procedure, or perhaps the choice of candidate nonlinear functions selected based on the original publication [brunton2016SINDy]. This is in fact one key point of departure from SINDy in that it requires selection of candidate nonlinear functions and that selection can affect the overall performance.

Fig. 7: - error prediction accuracy of all online methods. SINDy (blue) has large errors; to keep the bars visible, one case () is shown.

V-C Online Modeling and Control: Grasping

For the final evaluation of our method, we apply Alg. 1 to grasp objects using the two soft grippers with different finger configurations. The online training size is and the prediction horizon is set at . In this experiment, we manually designed desired trajectories for each finger.666 Note that this experiment is only one example to show how our online modeling and control approach can be implemented in practice. Other model-based controllers and desired trajectories can be used as well, and one can embed our online modeling method into their design directly, which is one key benefit of our proposed method. The same desired trajectory is used for grasping objects of different shape and same shape but different weight (Fig. 6), without any information about the objects. Each grasping task is repeated times to avoid bias. Successful grasps are considered those that the gripper grasps the object, holds it until fully lifted and keeps it for sec after lifting.

Results show that the asymmetric gripper functions better than the symmetric one in terms of grasping heavier objects, whereas the symmetric gripper performs better on grasping objects of varying shape (Table III). However, we can still deduce that our algorithm can be easily implemented into different robots to grasp different objects without requiring any knowledge about the robot and the objects.

Vi Conclusions

Soft multi-fingered grippers have been gaining momentum across applications. However, because of their infinite-dimensionality and non-linearities, necessary modeling and control algorithms are difficult to derive. The online modeling and control method proposed in this paper relies on Koopman operator theory for data-driven modeling and online MPC for control. Our proposed method can achieve solid prediction accuracy while training significantly faster compared to related data-driven methods currently in-use.

Our method can be directly embedded into model-based online control of robotics systems, without requiring explicit knowledge of system dynamics except for their configuration space or workspace. Results by testing with two soft 3-fingered robotic grippers attest that our method can grasp a variety of objects of different shape and/or weight without any a-priori knowledge. This work can be beneficial in applications like smart manufacturing and precision agriculture.

Object 1 2 3 4 5 6
Gripper Symmetric Gripper
Success Rate
Gripper Asymmetric Gripper
Success Rate
TABLE III: Success rates of grasping different objects

Future work aims to combine the online modeling approach with other model-based controllers (e.g., [Mucchiani2022closed]), and develop an approach for generating and adapting online the desired trajectories to grasp a wider range of objects.