Understanding Dynamic Tactile Sensing for Liquid Property Estimation

by   Hung-Jui Huang, et al.
Carnegie Mellon University

Humans perceive the world by interacting with objects, which often happens in a dynamic way. For example, a human would shake a bottle to guess its content. However, it remains a challenge for robots to understand many dynamic signals during contact well. This paper investigates dynamic tactile sensing by tackling the task of estimating liquid properties. We propose a new way of thinking about dynamic tactile sensing: by building a light-weighted data-driven model based on the simplified physical principle. The liquid in a bottle will oscillate after a perturbation. We propose a simple physics-inspired model to explain this oscillation and use a high-resolution tactile sensor GelSight to sense it. Specifically, the viscosity and the height of the liquid determine the decay rate and frequency of the oscillation. We then train a Gaussian Process Regression model on a small amount of the real data to estimate the liquid properties. Experiments show that our model can classify three different liquids with 100 volume with high precision and even estimate the concentration of sugar-water solution. It is data-efficient and can easily generalize to other liquids and bottles. Our work posed a physically-inspired understanding of the correlation between dynamic tactile signals and the dynamic performance of the liquid. Our approach creates a good balance between simplicity, accuracy, and generality. It will help robots to better perceive liquids in different environments such as kitchens, food factories, and pharmaceutical factories.


page 1

page 5

page 6

page 7


Extended Tactile Perception: Vibration Sensing through Tools and Grasped Objects

Humans display the remarkable ability to sense the world through tools a...

Tactile Sensing with a Tendon-Driven Soft Robotic Finger

In this paper, a novel tactile sensing mechanism for soft robotic finger...

Dynamic Modeling of Hand-Object Interactions via Tactile Sensing

Tactile sensing is critical for humans to perform everyday tasks. While ...

Walking on TacTip toes: A tactile sensing foot for walking robots

Little research into tactile feet has been done for walking robots despi...

Interpreting and Predicting Tactile Signals via a Physics-Based and Data-Driven Framework

High-density afferents in the human hand have long been regarded as esse...

Learning Intuitive Physics with Multimodal Generative Models

Predicting the future interaction of objects when they come into contact...

GelSlim3.0: High-Resolution Measurement of Shape, Force and Slip in a Compact Tactile-Sensing Finger

This work presents a new version of the tactile-sensing finger GelSlim 3...

I Introduction

Perceiving and understanding the physical world is a core capability of intelligent robots. Among the various sensing modalities, tactile sensing is typically used to understand physical interactions. For example, humans learn about an object’s hardness by pressing it [17] and a liquid’s viscosity by shaking its container. They achieve these tasks using dynamic tactile sensing, which focuses more on changes of the signal across time. Dynamic tactile sensing is in contrast to static tactile sensing, which focuses on understanding the signal (force distribution, contact surface deformation) at one shot (e.g., localizing contact). Despite the importance of dynamic tactile sensing, its use for robots is not as well studied as it is for humans.

Among the common perceptual tasks requiring contacts, we study robot dynamic tactile sensing by tackling the task of noninvasively estimating the properties of liquid inside a container. Liquid properties are often closely related to their dynamics. For example, liquid viscosity refers to the restraining force when liquid flows, which requires us to focus on the temporal dimension of the dynamic tactile signal. Further, liquid property estimation has a wide robotics application. For example, monitoring viscosity online provides immediate feedback for operations in the food processing industry [2]. However, it remains a challenge for robots to reliably estimate liquid properties without using specialized tools like viscometers.

Fig. 1: (a) The fluid in a bottle oscillates after a lateral perturbation. (b) We build a robot setup to shake containers. (c) Tactile signal from finger-tip GelSight sensors after the perturbation. (d) The principal motion of the GelSight markers after a perturbation on the bottle.

There are a number of works on content property estimation [1] [4] [13] [14]. Inspired by humans, most approaches choose to shake the container and apply data-driven models to the received dynamic tactile signal. They show results on classifying to solids or fluids in a container with to accuracy using a variety of sensors. These works are limited to classifying objects with very different physical properties (e.g., water and glycerine). Physics-based approach with sophisticated fluid dynamics analysis [13] achieved accuracy but only work on cylinder-shaped containers with precisely-measured size. Both data-driven and physics-based approaches require that content weight and volume are either fixed [6] [14] or pre-estimated by other methods [13]. They also cannot generalize to different-shaped containers.

In this work, in addition to this content classification task, we tackle a much more challenging task: estimating the viscosity and liquid volume given a bottle of sugar-water solution. It requires distinguishing sugar-water solutions with different concentrations based on slight differences in viscosity, agnostic to the volume of liquid in the container.

We interpret the liquid-property-relevant tactile signal with a physics-inspired approach, which complements physics-based with data-driven modeling to leverage the strengths of both. By understanding how dynamic tactile signal is generated, we construct a model using simple physics. In addition, the complex, unmodeled effects of fluid dynamics in irregularly shaped containers are modeled by a data-driven method. Our method starts by approximating our liquid-container system as a second-order system based on our observation that the movement of the liquid surface after perturbation is a damped oscillation. Based on physics, We show that the oscillation frequency and decay rate of the damped oscillation are directly correlated with the liquid’s height and viscosity. Therefore, the robot in our method first lifts, perturbs, and holds still the liquid container, and then collects the dynamic tactile signal that reflects the damped oscillation movement of the liquid. With our high-precision, vision-based touch sensor GelSight [19], this damped oscillation movement is precisely detected after filtering, as shown in Fig. 1d. We then learn a data-efficient model which takes the decay rate and the oscillation frequency of the dynamic tactile signal, and infers the liquid viscosity and volume (quantified as height) with very high precision.

Contributions: Our main contribution is solving a dynamic tactile sensing task using an approach that maintains a good balance between physics-based and data-driven modeling. We also contribute to estimating liquid properties with higher precision and greater generalizability. Our model achieves % classification accuracy on water, oil, and detergent. More importantly, our approach can estimate sugar concentration (ranging from to wt%) with wt% error and liquid height with mm error, using only tactile sensors. Note that wt% sugar-water solution is less viscous than whole milk. Our height estimation is, in fact, so precise that the error may come from measuring the ground truth by rulers. Our method is data-efficient ( training datapoints), agnostic to liquid volume, and can be easily generalized to different containers with similar shapes ( fine-tuning datapoints). We believe approaches balancing physics-based and data-driven modeling can also be applied to other dynamic tactile sensing tasks (e.g., rubbing a surface to estimate its texture) since they often involve complex dynamics that cannot be purely modeled by physics or learned with limited real-world data.

Ii Related Work

Ii-a Dynamic Tactile Sensing

Dynamic tactile sensing is important in various robotics tasks related to perception, manipulation, locomotion, etc. [3] [7] [11]. For example, Sinapov et al. [16] used a three-axis accelerometer to detect 20 different surfaces by performing 5 different exploratory scratching behaviors. They can recognize the surfaces with 80% accuracy based on spectro-temporal features. Taunyazov et al. [18] used dynamic tactile sensing to achieve grasp stability prediction during object handover and accurate food identification through tools. They also showed the ability to accurately localize taps on an acrylic rod based on vibrations caused by the tapping. Giguere and Dudek [5] used a tactile probe which is passively dragged along a surface for surface identification. They designed multiple features in 4-s time windows of dynamic tactile data and classified 10 different types of terrains.

Ii-B Content Estimation

Content estimation, or estimation of the object inside a given container, can be thought of as a specific type of object recognition. However, unlike other object recognition tasks, content estimation does not involve direct visual or physical access to the object, making the task much more challenging.

There are multiple previous works utilizing different modalities to solve this task. For example, Chen et al. [1] used accelerometers and contact microphones to classify the mechanical parts in the container by shaking it. Güler et al. [6] combined visual and tactile data to identify five types of contents. They used static deformation information of the paper container and achieved a 95% classification accuracy. Saal et al. [14] used tactile data recorded during shaking to estimate the viscosity of different liquids with the same weight. Matl et al. [13] used the force-torque sensor to estimate liquid mass, volume, and viscosity. They achieved 98% accuracy in classifying water, oil, and honey. Jonetzko et al. [9] integrated tactile and auditory data generated by shaking to classify 8 types of pills in 4 weight groups, achieving an average accuracy of 91%. Eppe et al. [4] elicited auditory information by shaking to classify content types and predict the weight. Jin et al. [8] used auditory information generated by rotation to classify 20 kinds of particles in the container.

In previous works, researchers mainly focused on solid contents, which are easier to recognize. For example, [1] used different mechanical parts such as bearings and nuts; [15] used glass, rice, beans, and screws. Some works target liquids, but only study very different liquids, such as yogurt and water [6], or water and glycerine [14]. In this work, in addition to classifying different liquids, we also target a much more challenging task: precise regression on liquid height and viscosity.

In addition, most works adopted a purely data-driven approach and the learned model is specific to a certain group of weight and container. The only counter case we found is [13]

, where the authors used an analytical model of liquid sloshing dynamics to estimate the properties of liquid in cylinder containers. However, the model requires the exact geometry of the container. In contrast, our physics-inspired model, combined with machine learning, is able to generalize to different liquids and containers and is agnostic to the amount of liquid. Besides, our approach is data-efficient and achieves better volume estimation and classification results on many shapes of containers without knowing their exact dimension.

Iii Dynamic Model of Liquid and Tactile Signal

Iii-a Overview

We aim to estimate the viscosity and volume of the liquid in a bottle using only tactile sensors. Typically, solving this problem involves two steps: initiating the dynamics of the liquid-container by an activation motion, and estimating the liquid properties using the collected dynamic tactile signal. Our method studies, models, and parameterizes the interacting tactile force between the sensor and liquid-container. The correlation between the parameterized forces and the targeted liquid properties is further learned in a data-driven way.

Another key component of our approach is the way we activate the system. While shaking the liquid-container is the most popular activation motion, it causes complex, nonlinear dynamics like turbulence. Inspired by the importance of step response to system characterization, we command the robot to perform a “unit step input”: laterally perturbing the liquid-container and then holding it still. The key is to collect dynamic tactile signals only in the “hold still” stage, where the system dynamics is much simpler and the signal is cleaner (Fig. 1d). In the rest of this section, we explain and model the simple physics principles governing the received dynamic tactile signal collected this way.

Iii-B Linear Model for Liquid Oscillation

After perturbation, the liquid will oscillate in the container. The viscous force of the liquid, which dissipates energy from the system, causes the oscillation to decay. To model the oscillation, we consider a simplified system with liquid oscillating in the container in a simple mode, as shown in Fig. 2. The length of the container is , the height of the liquid is , and the mass of the liquid is . The entire container is actively held still in the air by a gripper with GelSight sensors attached. The origin of the system coordinate is set at the center of mass of the liquid body before oscillation. With the small offset on the surface level from the steady-state, the center of mass of the liquid body is at and its velocity is . Assume the liquid’s internal viscous force can be treated as an imaginary viscous force applied on the center of mass of the liquid body. We can write the principle of work and energy as:


The first and second terms are the kinetic and potential energy of the system, and the sum is the total energy. The RHS of Eq. 1 is the energy dissipation rate caused by the viscous force. When the system perturbation is small, the viscous force is approximately linear to the velocity [10] of the center of mass and to the mass : . The constant factor is determined by the liquid viscosity. For example, compared to water, oil has a stronger viscous force and larger , so the system energy dissipates faster. By replacing in Eq. 1 with , we get a linear second-order system:


When the liquid is not too viscous ( is not too large), the solution of Eq. 2 is a linear damped oscillation signal:


where the decay rate and the oscillating frequency . The initial condition (the perturbation motion) determines the magnitude and the phase . The solution matches our observation that after lateral perturbation, the liquid surface oscillates but gradually damps away.

Fig. 2: Side view of liquid oscillating in a container. The center of mass of the liquid body is at in this perturbed state and in the steady-state.

As the center of mass of the liquid body oscillates, the robot gripper applies a lateral force on the side of the container to keep the container’s position. The reaction force of is applied on the GelSight sensors, which record the force. The center of mass of the liquid body follows Newton’s second law in the direction:


Taking the double derivative of the damped oscillation signal in Eq. 3, we have in Eq. 4 being a similar signal with the same decay rate and oscillation frequency but a different magnitude and phase. Therefore, the dynamic tactile signal is:


which oscillates in the same pattern as the liquid surface in Eq. 3 with a different magnitude and phase . Eq. 5 is our physics-inspired model and it explains the damped oscillating form of the dynamic tactile signal in Fig. 1d.

Based on our model, the dynamic tactile signal has decay rate , which directly relates to the liquid viscosity, and oscillation frequency , which directly relates to the height of the liquid and liquid viscosity. It matches our observation that a more viscous liquid causes faster oscillation decay, and liquid with more volume oscillates at a higher frequency. In practice, our model, like many other physics models, is a simplification of the real world. For example, the decay rate is non-linearly influenced by the shape of the container. We leave all the complex, unmodeled dynamics to be learned from the data-driven part of our approach (Section IV-C).

Iii-C Non-linear Energy Analysis

Our approach estimates liquid viscosity and height using and extracted from the dynamic tactile signal. However, extracting and by fitting the signal using Eq. 5 is not precise, because Eq. 5 does not fit the signal well along the entire time range. As shown in Fig. 1d, the signal decays faster at the beginning, not at the same rate throughout as modeled by Eq. 5. This is because the assumption that the viscous damping force is linear to the velocity of the center of mass does not hold when the system energy is high. In the linear case that , the energy dissipation rate is proportional to the system energy on average, so the system energy decays exponentially. When the system energy is high, the viscous damping force has higher order terms such as , and the energy dissipates faster than linear decay. Therefore, the system energy will decay faster than the exponential decay at the beginning, but behave approximately as an exponential decay once the system energy is lower. We take this affect into account when extracting and from the dynamic tactile signal.

Iv Method

Fig. 3: Pipeline of our method to estimate the viscosity and height of the liquid. We collect the tactile data during the free oscillation period after a lateral movement. We first track the marker motions from the GelSight [19] attached to both fingers. Then the top 16 markers with the largest motions are selected and a principal motion signal is calculated by taking PCA of their motions. We fit the principal motion to Eq. 6 and extract the oscillation frequency and decay rate as the features. Then we fit a Gaussian Process Regression model taken and as input to estimate the liquid height and viscosity.

In this section, we introduce our liquid properties estimation approach based on the physics-inspired liquid-container model presented in Section III. Our approach first retrieves and de-noises the dynamic tactile signal that represents the lateral force from the GelSight images, and then extracts and from Eq. 5. After that, we calculate the height of liquid , liquid viscosity , and other related physical properties of interest from the damping model with a data-driven model. The pipeline is shown in Fig. 3.

Iv-a Tactile Data Collection and Processing

We use two GelSight sensors on a two-fingered gripper, one on each finger, to collect the dynamic tactile signal due to its high resolution. GelSight sensor is an vision-based tactile sensor consisting of a soft elastomer with printed markers, a lighting system, and an embedded camera. On contact with objects, the camera captures micron-level deformation of the sensing surface, which is an elastomer with markers [19]. It can estimate shear force distribution by tracking the markers’ lateral motions in the image sequence collected by the embedded camera. When the liquid-container is held still after a lateral perturbation, we collect a sequence of GelSight images as the liquid oscillation motion subsides.

We extract marker contours in GelSight images by color segmentation and track their motion at sub-pixel accuracy. Since we are interested in signals in the low-frequency range, a Hz low pass filter is applied on the marker motions for noise reduction. Marker motions are then ranked by their magnitude and only the top markers that are supposed to be in the center of contact are preserved. We concatenate the x, y motion of each marker to get a

-dimensional vector at each time step, and apply PCA to find the

-dimensional principal oscillating direction. In our case, the percentage of variance explained by the first principal component is above

. We project the dimensional vector at each time step to our principal oscillating direction and get our 1D de-noised dynamic tactile signal that summarizes the damped oscillation motion of the top markers. This dynamic tactile signal is proportional to the shear force applied on GelSight, which is denoted by in Eq. 5.

Iv-B Fitting Linear Physics-Inspired Model

Although the de-noised dynamic tactile signal should be in the form of linear damped oscillation (Eq. 5) in the ideal case, in practice, our signal is the sum of three main components: a linear damped oscillation , a fast decay damped oscillation , and a time-dependent offset . As discussed in Section III-C, the dynamic tactile signal can be modeled as a linear damped oscillation as in Eq. 5, if we leave out the first few seconds where the system energy is high. However, the system starts with high energy, where the non-linear terms dominate the damping force and the signal decays faster. Therefore, we approximate with an additional term: a linear damped oscillation with a faster decay rate that only plays a part when the system energy is high. We also add an offset that is caused by in-hand slippage. is approximated as a quadratic function of time. In summary, we want to fit our dynamic tactile signal with a parameterized function :


where . Since by definition decays faster than , the model is constrained by . To find

, we minimize a loss function

that defines an error metric between and . While in Eq. 6 consists of unknown parameters, only and will be used to estimate the liquid properties. In order to better extract and , we care more about fitting the latter section of the signal well, where dominates the signal. Therefore, we choose a loss function that weighs the latter section of the signal more:


where is a small value for numerical stability. We perform optimization to minimize the loss function to get .

Iv-C Regression on Liquid Properties

Based on our physics-inspired model, the fitted parameters and in are directly related to liquid physical properties: viscosity and height. We learn a model to approximate the complex dynamics that relates and to the physical properties. Using and as features, we propose a parameterized quadratic regression model and a non-parameterized Gaussian Process Regression (GPR) model to estimate liquid height and viscosity . Although one might consider estimating liquid height by measuring its weight using force-torque sensors, this approach requires a known liquid density, which can vary in a wide range (from to for sugar-water solutions used in our experiments).

Our analysis in Section III-B suggests that and , which means . This motivates us to use quadratic models of and to estimate both and . Here, liquid viscosity is transformed to space as in [13] and [14]. In practice, the irregular shape of the container and other unmodeled dynamics can cause some complex effects that cannot be fully captured by the quadratic model. We therefore propose to fit a GPR on and to estimate both and , which can capture those higher-order effects. In our experiments, we use GPR with a mixture of RBF kernel and white kernel. Thanks to the low-dimensional physics-inspired feature space

, both the quadratic and GPR models predict liquid properties well without the need to use a more sophisticated model such as neural networks. In addition, our quadratic model has only

parameters and our GPR model also requires very little data.

Iv-D Generalizing to Containers with Similar Shapes

A major advantage of using our physics-inspired features is that we can generalize the property prediction models trained on one container to another container of similar shape, with only a few fine-tuning datapoints. Our experiments show that and , as functions of and in our trained model, have similar surface landscapes when trained with containers of similar shapes (as an example, compare Fig. 9(b) and Fig. 11(b)). To gain an intuitive understanding of this similarity, we can visit the relation in our physics-inspired model. When the size of the container changes, is simply scaled. We extend this observation to the estimation of viscosity and verify it in the experiments.

Assume we have container A and container B with similar shapes but different sizes. Given the trained model for container A, we parameterize the model in container B to have a similar landscape to :


where are the shifting and scaling parameters. Instead of fitting a model for container B from scratch, we only need to fit the four parameters , which require only a small amount of data.

V Experiments

In the experiment section, we want to answer the following questions: (1) How well does our approach differentiate liquid contents that are similar to ones used in the related work [13] [14]? (2) How beneficial is it to use GelSight? (3) How well does our approach perform in the challenging liquid property regression task? (4) Can our approach generalize to other liquids or other containers?

The experimental setup is shown in Fig. 4a. We use a DOF robot arm (UR5e by Universal Robotics) attached with a 2-fingered gripper (WSG50 by Weiss Robotics) to perform the activation motion. The dynamic tactile signal is captured by two Fingertip GelSight devices [12] mounted on both fingers of the gripper. The elastomer on GelSight has markers printed in a region with an average distance of from each other. The camera in the sensor streams videos with pixels resolution at . We use a ubiquitous, grooved cuboid container (Fig. 4b) as the liquid container. For each trial, we manually place the container horizontally in the gripper. To prevent slippage, we place the container in the gripper at around the center of mass and cover the grasping region with tape to increase friction. We set the grasping force to be the largest force the gripper can apply, which is around 25N. After grasping, the robot arm lifts and activates the liquid-container with a short lateral movement. We then hold the system still and record the GelSight videos for another seconds.

Fig. 4: (a) A 2-fingered gripper with two GelSights is used to grasp the bottle. (b) The grooved cuboid container. (c) The cylinder container. (d) The cuboid container. (e) The glass container.

V-a Liquid Classification

In this first experiment, we demonstrate our approach on a simple classification task. Our task is to classify three types of liquid with different viscosities: water ( cSt), olive oil ( cSt), and detergent ( cSt). Ground-truth liquid viscosity is measured by a viscometer (NDJ-5S by JIAWANSHUN). In this task, liquid volume can vary anywhere from to full. We collect a training dataset of datapoints ( different volumes for each liquid type, uniformly spanned in the range) and a testing dataset of datapoints ( different volumes for each liquid type, randomly sampled in the range).

Fig. 5 shows examples of dynamic tactile signals received with the three types of liquids. The dynamic tactile signal is recorded only in the “hold still” stage after perturbation and is derived from GelSight marker motions. The signal decays fastest in detergent and slowest in water. Using the training dataset, we train an SVM classifier with an RBF kernel that takes and as input. The distribution of the testing dataset and the decision boundaries of the trained classifier are shown in Fig. 5(a). Our classifier successfully classifies the testing dataset with an accuracy, which outperforms the accuracy reported in [13]. The data in Fig. 5(a) is well divided into bands corresponding to the three types of liquid. Within each band, we note that datapoints with lower are associated with a lower height (Fig. 5(b)).

Fig. 5: Dynamic tactile signal when the container is half-filled with water, olive oil, and detergent, respectively.
Fig. 6: (a) Classification of water, olive oil, and detergent on the testing dataset. Colored regions represent the classification regions determined by the trained SVM. (b) Ocsillation frequency vs. liquid height for different liquid types.

V-B Comparison with Force-Torque Sensor Measurement

Since we take the lateral reaction force applied on GelSight as the input tactile signal, it comes naturally that one can use force-torque sensors to directly measure the force as in [13]. We mount a -axis force-torque sensor (HEX-E by OnRobot) on the wrist of the robot arm to measure the dynamic tactile signal. Fig. 7 shows the filtered lateral force in repeated trials. While using a sensor with a similar specification sheet as in [13]

, our force-torque sensor measurement has a much lower signal-to-noise ratio (SNR) and only achieves a classification accuracy of

using the method presented in [13]. This difference might come from that our container being smaller with a less regular shape than [13] or the different way we perturb the system. In contrast, the GelSight signals have a much higher SNR and achieve accuracy. This experiment underlines the benefit of using GelSight.

Fig. 7: Filtered lateral force measurement using a 6-axis force-torque sensor on the robot wrist after a perturbation. Data is collected with the same setup as in Fig. 5. We achieve classification accuracy using this signal, which is only slightly better than the accuracy when guessing randomly.

V-C Estimating Sugar Concentration and Liquid Height

We show the capability of our approach by demonstrating on a challenging experiment for liquid properties regression. We task our robot to estimate the concentration , of viscosity , and volume (quantified as liquid height ) of sugar-water solutions. The concentration vary from to wt%, making the viscosity vary from to cSt. We estimate viscosity in scale as in [13] and [14] because liquid viscosity is an exponential function of sugar concentration. Fig. 7(c) shows the relation between concentration and of viscosity. In this experiment, the height of the liquid varies from mm (25% full) to mm (75% full). Out of this range, the liquid oscillation is too weak for perception. We make sugar-water solutions of different concentration levels equally spanned from to wt% (Fig. 7(a)). For each concentration level, we collect data with different height levels equally spanned from to mm. Together, it forms the training dataset of size .

The testing dataset is made of another set of sugar-water solution of different concentration levels equally spanned from to wt% (Fig. 7(b)). For each solution, we setup the liquid-container system on the discretized liquid heights, same as the training dataset. We thus have testing setups. For each testing setup, we perform three independent tests. We then take the average and of the signals from the same setup to reduce variance. We use this testing method for the following experiments as well.

(a) (b)
Fig. 8: (a) Sugar-water solutions in the training dataset, concentration from wt% (left) to wt% (right) (b) Sugar-water solutions in the testing dataset, concentration from wt% (left) to wt% (right) (c) The relationship of of viscosity and concentration of the sugar-water solutions.
Fig. 9: (a) Dynamic tactile signal of half-filled containers with sugar-water solutions of different concentration. (b) Dynamic tactile signal of containers with wt% sugar-water solution at different liquid heights.

Fig. 8(a) shows examples of dynamic tactile signals when activating solutions of different concentration levels. Higher concentration levels result in faster signal decay. We also show the tactile signal for liquid at different heights in Fig. 8(b). The signal oscillates faster when there is more liquid is in the container. For quantitative measurement of the volume and viscosity, we train three GPR models taking and as input to predict , , and , as introduced in Section IV-C. The fitting errors on the training and testing datasets are shown in Table I. The prediction result on the testing dataset is shown in Fig. 10. We omit the prediction result of from Fig. 10 because is almost linear to and their results have almost the same shape. The result shows that the testing data is close to the fitted GPR plane, and this model can estimate , , and with MAE of mm, wt%, and . Note that the ground truth of the liquid height has a measurement error of roughly mm. Results using GPR models also outperform results using parameterized quadratic models shown in the Appendix. In the rest of the paper, all results will refer to results on the testing dataset.

(mm) (wt%)
GPR Model MAE (training dataset) 0.81 15.7 0.16
GPR Model MAE (testing dataset) 0.56 15.3 0.16
TABLE I: Prediction error of liquid height , sugar concentration , and of viscosity using the GPR models.
Fig. 10: Prediction of liquid height and sugar concentration on the testing dataset using the GPR model. (a) Relation between liquid height and decay rate , oscillation frequency (b) Relation between sugar concentration and decay rate , oscillation frequency (c) Prediction of liquid height. (d) Prediction of sugar concentration.

V-D Generalization to Other Liquids

Our models for predicting viscosity and liquid height trained on the sugar-water solutions can directly generalize to other liquids without re-training or fine-tuning. Using the GPR models trained in Section V-C, we show results of estimating and of olive oil (), corn oil (), and mirin (). The testing dataset consists of setup in total: the liquid-container system on the discretized liquid height for the three types of liquid.

The prediction result is shown in Fig. 11 and Table II. The prediction error of in mirin is much more than that of the oils. We suspect that mirin might be a non-Newtonian fluid and its thickness can’t be explained only by viscosity. Saal et al. [14] reported their mean square error (MSE) of being . To compare, our MSE of is on olive oil, on corn oil, and on mirin.

Fig. 11: (a) The prediction result of liquid height on common liquids. (b) The prediction result of viscosity on common liquids (shown in scale).
MAE (mm) MAE
Olive Oil () 1.50 0.12
Corn Oil () 1.46 0.12
Mirin () 0.99 0.35
TABLE II: Prediction error of liquid height and of viscosity on common liquids.

V-E Experiments using Different Containers

We show in this experiment that our approach works on containers of different sizes, shapes, and materials. Similar to Section V-C, we collect training and testing datasets of sugar-water solutions (Fig. 7(a), 7(b)) with different liquid heights using the three containers: cylinder container (Fig. 4c), cuboid container (Fig. 4d), and glass container (Fig. 4e). The cylinder container and cuboid container are made of plastic. To prevent slippage, we 3D-print a mold that clamps on the cylinder container and creates a flat contact surface for the gripper. The details of the datasets are shown in Table III. The cuboid container and the grooved cuboid container (Fig. 4b) used in the previous experiments have a similar shape but different sizes. The glass container is much heavier and is of a different shape and size than other containers.

Container Type Cylinder Cuboid Glass
Range of Level (mm) 12 to 35 15 to 38 22 to 39
Trainig Dataset Size 117 108 63
Testing Dataset Size 104 96 56
TABLE III: Details of the training and testing datasets collected on the cylinder, cuboid, and glass containers.

The prediction results are shown in Fig. 12 and Table IV. We can see that the trained GPR model using the grooved cuboid container (Fig. 9(b)) has a similar shape with the one of the cuboid container (Fig. 11(b)) but different from the one of the cylinder container (Fig. 11(c)). This supports our discussion in Section IV-D. We also observe the GPR model having a very different landscape when using the glass container. A potential reason is that the liquid-container dynamics is different when the container weight is not negligible (the glass container has a similar weight to the liquid inside on average).

(mm) (wt%)
MAE (Cylinder Container) 0.84 13.5 0.15
MAE (Cuboid Container) 1.96 20.4 0.23
MAE (Glass Container) 0.91 12.9 0.14
TABLE IV: Prediction error of liquid height , sugar concentration , and of viscosity using GPR models in the cylinder, cuboid, and glass containers.
Fig. 12: The sugar concentration prediction results with different containers. (a) Train and test with the cylinder container. (b) Train and test with the cuboid container. (c) Train and test with the glass container.

V-F Training Data Efficiency

A key advantage of our approach is its data efficiency. We train GPR models using subsets of the training datasets in all three containers. The subsets have a size ranging from to data and are uniformly sampled from the original training datasets in Section V-C and V-E. We only show results on concentration prediction because it is more challenging than liquid height prediction. The change of concentration prediction error when using different amounts of training data is shown in Fig. 13. For a smooth container like the cylinder container, we can train our GPR model with as little as data and has good prediction results (). The cuboid containers take data to train our GPR model, which is still much less than what a neural network requires.

Fig. 13: The prediction error of sugar concentration when trained with different amount of data.

V-G Generalization to Different Containers

Finally, we show that our model can generalize across similar-shaped containers with very little fine-tuning data. Given the GPR model trained in the grooved cuboid container (Fig. 9(b)), we fine-tune and test on the cuboid container using the approach in Section IV-D. The fine-tuning datasets have a size ranging from to data and are the same as the training datasets of the cuboid container used in Section V-F. We compared our results with the baseline model, which is a GPR model directly trained from the fine-tuning data. The concentration prediction MAE when using different amounts of fine-tuning data is shown in Fig. 14. It takes datapoints to fine-tune our model. In fact, with more than fine-tuning datapoints, our fine-tuned model always has better or similar performance than the baseline model.

Fig. 14: The prediction error of sugar concentration on the cuboid container. Red is the model directly trained on the fine-tuning dataset. Blue is the model first trained on grooved cuboid container then fine-tuned on the fine-tuning dataset.

Vi Future Work

We believe our work is just the beginning of a sequence of super-human perception research based on dynamic tactile signals. One avenue to improve our approach on liquid properties estimation is making use of the dynamic tactile signals collected in the activation stage as well as the free-oscillation stage. In the future, we will survey models for signals in the activation stage and explore other ways to activate the system, such as perturbing in different directions. Other extensions of our approach include modeling the damped oscillation signal with more features and learning a more complex model than GPR models. We will also explore how the physics-inspired models can be applied to other dynamic tactile sensing tasks, such as estimating solid content properties.

Vii Conclusion

In this paper, we present an approach to precisely estimate the liquid viscosity and volume in the container using dynamic tactile sensing. We introduce a physics-inspired model to interpret the damped oscillation signal after perturbation and relate the signal to the liquid properties. We then learn a GPR model to estimate liquid properties. In the experiment, our approach achieves 100% classification accuracy among water, oil, and detergent. For the challenging sugar-water properties regression task, we can predict the height of the liquid with mm error and the sugar concentration with wt% error. Our approach is also data-efficient, agnostic to volume, and can easily generalize to different types of liquid and containers. Our work elevates dynamic tactile sensing to a much higher precision level in this task and explains its principal mechanisms. We believe this lightweight learning approach based on simple physics can be applied to other dynamic tactile sensing tasks.


This work was supported by Toyota Research Institute. The authors would like to thank Yifan You for high-quality feedback in writing. We would also like to thank Uksang Yoo and Ruihan Gao for proofreading the paper.


  • [1] C. L. Chen, J. O. Snyder, and P. J. Ramadge (2016) Learning to identify container contents through tactile vibration signatures. In 2016 IEEE International Conference on Simulation, Modeling, and Programming for Autonomous Robots (SIMPAR), pp. 43–48. Cited by: §I, §II-B, §II-B.
  • [2] P. Cullen, A.P. Duffy, C. O’Donnell, and D. O’Callaghan (2000-12) Process viscometry for the food industry. Trends in Food Science & Technology 11, pp. 451–457. External Links: Document Cited by: §I.
  • [3] F. De Boissieu, C. Godin, B. Guilhamat, D. David, C. Serviere, and D. Baudois (2009) Tactile texture recognition with a 3-axial force mems integrated artificial finger.. In Robotics: Science and Systems, pp. 49–56. Cited by: §II-A.
  • [4] M. Eppe, M. Kerzel, E. Strahl, and S. Wermter (2018) Deep neural object analysis by interactive auditory exploration with a humanoid robot. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vol. , pp. 284–289. External Links: Document Cited by: §I, §II-B.
  • [5] P. Giguere and G. Dudek (2011) A simple tactile probe for surface identification by mobile robots. IEEE Transactions on Robotics 27 (3), pp. 534–544. Cited by: §II-A.
  • [6] P. Güler, Y. Bekiroglu, X. Gratal, K. Pauwels, and D. Kragic (2014) What’s in the container? classifying object contents from vision and touch. In 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, Vol. , pp. 3961–3968. External Links: Document Cited by: §I, §II-B, §II-B.
  • [7] N. Jamali and C. Sammut (2010) Material classification by tactile sensing using surface textures. In 2010 IEEE International Conference on Robotics and Automation, pp. 2336–2341. Cited by: §II-A.
  • [8] S. Jin, H. Liu, B. Wang, and F. Sun (2019) Open-environment robotic acoustic perception for object recognition. Frontiers in neurorobotics 13, pp. 96. Cited by: §II-B.
  • [9] Y. Jonetzko, N. Fiedler, M. Eppe, and J. Zhang (2020)

    Multimodal object analysis with auditory and tactile sensing using recurrent neural networks

    In International Conference on Cognitive Systems and Signal Processing, pp. 253–265. Cited by: §II-B.
  • [10] L. Keith J. and J. H. Meiser (1982) Physical chemistry. Menlo Park, Calif: Benjamin/Cummings Pub. Co.. Cited by: §III-B.
  • [11] H. Kolvenbach, C. Bärtschi, L. Wellhausen, R. Grandia, and M. Hutter (2019) Haptic inspection of planetary soils with legged robots. IEEE Robotics and Automation Letters 4 (2), pp. 1626–1632. Cited by: §II-A.
  • [12] R. Li, R. Platt, W. Yuan, A. ten Pas, N. Roscup, M. A. Srinivasan, and E. Adelson (2014) Localization and manipulation of small parts using gelsight tactile sensing. In 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, Vol. , pp. 3988–3993. External Links: Document Cited by: §V.
  • [13] C. Matl, R. Matthew, and R. Bajcsy (2019) Haptic perception of liquids enclosed in containers. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vol. , pp. 7142–7149. External Links: Document Cited by: §I, §II-B, §II-B, §IV-C, §V-A, §V-B, §V-C, §V.
  • [14] H. P. Saal, J. Ting, and S. Vijayakumar (2010) Active estimation of object dynamics parameters with tactile sensors. In 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, Vol. , pp. 916–921. External Links: Document Cited by: §I, §II-B, §II-B, §IV-C, §V-C, §V-D, §V.
  • [15] J. Sinapov, C. Schenck, and A. Stoytchev (2014) Learning relational object categories using behavioral exploration and multimodal perception. In 2014 IEEE International Conference on Robotics and Automation (ICRA), pp. 5691–5698. Cited by: §II-B.
  • [16] J. Sinapov, V. Sukhoy, R. Sahai, and A. Stoytchev (2011) Vibrotactile recognition and categorization of surfaces by a humanoid robot. IEEE Transactions on Robotics 27 (3), pp. 488–497. Cited by: §II-A.
  • [17] M. A. Srinivasan and R. H. LaMotte (1995) Tactual discrimination of softness. Journal of Neurophysiology 73 (1), pp. 88–101. Note: PMID: 7714593 External Links: Document, Link, https://doi.org/10.1152/jn.1995.73.1.88 Cited by: §I.
  • [18] T. Taunyazov, L. S. Song, E. Lim, H. H. See, D. Lee, B. C. Tee, and H. Soh (2021) Extended tactile perception: vibration sensing through tools and grasped objects. arXiv preprint arXiv:2106.00489. Cited by: §II-A.
  • [19] W. Yuan, S. Dong, and E. H. Adelson (2017) GelSight: high-resolution robot tactile sensors for estimating geometry and force. Sensors 17 (12). External Links: Link, ISSN 1424-8220, Document Cited by: §I, Fig. 3, §IV-A.


Vii-a Experiment Results of Quadratic Regression Models

In addition to estimating sugar-water concentration and liquid height using GPR models as shown in Section V-C, we also train parameterized quadratic regression models to make the predictions as introduced in Section IV-C. The prediction result on the testing dataset is shown in Fig. 15 and the training and testing errors are shown in Table V. Quadratic models can estimate , , and with MAE of mm, wt%, and . The testing result using quadratic models is slightly worse than the non-parametric GPR models.

Fig. 15: Prediction of liquid height and sugar concentration on the testing dataset using the quadratic regression model. (a) Relation between liquid height and decay rate , oscillation frequency . (b) Relation between sugar concentration and decay rate , oscillation frequency . (c) Prediction of liquid height. (d) Prediction of sugar concentration.
(mm) (wt%)
Quadratic Model MAE (training dataset) 1.13 21.1 0.23
Quadratic Model MAE (testing dataset) 0.84 20.0 0.23
TABLE V: Prediction error of liquid height , sugar concentration , and of viscosity using quadratic regression models.