Ergonomically Intelligent Physical Human-Robot Interaction: Postural Estimation, Assessment, and Optimization

by   Amir Yazdani, et al.

Ergonomics and human comfort are essential concerns in physical human-robot interaction applications, and common practical methods either fail in estimating the correct posture due to occlusion or suffer from less accurate ergonomics models in their postural optimization methods. Instead, we propose a novel framework for posture estimation, assessment, and optimization for ergonomically intelligent physical human-robot interaction. We show that we can estimate human posture solely from the trajectory of the interacting robot. We propose DULA, a differentiable ergonomics model, and use it in gradient-free postural optimization for physical human-robot interaction tasks such as co-manipulation and teleoperation. We evaluate our framework through human and simulation experiments.



There are no comments yet.


page 2

page 8


DULA and DEBA: Differentiable Ergonomic Risk Models for Postural Assessment and Optimization in Ergonomically Intelligent pHRI

Ergonomics and human comfort are essential concerns in physical human-ro...

Motivating Physical Activity via Competitive Human-Robot Interaction

This project aims to motivate research in competitive human-robot intera...

Towards Partner-Aware Humanoid Robot Control Under Physical Interactions

The topic of physical human-robot interaction received a lot of attentio...

Real-time Robot-assisted Ergonomics

This paper describes a novel approach in human robot interaction driven ...

The role of haptic communication in dyadic collaborative object manipulation tasks

Intuitive and efficient physical human-robot collaboration relies on the...

A Robot Teleoperation Framework for Human Motion Transfer

Transferring human motion to a mobile robotic manipulator and ensuring s...

Plan or not: Remote Human-robot Teaming with Incomplete Task Information

Human-robot interaction can be divided into two categories based on the ...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.


Improving workplace ergonomics and reducing the risk of work-related musculoskeletal disorders (WMSDs) have been the focus of researchers for many years. High rate of injuries in industries has further motivated these studies. WMSDs are the second-largest source of disabilities worldwide Vos et al. (2015), and awkward postures are the main contributor to the work-related injuries da Costa and Vieira (2010).

Figure 1: The framework for ergonomically intelligent p-HRI.

Industry 4.0 initiative Gorecky et al. (2014) and the development of collaborative workplaces where smart agents (e.g., collaborative robots Sherwani et al. (2020)) physically collaborate with humans to perform different types of tasks, highlights the importance of human comfort and ergonomics in physical human-robot interaction (p-HRI). The literature in this area is very limited and sparse. In this paper, we introduce a framework for ergonomically intelligent p-HRI that include posture estimation, assessment, optimization and correction. Figure 1 shows our proposed framework.

p-HRI covers a vast number of applications such as direct p-HRI Guerrero et al. (2013), co-manipulation Kim et al. (2017), assistive holding Chen et al. (2018), handover Busch et al. (2018), and teleoperation Rahal et al. (2020). We categorize the teleoperation into three main types: goal-constrained teleoperation (e.g., pick and place with a desired pose for the object), path-constrained teleoperation (e.g., turning a valve where the path to follow is constrained based on the diameter of the valve), and trajectory-constrained teleoperation (e.g., arc welding where the operator should follow a velocity profile in addition to the welding path). Figure 2 visualizes the different types of p-HRI applications.

A common and practical solution to improve workplace ergonomics is to perform risk assessments and analyze human comfort during the task. Ergonomists usually perform this process for the worst posture during the task by taking measurements of human posture onsite or from recorded images or videos. They normally use risk assessment tools Ramaganesh et al. (2021) to assess the task and provide intervention and improvement in forms of user training Bazazan et al. (2019), workstation modification Shikdar and Al-Hadhrami (2007) or task rotation Motabar and Nimbarte (2021). However, in most cases, it is not possible to have an ergonomist present during the entire task to perform a comprehensive risk assessment, which results in missing events or only partial understanding of awkward postures. In addition, these sampling strategies lead to failures in detecting the most hazardous combinations of force and posture, increasing the inaccuracy of an assessment. Finally, although postural correction by ergonomists generally helps reduce the risk of injuries, in many cases, they are not optimal or tailored to specific task requirements.

Early improvements on continuous monitoring of the posture include using motion capture (MoCap) systems to estimate the posture and task parameters (e.g., frequency and duration of the task) still require significant time and effort for set up Alvarez et al. (2017). Moreover, putting markers on human operators can be inconvenient. Alternative markerless techniques are more adaptable, minimally intrusive, and less expensive. However, they need calibration to deal with errors and uncertainties from the sensors Xiao and Zarar (2018); Chen and Davis (2000). Vision-based markerless methods can also be perturbed by the lighting, background color, and even the user’s clothing Erol et al. (2007). The proximity between the human and robot and in p-HRI increases occlusion and reduces the accuracy in estimate posture as mentioned in Busch et al. (2017).

Postural optimization in p-HRI has recently received substantial attention in research (see Table 1) and developing a model of human comfort lies at the heart of effective postural optimization. Researchers either use risk assessment tools from ergonomics or propose computational models for human comfort and ergonomics. Among all risk assessment tools, RULA McAtamney and Corlett (1993) and REBA Hignett and McAtamney (2000) depend most on human posture and provide quantitative scores, making them good choices for postural optimization applications. However, the discrete scores and the presence of plateaus in RULA and REBA Busch et al. (2017) create challenges when using them in gradient-based postural optimization. Based on our experience, using the risk assessment models directly in gradient-free optimization is time-expensive, and the plateaus often prevent progress toward the optimal global solution in postural optimization. Thus, researchers in p-HRI often use computational models based on approximations of ergonomic assessment models in gradient-based postural optimizations; quadratic approximations Rahal et al. (2020); Busch et al. (2017, 2018) being the standard approach in the literature. However, these approximations deviate far from the scientifically validated assessments, causing doubt that they can reliably provide the same level of ergonomic benefit.

Figure 2: Categories of physical HRI applications.

Assuming perfect ergonomics assessment and optimization, the postural correction process is also a challenging part in improving the ergonomics in p-HRI. Manual correction by ergonomists is tedious and cannot be applied continuously for the human operator. More systematic approaches such as continuous visual Lorenzini et al. (2018) and vibrotactile Kim et al. (2018) feedback to the user perform well in correcting the posture. However, they need different devices to provide feedback, require the user to accept and apply the correction. Guiding the human toward the optimal posture using the force from the robot Rahal et al. (2020) also may cause extra force on the human and increase the risk of injuries, as well as affecting the realistic haptic feeling.

To overcome these issues in postural estimation and optimization in p-HRI, we propose ergonomically intelligent physical human-robot interaction (see Fig. 1). It includes four main parts, targeting the drawbacks of current methods:

  • Posture estimation in p-HRI solely from the trajectory of the interacting robot: We show that the interacting robot is an adequate sensor for continuous posture estimation of the human and for monitoring the risk of injuries. We model the posture estimation problem as a partially observable dynamical system where the robot’s trajectory at the interaction point is the only observation. We provide an approach for estimating human segment lengths using interactive motion routines and a particle filter to infer the human joint angles.

  • Differentiable ergonomics assessment model: We learn an accurate, continuous, and differentiable model called DULA

     (Differentiable Upper Limb Assessment) for upper body ergonomics using a standard neural network from a dataset of exhaustively sampled postures, labeled by the standard RULA risk assessment tool.

  • Gradient-based postural optimization: We use DULA in a gradient-based optimization and compare it with a gradient-free optimization directly using RULA.

  • Automatic postural correction: We provide a new approach to correct human posture in teleoperation through the velocity scale between the leader and follower robots.

Ergonomics Model Analytical Models Learned Models Risk Assessment Tools
Approximation Quadratic No-Approximation
Optimization Gradient Based Peternel et al. (2018) Rahal et al. (2020)
Peternel et al. (2017) This work Busch et al. (2017)
Chen et al. (2018) Busch et al. (2018)
Peternel et al. (2020)
Kim et al. (2017)
Gradient Free Marin et al. (2018) van der Spaa et al. (2020)
No Optimization Shafti et al. (2019)
Legend Physical-HRI: Assistive Holding, Physical-HRI: Handover, Physical-HRI: Co-Manipulation
Teleop: Goal-Constrained with Postural Optimization By Interface Reconfiguration
Teleop: Goal-Constrained with Online Postural Optimization
Table 1: State of the art of postural optimization in p-HRI and teleoperation.

In this paper, we focus on the first three parts of our framework. We introduce our approach for general p-HRI scenarios. However, we implement and evaluate them in a simple teleoperation application as an example. We initially proposed the idea of ergonomically intelligent teleoperation systems in HRI Pioneers workshop Yazdani and Sabbagh Novin (2021). The posture estimation idea for teleoperation is already published at IEEE CASE conference Yazdani et al. (2021b), and we introduced the differentiable ergonomics model for postural optimization in RSS Robotics for People workshop Yazdani et al. (2021a). In this paper, we do not add any novel experimental evolution. Instead, we formalize the framework for general p-HRI and more tightly link the previous contributions. We believe this paper brings all the previous work together, improving clarity and advancing the discussion of this method.

Related Work

In this section we provide the state of the art in posture estimation, risk assessment and postural optimization in p-HRI.

Posture Estimation in p-HRI

For many years, MoCap systems using reflective markers (Das et al. (2011)

) were the dominant approach for posture estimation. However, advances in computer vision have led to increased popularity in markerless techniques 

Saremi et al. (2018); Mathis et al. (2018). In p-HRI and teleoperation, researchers mainly have used external sensors (i.e., a vision system or IMU) to estimate a user’s posture, especially for hand gestures Vartholomeos et al. (2016).

The idea of solely using the interacting robot’s trajectory (leader robot in teleoperation) for posture estimation in p-HRI has been introduced concurrently with this research in Rahal et al. (2020)

, where they solved the inverse kinematics of the human arm. Unlike our probabilistic approach, their heuristic does not always hold across different tasks. In comparison, our work provides a probabilistic distribution over the human posture and ergonomic risk score. Moreover, our human model covers the entire upper body, including the arm and torso. Later,

Vianello et al. (2021) proposed probabilistic prediction of human postures for a given robot trajectory executed in a collaborative scenario. They learned the distribution of the null space of the Jacobian and the weights of the weighted pseudo-inverse from demonstrations. Their approach highly depends on the task and demonstrations by a human. In comparison, our approach is task-independent and does not require any demonstration.

Defining a model for human joint limits is challenging. Studies show that the range of motion for a joint varies depending on the positions of other joints (inter-joint dependency) or other degrees-of-freedom in the same joint (intra-joint dependency) 

Wang et al. (1998); Jiang and Liu (2018) and vary by gender and person. Akhter and Black (2015) used a dataset of recorded MoCap of human motion to develop a discontinuous mathematical model for a posture-dependant range of motion and check the validity of a full-body posture. Jiang and Liu (2018) used the above model to label the validity of a set of randomly generated postures and learned a differentiable neural network based on the generated data, and used it as a constraint in the inverse kinematics optimization. Their arm model only includes the shoulder and elbow and not the wrist. We use this learned network for checking the validity of the arm posture.

Ergonomics Models and Postural Optimizations in p-HRI

The literature in postural optimization in p-HRI and teleoperation is minimal. Table 1 summarizes the relevant literature for in both areas.

Researchers have examined ergonomics and postural optimization in three different types of p-HRI: (1) Assistive Holding (e.g., Marin et al. (2018); Shafti et al. (2019); Chen et al. (2018)), (2) Object handover (e.g., Busch et al. (2017, 2018); Peternel et al. (2017)), and (3) Co-Manipulation (e.g., van der Spaa et al. (2020); Peternel et al. (2018, 2017); Kim et al. (2017)). In the first two tasks the postural optimization provides optimized posture for the human and joint configurations for the robot while maintaining contact with the object at the interaction interface. In co-manipulation, the postural optimization outputs a trajectory of optimal postures for the human and an optimal joint-space trajectory for the robot to perform the task. In teleoperation, postural optimization has only been developed for goal-constrained tasks.

Ergonomic assessment models provide the primary cost in postural optimization objectives. p-HRI researchers have proposed many computational models to assess ergonomics and comfort of users, including peripersonal space Chen et al. (2018), muscle fatigue Peternel et al. (2017), and joint overloading Kim et al. (2017). To make the optimization simpler, Marin et al. (2018)

suggested a contextual ergonomics model, which is a set of Gaussian process models including joint angles, moments, reaction, load, and muscle activation trained with the musculoskeletal simulation task contexts. Using Gaussian process models enables a 2D latent space to search while their cost function is defined in the high-dimensional musculoskeletal space.

Some literature use approximations of the ergonomics risk assessment tools. Busch et al. (2018) proposed a differentiable surrogate of the REBA score by fitting a weighted combination of quadratic functions plus a constant for the task payload. Rahal et al. (2020) suggested a quadratic approximation for RULA, which is the summation of the quadratic norm of the deviation from the human neutral posture for shoulder, elbow, and wrist joints angles. Their approximation conceptually agrees with the qualitative idea behind RULA, in which the risk score goes higher when the human deviates more from the neutral posture.

van der Spaa et al. (2020) provided the only study of directly using risk assessment tools in derivative-free postural optimization. They added the REBA score to the transition cost function in an A* optimization for task and motion planning of a robot in co-manipulation. Finally, Shafti et al. (2019) identified six main causes of low ergonomic states, leading to six universal robot responses to allow the human to return to an optimal ergonomics state. Their framework identifies the causes and controls the robot to constantly adapt to the environment in an ergonomic way.

The table shows that most of the publications use analytical models and quadratic approximation to risk assessment tools in their approaches. Due to the differentiability of the models, they can be used in gradient-based postural optimization, which is faster and easier to implement. However, the reliability and optimality of the level of ergonomic benefit they provide are questionable comparing to standard risk assessment tools in ergonomics. Table 1 also shows an excellent opportunity for researchers to develop accurate ergonomics models learned from standard risk assessment tools and benefit from them in gradient-based optimization. We targeted this area of research and proposed DULA, a differentiable and continuous ergonomics model that is learned from the standard assessment tool, RULA.

Problem Statement

In this section, we introduce the problem of posture estimation and optimization in p-HRI. Among the different p-HRI applications, we focus on co-manipulation and teleoperation since they have the most range of continuous motions. However, the same problem statement can be extended to other applications with minor modifications.

Posture Estimation from the Robot in p-HRI

In p-HRI, the physical interaction occurs at the interaction point. This is the contact point either between the human body and the robot in direct p-HRI and teleoperation or between the human body and the shared object in co-manipulation, assistive holding, and handover. In the latter case, we assume the robot firmly grasps the shared object, adding it to the robot’s kinematic chain.

We seek to solve the problem of estimating the human joint-space trajectory using only the observed task-space poses and velocities of the interaction point from the robot. The mapping from human joint angles into the pose of the interaction point is through the kinematics of the human model parameterized by the segment length . We estimate independently, prior to posture estimation. Hence,


where is the observation function, and is the forward kinematics of the human model. This defines only a partial observation of the human posture because of redundancy in the human kinematics and a noisy measurement at the interaction point, which may change slightly during a task.

We can write the postural estimation as an optimization:


where and are the joint limits,

is the weight vector for position and orientation elements, and

is the human motion model defined by:


The high degree-of-freedom in human kinematics makes this problem redundant with an infinite number of solutions. We seek the solution closest to the actual human’s posture.

Postural Optimization in Co-Manipulation

In co-manipulation, the robot helps a human to move an object (usually a heavy one) from an initial pose to the desired pose Peternel et al. (2017). There is no specific trajectory for the object to follow; the human plans and leads the motion, and the robot follows and provides help carrying the object. Meanwhile, the ergonomically intelligent p-HRI seeks to find the optimal posture for the human at each instance while holding the pose and velocities at the interaction point close to the current state. Once found, the system will suggest the solution to the human. We call this online postural optimization. Using the same notation of our posture estimation approach, we can formalize finding the ergonomically optimal posture with the following optimization problem:


where is the optimal posture at time . It is important to note that the optimal posture from Eq. (4) for each time step is then suggested to the human to move towards. The human can refuse or accept and try to apply it as much as possible while completing the task.

Postural optimization in Teleoperation

The main distinction of teleoperation from other p-HRI applications is the motion (and force) coupling between the leader and follower robots. The motion coupling between the two robots is not necessarily a one-to-one scale, and the coupling can be paused and resumed. This enables the teleoperator to disengage the leader robot in the middle of a task in many types of application, reposition their postures (and the leader robot as the result), and resume the teleoperation from a a more comfortable posture Peternel et al. (2020). The relative position trajectory of the follower robot remains unchanged; however, its velocity trajectory changes due to the pause. The motion scaling also helps keep the teleoperator’s posture in a smaller comfort zone while the follower robot operates in a much wider zone.

To use this feature of teleoperation in postural optimization, we define three types of postural optimization, customized for different types of teleoperation from Fig. 2.

Online Postural Optimization

Online postural optimization in teleoperation is very similar to online postural optimization in co-manipulation. The difference is that instead of having initial and goal poses for the shared object in co-manipulation, in teleoperation, we have initial and goal poses for the end-effector of the follower robot (or the object it manipulates). Hence, the ergonomically intelligent system should find the optimal posture of the human at each instance, in which the pose and velocities of the follower robot’s end-effector are not far from the current states, and suggest it to the human. The formulation in Eq. 4 works here too, where are observed from the leader robot.

Initial Postural optimization

The remote connection between the leader and follower robots makes it possible to start the teleoperation task from any initial human posture and the corresponding leader robot’s initial posture and perform the same task with the follower robot, considering the human and leader robot’s workspace. We use this feature to propose initial postural optimization for path-constrained and trajectory-constrained teleoperation tasks. In this type of postural optimization, the ergonomically intelligent system observes the human doing the task once, then calculates the optimal initial posture for the human to start the same task with the same task-space motion for the follower robot. It also requires recording the entire motion of the robots and the human during the task.

We form initial postural optimization as the following:


where is the recorded velocity of the interaction point during the performed task, and is the Jacobian of human kinematic chain. Here, the superscripts refer to human, leader robot, and follower robot, respectively.

Postural Optimization by Interface Reconfiguration

An inherent feature of path-constrained teleoperation is the ability to pause the teleoperation by disengaging the leader and follower robots from each other, move the leader robot to a new position, and resume the teleoperation. This is usually done by a clutch switch placed under the user’s foot. Benefiting from this feature, we propose postural optimization by interface reconfiguration, in which, when the ergonomically intelligent system detects high-risk postures during a previously recorded task, suggests to pause the teleoperation. When teleoperation is paused, the system calculates a new optimal posture for the human to resume the teleoperation from it and continue the task.

The formulation is similar to the initial postural optimization but covers the time after the pause:


where is the time at pause in teleoperation.

Approach and Implementation

This section proposes our approaches for solving the problems mentioned in the previous section and details their implementations.

Human Kinematics Model

We use a 10-DOF kinematics model (see Fig. 3) to analyze the upper body motion of a human sitting on a chair and interacting with a robot. We assume that the chair is stationary with a known position w.r.t. the robot, and the human sits on the center of the chair. In Yazdani et al. (2021b), we proposed an approach based on circle point analysis Mooring et al. (1991) for estimating human segment lengths in p-HRI using interactive motion routines and we use it here. We encode human motion limits based on both fixed limits and the posture-dependant joint limit model from Jiang and Liu (2018).

Human posture update during time follows Eq. 3

. We sample joint accelerations from a Gaussian distribution

where is the acceleration covariance matrix.

We model the probabilistic observation likelihood function in Eq. 1 by a Gaussian distribution over the human hand’s pose and velocity


in which is the kinematic covariance matrix.

Particle Filter for Posture Estimation

We approximate the solution for the partially-observable problem of posture estimation in Eq. 2 by a particle filter  Thrun et al. (2005) with some modifications. As the estimation of the human model from the trajectory of the leader robot has high ambiguity due to the redundancy, we add the joint angular velocities to our state variables and use the velocity of the robot’s end-effector as observation. As a prior, we encode that the human starts the task in a static and neutral posture. We initialize

particles using a truncated normal distribution with the mean at the neutral posture

and set the initial angular velocities to zero.

DULA: Differentiable Upper Limb Assessment Model for Postural Optimization

Solving Eqs. 4, 5, 6 using a gradient-based optimization requires a differentiable and continuous ergonomics model as objective functions. Instead of using analytical models or approximations, we introduce DULA (Differentiable Upper Limb Assessment) which is an accurate, continuous, and differentiable model for upper body ergonomics. It is learned using a standard neural network from a dataset of exhaustively sampled postures, labeled by RULA.

To learn a continuous and differentiable model for RULA, we designed a fully connected regression-based neural network. While RULA provides discrete integer scores from 1 to 7, we choose to predict continuous labels. If we had instead performed multi-class classification based on the discrete labels, the resulting model would be less useful in optimization as there is no natural choice of a smooth objective to minimize or constrain ergonomic cost. This would negate our desire for a computationally helpful model. Hence, we perform regression to the multi-class labels.

Figure 3: The structure of the DULA neural network.

The structure of the neural network is shown in Fig. 3

. It includes 4 hidden layers with ReLU activation function. We found that a network with 124 units for the first three layers and 7 units for the last hidden layer worked best. We train the network using the standard mean squared error loss function for 2000 epochs using a learning rate of 0.001. We used 5-fold cross-validation to find the optimal network parameters.

We developed a dataset of 7.5 million upper body postures of a human. We additionally defined task parameters based on the RULA worksheet—the frequency of the arm and body motions; type and maximum load on the arm and body; neck angle; and whether any legs, feet, or arms are supported. We developed a script for automatic RULA assessment based on the posture and tasks parameters and verified it with several ergonomists. We used this script to label the posture dataset. Since postures with labels 1, 2, 6, and 7 are not frequent in the full range of human motion, we balanced the dataset by forcing the data generation scripts to generate enough data points with those labels. We split the dataset into 80% training and 20% testing sets.

Training of the neural networks takes about 60 hours on a Core i9 CPU with GPU support. We implemented and trained our network using PyTorch 

Paszke et al. (2019).

Postural optimization

We explore two methods for postural optimization. Initially, we use cross-entropy method (CEM) Kobilarov (2012); De Boer et al. (2005)

as a sample-based and gradient-free optimization. In our implementation, we use Gaussian probability distribution N=10000 samples.

As an alternative method, we utilize a derivative-based optimization using our learned DULA model. We solve the nonlinear and constrained least-square postural optimization using sequential quadratic programming (SQP) Nocedal and Wright (2006) solver in SciPy. We calculated DULA gradients using automatic differentiation in PyTorch.

The following section examines variations of postural correction optimizations using these two methods and compares the resulting RULA score distributions.

Evaluation & Experimental Protocol

We evaluated our proposed postural estimation and optimization approaches through human subject studies and simulation experiments, respectively.

Human Subject Study for Posture Estimation

We conducted a human subject experiment in which participants interact with a 6-DOF Quanser HD haptic interface as the leader robot (see Fig. 7) in teleoperation, and we recorded their upper body motion using a 12-camera Optitrack MoCap system for comparison. We recruited 8 participants (4 females, 4 males) with ages ranging from 25 to 33 years and heights in the range of cm. Participants were graduate students from various programs and did not have any experience with teleoperation robots and received a 15-min training with the robot. Each participant performed 4 tasks visualized in Fig 4. We provided a printed visual guide on the table for the first three tasks. However, the participants were not required to follow the path accurately. The participants were not told what posture to initialize the task from and how high they should be above the table to do the task. The robot collected data from the participant’s motion.

To compare with other well-known approaches, we solved the postural estimation problem using two other deterministic methods: (1) boosted and bounded online least-squares IK optimization (Online-IK) in which we simply solve the inverse kinematics optimization independently at each time step by initializing it with the solution from the previous step, and (2) boosted and bounded offline least-squares trajectory IK optimization (Offline-TrajIK) in which we solve the inverse kinematics problem for the whole trajectory by initializing it with the solution trajectory from Online-IK. We used the dogleg algorithm with rectangular trust regions Nocedal and Wright (2006) in SciPy as the solver.

Figure 4: The tasks for human subject study: repetitive motions following a straight line in the direction (a), a straight line in the direction  (b), a circular path (c), and repetitive motions between random sides of two blocks positioned at different heights, with an unprescribed motion and high range of hand rotation (d).

Simulated Environment For Postural Optimization

We developed an open-source simulator for postural correction using ROS. It includes a human seated on a stool and a KUKA LWR-4 robot (two robots for teleoperation). The simulated human behaves like a human in two ways: (1) physically controlling the interactive task and (2) accepting or rejecting the recommended optimal postural corrections.

Figure 5: Overview of teleoperation simulation.

We model this as an optimal motion planning framework with re-planning that finds a human joint trajectory that interacts with the robot to do the task while moving toward the optimal ergonomic posture. For example, it is formulated as the following for a teleoperation task:


where is the trajectory of human posture from time to the time of the horizon , is the goal pose of the follower robot, and is the pose of the follower robot at time . Here, is a scalar number that models the postural correction acceptance and the effort of the human operator towards applying the postural correction. Details of the motion planning for the humans and the robots are presented in Fig. 5. We note that this model is likely a simplification of true human interactive behavior. We do not advocate for its use over human subject studies. Instead, we propose it as a useful tool for systematically exploring new human safety assessment and improvement algorithms.

Figure 6: Deviation of the posture estimated by our approach vs Online-IK and Offline-TrajIK methods among all the tasks.
Figure 7: Video-overlaid skeletons show the posture estimated from our proposed approach (red) and MoCap (green).


This section provides results for our ergonomically intelligent p-HRI system. We discuss the proposed posture estimation approach’s performance compared with MoCap, and their corresponding estimated risk assessment results. Moreover, we showcase the accuracy of the learned model for DULA and its effect on postural optimization.

Posture Estimation

Fig. 6 compares our posture estimation approach with two other least-squares IK solutions for redundant robots for all participants among 4 tasks. From the statistical analysis of the results, we can conclude that our probabilistic particle filter

approach performance is not significantly different from the other two methods. Overall, the approach generally agrees with MoCap with a median deviation less than 0.09rad (less than 5deg) and upper quartile less than 0.25rad (less than 15deg) considering the observation solely from the stylus trajectory and having no extra sensors.

Since there is no ground truth for human posture to compare with, we provide a qualitative evaluation using video frames of a participant during task (3) with overlaid reconstructed skeletons from MoCap (green) and our approach (red) in Fig. 7. The estimated postures align well with the MoCap postures during the entire task.

Fig. 8 shows the maximum RULA score during a task for the postures from our approach and MoCap postures. Our approach successfully identified all instances where the score was higher than 2 (i.e., future investigation or change may be or is needed) in all 32 trials. It also resulted in the same interpretation of the RULA in 27 trials (84.37%) and the same RULA score in 21 trials (65.63%).

Figure 8: Comparison of the maximum value of RULA scores of a task using estimated posture from our approach and MoCap for all the participants and trials.

Overall, the results showed that our posture estimation solely from the robot has the potential to be used for continuous monitoring of ergonomics in p-HRI and rising alerts when further investigation or change in the task is required.

Postural Optimization

The trained model for DULA performs well on the test dataset and results in 99.73% accuracy. In the accuracy test, we round our continuous output to the nearest integer when reporting accuracy. The lowest diagonal element in the confusion matrix for DULA is 99.38%, which shows the high accuracy of the learned model across all ranges.

Figure 9 shows the postural correction results for gradient-free (left) and gradient-based (right) optimization in a simple teleoperation task. It presents the RULA scores for calculated optimal posture (green), human posture without postural correction (orange), and the human posture after applying the correction (blue) according to the human control and acceptance model during the task. The plot shows that the risk is reduced after the simulated human applies the suggested optimal postural correction. Also, the task completion time increases without a significant decrease in risk. We can also see that the gradient-based approach provides a smoother motion concerning the RULA risk score and avoids going through postures with a risk higher than 4. It also results in a shorter task completion time.

Figure 10 provides more information on comparing gradient-free and gradient-based postural optimization. It shows that the median risk scores of target optimal postures from the gradient-free approach is lower. However, the postures of the simulated human are more comfortable after applying the optimal posture calculated from the gradient-based optimization. We believe it is due to the smoother optimal postures that has been suggested to the simulated human from the gradient-based method. Moreover, the gradient-based approach is much faster. Each iteration of the gradient-free approach using 10000 samples takes 2.7 minutes to solve, while the gradient-based approach takes only tenths of a second.

Figure 9: Comparison between gradient-free and gradient-based postural correction on a teleoperation task. Lower scores are better.
Figure 10: Comparison in optimal posture and corrected posture between gradient-free and gradient-based optimization.


This paper proposed a framework for ergonomically intelligent physical human-robot interaction that includes posture estimation, risk assessment, and postural optimization. We showed that the interacting robot is an adequate sensor to continuously monitor posture to assess the ergonomics of an interactive task. We described a probabilistic approach based solely on the trajectory of the robot, which is already necessary to perform the p-HRI task. Additionally, we introduced DULA, a differentiable and continuous ergonomics model to assess human upper body posture. DULA is learned to replicate the non-differentiable RULA using a neural network, and it is 99.73% accurate and computationally designed for efficient postural optimization for p-HRI and other related applications. We also introduced a framework for postural optimization in p-HRI using DULA and presented a demo task. The results reveal that postural optimization using DULA lowers the risk of injuries.

There are several directions for future work. For the posture estimation part, Our approach can be used for full arm representation in VR systems based on the pose of controllers and headsets. When available, it would be straightforward to combine our proposed approach with other sensing modalities (e.g., vision or MoCap). We are examining the combination of our approach with OpenPose Cao et al. (2019) using incremental smoothing method Kaess et al. (2012) in order to improve estimation results and decrease runtime. The risk assessment part can be immediately extended by following the same procedure for REBA, a whole-body assessment tool. Additionally, we intend to conduct a human subject study to evaluate our postural optimization and correction approach. We wish to compare different means of feedback for the posture correction to the human, including visual, auditory, and haptic feedback.


  • I. Akhter and M. J. Black (2015) Pose-conditioned joint angle limits for 3d human pose reconstruction. In

    Proceedings of the IEEE conference on computer vision and pattern recognition

    pp. 1446–1455. Cited by: Posture Estimation in p-HRI.
  • M.T. Alvarez, D. Torricelli, A. J. del-Ama, D. Pinto, J. Gonzalez-Vargas, J. C. Moreno, A. Gil-Agudo, and J. L. Pons (2017) Simultaneous estimation of human and exoskeleton motion: a simplified protocol. In IEEE Int. Conference on Rehabilitation Robotics, pp. 1431–1436. Cited by: Introduction.
  • A. Bazazan, I. Dianat, N. Feizollahi, Z. Mombeini, A. M. Shirazi, and H. I. Castellucci (2019) Effect of a posture correction–based intervention on musculoskeletal symptoms and fatigue among control room operators. Applied ergonomics 76, pp. 12–19. Cited by: Introduction.
  • B. Busch, G. Maeda, Y. Mollard, M. Demangeat, and M. Lopes (2017) Postural optimization for an ergonomic human-robot interaction. In International Conference on Intelligent Robots and Systems, pp. 2778–2785. Cited by: Table 1, Introduction, Introduction, Ergonomics Models and Postural Optimizations in p-HRI.
  • B. Busch, M. Toussaint, and M. Lopes (2018) Planning ergonomic sequences of actions in human-robot interaction. In International Conference on Robotics and Automation, pp. 1916–1923. Cited by: Table 1, Introduction, Introduction, Ergonomics Models and Postural Optimizations in p-HRI, Ergonomics Models and Postural Optimizations in p-HRI.
  • Z. Cao, G. Hidalgo, T. Simon, S. Wei, and Y. Sheikh (2019)

    OpenPose: realtime multi-person 2d pose estimation using part affinity fields

    IEEE transactions on pattern analysis and machine intelligence 43 (1), pp. 172–186. Cited by: Conclusion.
  • L. Chen, L. F. Figueredo, and M. R. Dogar (2018) Planning for muscular and peripersonal-space comfort during human-robot forceful collaboration. In IEEE-RAS International Conference on Humanoid Robotics, pp. 1–8. Cited by: Table 1, Introduction, Ergonomics Models and Postural Optimizations in p-HRI, Ergonomics Models and Postural Optimizations in p-HRI.
  • X. Chen and J. Davis (2000) Camera placement considering occlusion for robust motion capture. Technical report Computer Graphics Laboratory, Stanford University, Tech. Rep. Cited by: Introduction.
  • B. R. da Costa and E. R. Vieira (2010)

    Risk factors for work-related musculoskeletal disorders: a systematic review of recent longitudinal studies

    American journal of industrial medicine 53 (3), pp. 285–323. Cited by: Introduction.
  • S. Das, L. Trutoiu, A. Murai, D. Alcindor, M. Oh, F. De la Torre, and J. Hodgins (2011) Quantitative measurement of motor symptoms in parkinson’s disease: a study with full-body motion capture data. In IEEE Int. Conf. of Engineering in Medicine and Biology Society, pp. 6789–6792. Cited by: Posture Estimation in p-HRI.
  • P. De Boer, D. P. Kroese, S. Mannor, and R. Y. Rubinstein (2005) A tutorial on the cross-entropy method. Annals of operations research 134 (1), pp. 19–67. Cited by: Postural optimization.
  • A. Erol, G. Bebis, M. Nicolescu, R. D. Boyle, and X. Twombly (2007) Vision-based hand pose estimation: a review. Computer Vision and Image Understanding 108 (1-2), pp. 52–73. Cited by: Introduction.
  • D. Gorecky, M. Schmitt, M. Loskyll, and D. Zühlke (2014) Human-machine-interaction in the industry 4.0 era. In 2014 12th IEEE international conference on industrial informatics (INDIN), pp. 289–294. Cited by: Introduction.
  • C. R. Guerrero, J. C. F. Marinero, J. P. Turiel, and V. Muñoz (2013) Using “human state aware” robots to enhance physical human–robot interaction in a cooperative scenario. Computer methods and programs in biomedicine 112 (2), pp. 250–259. Cited by: Introduction.
  • S. Hignett and L. McAtamney (2000) Rapid entire body assessment (reba). Applied ergonomics 31 (2), pp. 201–205. Cited by: Introduction.
  • Y. Jiang and C. K. Liu (2018) Data-driven approach to simulating realistic human joint constraints. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1098–1103. Cited by: Posture Estimation in p-HRI, Human Kinematics Model.
  • M. Kaess, H. Johannsson, R. Roberts, V. Ila, J. J. Leonard, and F. Dellaert (2012) ISAM2: incremental smoothing and mapping using the bayes tree. International Journal of Robotics Research 31 (2), pp. 216–235. Cited by: Conclusion.
  • W. Kim, J. Lee, L. Peternel, N. Tsagarakis, and A. Ajoudani (2017) Anticipatory robot assistance for the prevention of human static joint overloading in human–robot collaboration. IEEE Robotics and Automation Letters 3 (1), pp. 68–75. Cited by: Table 1, Introduction, Ergonomics Models and Postural Optimizations in p-HRI, Ergonomics Models and Postural Optimizations in p-HRI.
  • W. Kim, M. Lorenzini, K. Kapıcıoğlu, and A. Ajoudani (2018) Ergotac: a tactile feedback interface for improving human ergonomics in workplaces. IEEE Robotics and Automation Letters 3 (4), pp. 4179–4186. Cited by: Introduction.
  • M. Kobilarov (2012) Cross-entropy motion planning. International Journal of Robotics Research 31 (7), pp. 855–871. Cited by: Postural optimization.
  • M. Lorenzini, W. Kim, E. De Momi, and A. Ajoudani (2018) A real-time graphic interface for the monitoring of the human joint overloadings with application to assistive exoskeletons. In International Symposium on Wearable Robotics, pp. 281–285. Cited by: Introduction.
  • A. G. Marin, M. S. Shourijeh, P. E. Galibarov, M. Damsgaard, L. Fritzsch, and F. Stulp (2018) Optimizing contextual ergonomics models in human-robot interaction. In International Conference on Intelligent Robots and Systems, pp. 1–9. Cited by: Table 1, Ergonomics Models and Postural Optimizations in p-HRI, Ergonomics Models and Postural Optimizations in p-HRI.
  • A. Mathis, P. Mamidanna, K. M. Cury, T. Abe, V. N. Murthy, M. W. Mathis, and M. Bethge (2018)

    DeepLabCut: markerless pose estimation of user-defined body parts with deep learning

    Technical report Nature Publishing Group. Cited by: Posture Estimation in p-HRI.
  • L. McAtamney and E. N. Corlett (1993) RULA: a survey method for the investigation of work-related upper limb disorders. Applied ergonomics 24 (2), pp. 91–99. Cited by: Introduction.
  • B. W. Mooring, Z. S. Roth, and M. R. Driels (1991) Fundamentals of manipulator calibration. Wiley New York. Cited by: Human Kinematics Model.
  • H. Motabar and A. D. Nimbarte (2021) The effect of task rotation on activation and fatigue response of rotator cuff muscles during overhead work. Applied Ergonomics 97, pp. 103461. Cited by: Introduction.
  • J. Nocedal and S. Wright (2006) Numerical optimization. Springer. Cited by: Postural optimization, Human Subject Study for Posture Estimation.
  • A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala (2019) PyTorch: an imperative style, high-performance deep learning library. In Neural Info. Proc. Sys., Cited by: DULA: Differentiable Upper Limb Assessment Model for Postural Optimization.
  • L. Peternel, W. Kim, J. Babič, and A. Ajoudani (2017) Towards ergonomic control of human-robot co-manipulation and handover. In IEEE-RAS International Conference on Humanoid Robotics, pp. 55–60. Cited by: Table 1, Ergonomics Models and Postural Optimizations in p-HRI, Ergonomics Models and Postural Optimizations in p-HRI, Postural Optimization in Co-Manipulation.
  • L. Peternel, N. Tsagarakis, D. Caldwell, and A. Ajoudani (2018) Robot adaptation to human physical fatigue in human-robot co-manipulation. Autonomous Robots, pp. 1–11. Cited by: Table 1, Ergonomics Models and Postural Optimizations in p-HRI.
  • L. Peternel, C. Fang, M. Laghi, A. Bicchi, N. Tsagarakis, and A. Ajoudani (2020) Human arm posture optimisation in bilateral teleoperation through interface reconfiguration. In IEEE RAS/EMBS International Conference for Biomedical Robotics and Biomechatronics, Cited by: Table 1, Postural optimization in Teleoperation.
  • R. Rahal, G. Matarese, M. Gabiccini, A. Artoni, D. Prattichizzo, P. R. Giordano, and C. Pacchierotti (2020) Caring about the human operator: haptic shared control for enhanced user comfort in robotic telemanipulation. IEEE Tran. on Haptics 13 (1), pp. 197–203. Cited by: Table 1, Introduction, Introduction, Introduction, Posture Estimation in p-HRI, Ergonomics Models and Postural Optimizations in p-HRI.
  • M. Ramaganesh, R. Jayasuriyan, T. Rajpradeesh, S. Bathrinath, and R. Manikandan (2021) Ergonomics hazard analysis techniques-a technical review. Materials Today: Proceedings. Cited by: Introduction.
  • S. Saremi, S. Mirjalili, A. Lewis, A. W. C. Liew, and J. S. Dong (2018) Enhanced multi-objective particle swarm optimisation for estimating hand postures. Knowledge-Based Systems. Cited by: Posture Estimation in p-HRI.
  • A. Shafti, A. Ataka, B. U. Lazpita, A. Shiva, H. A. Wurdemann, and K. Althoefer (2019) Real-time robot-assisted ergonomics. In International Conference on Robotics and Automation, pp. 1975–1981. Cited by: Table 1, Ergonomics Models and Postural Optimizations in p-HRI, Ergonomics Models and Postural Optimizations in p-HRI.
  • F. Sherwani, M. M. Asad, and B. Ibrahim (2020) Collaborative robots and industrial revolution 4.0 (ir 4.0). In 2020 International Conference on Emerging Trends in Smart Technologies (ICETST), pp. 1–5. Cited by: Introduction.
  • A. A. Shikdar and M. A. Al-Hadhrami (2007) Smart workstation design: an ergonomics and methods engineering approach. International Journal of Industrial and Systems Engineering 2 (4), pp. 363–374. Cited by: Introduction.
  • S. Thrun, W. Burgard, and D. Fox (2005) Probabilistic robotics. MIT press. Cited by: Particle Filter for Posture Estimation.
  • L. van der Spaa, M. Gienger, T. Bates, and J. Kober (2020) Predicting and optimizing ergonomics in physical human-robot cooperation tasks. In International Conference on Robotics and Automation, pp. 1799–1805. Cited by: Table 1, Ergonomics Models and Postural Optimizations in p-HRI, Ergonomics Models and Postural Optimizations in p-HRI.
  • P. Vartholomeos, N. Katevas, A. Papadakis, and L. Sarakis (2016) Design of motion-tracking device for intuitive and safe human-robot physical interaction. In IEEE Int. Conference on Telecommunications and Multimedia, pp. 1–6. Cited by: Posture Estimation in p-HRI.
  • L. Vianello, J. Mouret, E. Dalin, A. Aubry, and S. Ivaldi (2021) Human posture prediction during physical human-robot interaction. IEEE Robotics and Automation Letters. Cited by: Posture Estimation in p-HRI.
  • T. Vos, R. M. Barber, B. Bell, A. Bertozzi-Villa, S. Biryukov, I. Bolliger, F. Charlson, A. Davis, L. Degenhardt, D. Dicker, et al. (2015) Global, regional, and national incidence, prevalence, and years lived with disability for 301 acute and chronic diseases and injuries in 188 countries, 1990–2013: a systematic analysis for the global burden of disease study 2013. The Lancet 386 (9995), pp. 743–800. Cited by: Introduction.
  • X. Wang, M. Maurin, F. Mazet, N. D. C. Maia, K. Voinot, J. P. Verriest, and M. Fayet (1998) Three-dimensional modelling of the motion range of axial rotation of the upper arm. Journal of biomechanics 31 (10), pp. 899–908. Cited by: Posture Estimation in p-HRI.
  • X. Xiao and S. Zarar (2018) A wearable system for articulated human pose tracking under uncertainty of sensor placement. In IEEE Int. Conference on Biomedical Robotics and Biomechatronics, pp. 1144–1150. Cited by: Introduction.
  • A. Yazdani, R. S. Novin, A. Merryweather, and T. Hermans (2021a) DULA: a differentiable ergonomics model for postural optimization in physical hri. arXiv preprint arXiv:2107.06875. Cited by: Introduction.
  • A. Yazdani, R. S. Novin, A. Merryweather, and T. Hermans (2021b) Is the leader robot an adequate sensor for posture estimation and ergonomic assessment of a human teleoperator?. In IEEE International Conference on Automation Science and Engineering, Cited by: Introduction, Human Kinematics Model.
  • A. Yazdani and R. Sabbagh Novin (2021) Posture estimation and optimization in ergonomically intelligent teleoperation systems. In Companion of the 2021 ACM/IEEE International Conference on Human-Robot Interaction, pp. 604–606. Cited by: Introduction.