A User-Centered Interface for Enhanced Conjoined Human-Robot Actions in Industrial Tasks

05/20/2021 ∙ by Juan M. Gandarias, et al. ∙ Istituto Italiano di Tecnologia 0

This paper presents a user-centered physical interface for collaborative mobile manipulators in industrial manufacturing and logistics applications. The proposed work builds on our earlier MOCA-MAN interface, through which a mobile manipulator could be physically coupled to the operators to assist them in performing daily activities. The new interface instead presents the following additions: i) A simplistic, industrial-like design that allows the worker to couple/decouple easily and to operate mobile manipulators locally; ii) Enhanced loco-manipulation capabilities that do not compromise the worker mobility. Besides, an experimental evaluation with six human subjects is carried out to analyze the enhanced locomotion and flexibility of the proposed interface in terms of mobility constraint, usability, and physical load reduction.



There are no comments yet.


page 1

page 2

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

It is a fact that the introduction of robotic technologies in industrial environments has been one of the breakthroughs in manufacturing and logistics applications. However, most current industrial robotic systems lack the flexibility and reconfigurability to quickly and easily adapt to workers’ demands. Recently, the awakening of collaborative robotic technologies opens up a new horizon of opportunities for these [ajoudani2020smart] and other applications, such as healthcare [ruiz2021upper] or disaster response [kruijff2014designing].

However, humans’ dexterity and mobility create unstructured environments where tasks require a high degree of flexibility in perception, motion and control that current robots lack [ajoudani2018progress]. Some of the most relevant challenges include a high level of complexity due to kinematic redundancy, unstructured working environments, coexistence/cooperation with human workers, and shortage of systematic industrial design of physical Human-Robot Interaction (pHRI) interfaces.

Existing collaborative technologies for industrial applications can be categorized as exoskeletons [sylla2014ergonomic], supernumerary limbs [parietti2014bracing] and collaborative robotic manipulators (cobots) [peshkin2001cobot]. Exoskeletons can have unintended negative consequences such as reduced flexibility, which can lead to new sources of musculoskeletal disorders (MSDs) and accidents [fox2020exoskeletons]. Another obstacle to the use of exoskeletons is the non-acceptance due to the general discomfort of wearing them [vatsal2017wearing]. Supernumerary limbs, which can augment human capabilities, imply fewer mobility limitations to their users compared to exoskeletons. However, since the users carry the extra limbs, they can experience fatigue in the long term. A research study on the user-centered design requirements for a wearable supernumerary robotic arm in construction states that workers prioritize greater dexterity and lighter weight [vatsal2017wearing]. On the other hand, Cobots contribute to better ergonomics and comfort of their human counterparts [kim2019adaptable].

Fig. 1: The proposed interface allows for conjoined loco-manipulation actions without constraining the user’s motion in the joint space. The interface is evaluated in a collaborative painting task, representing an effort-demanding industrial activity.
Fig. 2: (a) A CAD rendered image of the user-centered physical interface. (b) The MOCA robotic platform integrates a mobile base and a cobot manipulator. (c) The complete MOCA-MAN system with the proposed interface. The sEMG and XSens sensors are used only for experimental purposes and are not necessary for the regular use of the MOCA-MAN framework.

This paper builds on our previous work [kim2020moca], in which we presented MOCA-MAN. The proposed framework intended to exploit the reconfiguration potential of a MObile Collaborative robot Assistant (MOCA) as a collaborative tool that allowed the users to vary between autonomous and conjoined modes (see Fig. 1). The focus of the work was the conjoined mode, in which the users could physically attach to MOCA and extend their loco-manipulation capabilities. In this direction, a physical interface is introduced in this paper that implements enhanced features for industrial applications w.r.t. our previous work, from the conjoined manipulation standpoint. In particular, the contribution and novelty of this paper are two-folded:

  • An admittance-type physical interface with a simplistic and systematic industrial design for enhanced conjoined loco-manipulation actions integrated in the MOCA-MAN framework.

  • An experimental evaluation with six human subjects performing an industrial-like painting activity. The physical load and mobility restrictions are analyzed and compared when conducting the task with and without MOCA-MAN. Besides, a usability questionnaire is carried out to evaluate users’ subjective opinions and preferences.

The remainder of the paper is organized as follows: Section II presents an overview on the MOCA-MAN system, describing the concept, the hardware and components, and the control framework. Section IV presents the experiments and a discussion on the results. Finally, Section V presents the conclusions and prospective works.

Ii System Overview

This section describes the MOCA-MAN concept and the components of the system used in this paper. Fig. 2 details the components used in this work. The system is composed of two main elements: (a) a user-centered physical interface (see section II-A) and (b) the MOCA robotic platform (see section II-B). Besides, a motion capture system (MoCap) and a surface electromiography (sEMG) sensor-based system have been used to perform the experiments, as demonstrated in Fig. 2(c). Both systems are presented in section IV.

Ii-a User–Centered Physical Interface

As aforementioned, the MOCA-MAN concept was previously presented in [kim2020moca]. That work also described this system’s potential and main benefits for industrial Human-Robot Collaboration (HRC) tasks. The system presented was composed of a physical interface with a magnetic coupling system that allowed the worker to attach/detach to the robot and switch MOCA operating modes between autonomous and local operation.

However, certain aspects could be improved regarding the local operation mode for application to real industrial tasks. First, the worker needed to use both hands to control the system. One hand was required to locally attach and operate the MOCA. On the other side, changing the mode and system settings were done with an sEMG sensor located on the other arm. Gesture control, although intuitive and easy to use, may fail in recognition. Second, the magnetic coupling restricted the worker’s mobility to some extent. Overall, this paper presents an enhanced physical interface to improve our previous system in terms of robustness and practicality for logistics and manufacturing tasks.

The proposed physical interface is presented in Fig. 2 (a) and consists of four main components: i) An Arduino Nano microcontroller and a button panel that allows the user to configure different parameters and communicate with the MOCA through the platform Robot Operating System (ROS); ii) A force and torque sensor (FT Sensor) to measure the worker interaction forces; iii) and an admittance-type controller to locally operate the mobile manipulator (see section III-B). In summary, the following features are exhibited by the proposed interface:

  • A simplistic design that promotes usability and eases the human-robot coupling. As a result, the whole MOCA-MAN system can be operated locally using one hand only. Besides, the interface does not impede the worker’s mobility, allowing for a better postural adjustment, and improving the ergonomics during conjoined actions.

  • User-centered enhanced capabilities. This way, the worker can adjust different controller parameters online, adapting the MOCA behavior based on their preferences.

  • The interface is programmable and configurable, allowing for better flexibility among the MOCA-MAN functionalities. In this respect, the system functionalities can be easily changed depending on the requirements of the task.

Ii-B The MOCA Robotic Platform

The other main component of the proposed system is the MOCA mobile manipulator (see Fig. 2 (b)), which has been previously presented in [wu2019teleoperation]. The admittance interface described in the previous section is placed on the robotic arm’s end-effector near the SoftHand. The system, besides, is composed of a Robotnik SUMMIT-XL STEEL mobile platform and a 7 DoF Franka Emika Panda manipulator. The complete system is controlled by a whole-body impedance controller described in detail in section III-A. This controller ensures a compliance behavior of the end-effector when physical interactions with the environment occur. This is a crucial requirement both for safety reasons and for executing joint human-robot manipulation tasks.

Iii Control Framework

Iii-a Whole-body Impedance Controller

This section describes how the basic weighted whole-body Cartesian impedance controller was designed starting from the dynamic model of MOCA and how the control modes are generated [lamon2020towards]. An advanced version of such controller can be found in [wu2021unified].

The whole-body dynamic model can be formulated as the series connection of the mobile base with the arm. While the arm can receive torque inputs at , the mobile platform is controlled by means of a low-level velocity controller that maps velocities expressed in the joint space of the platform onto an angular velocity applied to the omni-wheels, at . To map high-level torque references into suitable velocities for the base , we make use of an admittance controller, with virtual inertia and damping, and , respectively.

The resulting whole-body decoupled dynamics is:



is the arm joint angles vector,

is the symmetric and positive definite inertial matrix of the arm, is the Coriolis and centrifugal force, is the gravity vector, , and are the commanded torque vector and external torque vector, respectively.

It is important to highlight that the value of and could be tuned to obtain the desired responsiveness of the system (the higher , the higher the torques that should be applied to generate an acceleration and similarly for , where higher values generates smoother, and hence slower, responses of the system) [lamon2020visuo].

Equation (1) can be summarized by


where is the symmetric positive definite joint-space inertia matrix, is the joint-space Coriolis/centrifugal matrix, and the joint-space gravity. Finally, and represent joint-space input and external torque, respectively.

The whole-body impedance controller generates high level torque references that are then passed to the mobile platform admittance controller and to the arm low-level torque controller (that compensates for the joint-level torque due to gravity and Coriolis/centrifugal). Such torques are defined in the following way (for the sake of readability, the dependencies are dropped from now on):


that fulfills the general relationship between the generalized joint torques and the Cartesian generalized force (the tracked reference) , where is the dynamically consistent Jacobian, can be regarded as the weighted Cartesian inertia, and is the Cartesian inertia. Finally, can be used to generate torques that does not interfere with the Cartesian force , since they are projected onto the null-space of the Cartesian task space.

The positive definite weighting matrix is defined as:


where is the tunable positive definite weight matrix of the controller. In particular, in this work, is diagonal and dynamically selected depending on the task:


where , are constant scalar values, a higher value of this will penalize the motion of that joint. For instance, to obtain higher mobility of the arm than the base, we set .

The desired impedance behaviors are obtained by:




In particular, in this work, the reference Cartesian positions are defined by the admittance-type interface in sec. III-B, and the projected task is used to minimize the arm motions with respect to the desired configuration .

Iii-B Admittance-type Interface for Local Operation

The proposed interface allows the user to command the desired end-effector’s velocities and positions . An admittance control law [cherubini2016collaborative] was developed to transfer the human’s interaction forces to the desired end-effector movements. The admittance relationship is given by


where and are the positive definite diagonal inertia and damping matrices, and represents the forces measured by the FT sensor as the difference of and the Cartesian force generated by the whole-body impedance controller at the end-effector . Since the installed FT sensor frame is displaced from the end-effector frame (see Fig.2), the measured forces are transformed to compute . Besides, to command the desired positions from equation (8), is transformed into via a discrete-time integration.

Iv Experiments and Results



Fig. 3: (a) Snapshots of the experiment conducted with (blue) and without (red) the MOCA-MAN interface. (b) Comparison of the sEMG values (AD and BC) and human right hand pose in the experiments performed with (blue) and without (red) the MOCA-MAN interface.

Iv-a Experimental Protocol and Setup

Six healthy volunteers, three males and three females, (age: years; mass: Kg; height: cm)111Subject data is reported as: mean standard deviation. were recruited in the experimental session. Written informed consent was obtained after explaining the experimental procedure and a numerical ID was assigned to anonymise the data. The whole experimental activity was carried out at Human-Robot Interfaces and Physical Interaction (HRII) Lab, Istituto Italiano di Tecnologia (IIT) in accordance with the Declaration of Helsinki. The protocol was approved by the ethics committee Azienda Sanitaria Locale (ASL) Genovese N.3 (Protocol IIT_HRII_ERGOLEAN 156/2020).

During the experiments, the Xsens MVN Biomech suit, an inertial-based system commercialized by Xsens Technologies B.V. (Enschede, Netherlands), was employed to measure the whole-body motion. Muscle activity was recorded using the Delsys Trigno platform, a wireless sEMG system commercialised by Delsys Inc. (Natick, MA, United States). sEMG signals were measured in two locations on the subjects’ right arm: Anterior Deltoid (AD), and Biceps (BC). Afterwards, they were filtered and normalized to Maximum Voluntary Contractions (MVC). Each subject was asked to perform a task in two different conditions: with and without the MOCA-MAN assistance. Fig. 3 (a) depicts an excerpt of the video in both conditions. The measured sEMG signals and Cartesian positions of the human’s right hand from one particular subject are shown in Fig. 3 (b). The task includes two different phases described below. A video of the experiment is available in 222https://www.youtube.com/watch?v=WgnJihYHjYo.

Iv-A1 Phase 1: Grasping and Carrying

The subjects are asked to grasp a paint sprayer ( kg) and move toward a target position in front of a wall. When the task is performed without the MOCA-MAN, the subjects had to grasp the tool with their right hand and move alone toward the target. On the other hand, when assisted by the MOCA-MAN, the subjects have to accomplish the task by taking advantage of the dedicated buttons board. The sequence of actions is the following: i) activate the MOCA-MAN, whose mode was initially set on ”manipulation”; ii) grasp the painting tool with the SoftHand, iii) change the mode to ”locomotion”; iii) move toward the target guiding the MOCA through the admittance interface.

Iv-A2 Phase 2: Painting

In the second phase, the subjects are required to paint the wall, ahead of the target position, by following two predefined paths marked on its surface (see Figure 1), back and forth, one after the other. Without the MOCA-MAN the subjects have to carry out the task with their own strengths. Conversely, when assisted by the MOCA-MAN, the subjects could count on its support to perform the painting activity.

A comparison on the subjects’ muscle activity, when performing the task with and without the MOCA-MAN, is carried out to report the benefits in terms of reduction of the physical load. Besides, an analysis of the mobility constraint is carried out. This analysis is based on evaluating the cross-correlation index of the main joints involved during phases 1 and 2 of the experiment. The joints chosen are the knee for locomotion and the shoulder and elbow for manipulation. The cross-correlation index between a pair of series and , where and , measures the similarity in shape of the two curves as a scalar between 0 and 1. In this case, the series and are obtained from the Xsens sensors measurements when performing the task with and without the MOCA-MAN, and is computed as in [wren2006cross]. In order to evaluate the usability of the proposed interface, the subjects are asked to carry out two questionnaires. Single Easy Question(SEQ): a post-task single-question measuring users’ perception of usability [brook1996quick]. System Usability Scale (SUS): ten different questions that addressed the usability and learn ability of a system [sauro2009comparison]. The results of the experiments and questionnaires are reported in the next section.

Iv-B Results and Discussion

Iv-B1 Motion Analysis

Fig. 4 (a) shows the data considered for the cross-correlation analysis of one particular subject. The blue and green sections define the data considered for the analysis during locomotion and manipulation phases, respectively. Fig. 4 (b) and (c) represent the R-value from the data mentioned in phases 1 and 2. In this particular case, it can be seen how the motion performed by the user at the joint level is similar when executing the task with and without MOCA-MAN. In particular, it can be seen how the similarity is more significant in the case of manipulation than in locomotion. Moreover, in Fig. (b), the curves are not centered at 0, which shows that the locomotion without MOCA-MAN is faster than with MOCA-MAN. This is also reported in Fig. (a), although the curves’ shape still presents certain similarities. The results of the cross-correlation analysis for all the subjects are included in table I.


(b)                                (c)

Fig. 4: Motion analysis of one particular subject: (a) Knee, Shoulder and Elbow joint angles with (top) and without (bottom) MOCA-MAN. (b), (c) correlation indexes during phases 1 and 2, respectively.
7.17 30.42 7.48 22.59
S1 20.37 35.92 64.80 15.31 19.72 26.60 62.07 15.07 0.31 0.85 0.85
19.68 42.83 10.78 21.10
S2 31.84 57.34 38.19 25.30 23.62 48.44 54.36 35.80 0.48 0.80 0.78
29.89 62.87 9.53 24.78
S3 37.87 96.38 21.07 34.77 17.60 66.39 45.85 62.67 0.33 0.82 0.77
33.48 65.46 3.75 21.20
S4 67.5 100 50.40 34.54 10.62 30.00 64.67 29.33 0.35 0.81 0.76
12.96 33.57 2.48 9.67
S5 26.3 55.82 50.72 39.86 22.98 45.89 89.21 78.93 0.57 0.96 0.72
18.63 38.98 3.10 16.10
S6 44.41 100 58.05 61.02 15.04 40.36 79.39 60.11 0.33 0.66 0.63
Mean 20.30 (9.95) 45.69 (14.96) 47.21 35.13 6.19 (3.55) 20.91 (7.36) 65.93 46.99 0.40 0.82 0.75
(std) 38.05 (16.72) 74.24 (27.97) (15.58) (15.38) 18.26 (4.95) 42.95 (14.35) (15.98) (24.06) (0.11) (0.09) (0.07)
TABLE I: Muscle activation of the anterior deltoid (AD) and biceps (BC), and cross-correlation index () for each subject (S1, …, S6). Results are in %, except for the cross-correlation coefficient, which ranges from 0 to 1. In the cells with two background colors: white–with Moca-Man, gray–without Moca-Man. (Mean, Maxmimum, Knee, Shoulder, Elbow).

Iv-B2 Physical Load Analysis

The results of this analysis are reported in table I for the six subjects. Overall, they demonstrate that muscle activity is considerably reduced. Specifically, both mean and maximum muscle activity are decreased, achieving a reduction of 47.21% and 35.15% in the AD, and of 65.93% and 46.99% in the BC, respectively.

Iv-B3 Usability Analysis

The results of the usability analysis are reported in Fig. 5. In general, they imply that performing the proposed task with MOCA-MAN involves little effort. Furthermore, they assume that the interface is intuitive and easy to use and report the proposed interface’s potential for performing industrial loco-manipulation tasks.

Fig. 5: Results of the SEQ (yellow) and SUS (green) questionnaires for the six subjects.

V Conclusions

In this paper, we have presented a user-centered physical interface for collaborative mobile manipulators in industrial applications. The advantages introduced by the interface proposed in this work have been described. Besides, an experiment has been carried out with human subjects performing a painting activity. The user’s mobility and the variation of physical effort during the task have been analyzed, plus a questionnaire to evaluate the system’s usability. The outcomes have demonstrated the potential and suitability of the proposed interface for conducting typical industrial tasks. Future work will expand the use of this interface to increase flexibility and allow the user to change other control parameters of the robot, such as the interface’s admittance. We also propose to perform a comparative analysis of the proposed system with other assistive robotic systems such as exoskeletons or supernumerary limbs.