Telepresence systems allows users to perform various tasks in a remote environment . This functionality supports a broad range of applications, such as teleconferencing , remote healthcare [3, 4], and inspection in hazardous environments [5, 6]. Teleoperated mobile robots are used to perform complex tasks in remote environments and can be considered as a particular example of telemanipulation systems . One challenge is to have a low-cost mobile robot with manipulation ability that can be easily operated semi-autonomously to perform simple tasks in the remote environment.
In this regard, telepresence has been applied on different robotic platforms, mostly with simple locomotion systems equipped with a monocular camera and a screen as a human-machine interface [8, 9]. Although these platforms provide the basic functionality for telepresence applications, they incorporate limited human morphology and mostly do not consider bidirectional user interaction. In particular, the usage of a single camera for capturing the remote environment provides only a limited illusion of spatial attendance. Furthermore, classic motion control of mobile platforms in today’s telepresence systems are often restricted to simple interaction devices. Accordingly, an effective manipulation capability and proficient locomotion system along with actuated camera systems are all equally important to provide a better immersion and telepresence experience .
In this paper, we introduce the MAVI telepresence platform which is designed specifically for semi-autonomous indoor teleoperation scenarios. MAVI is a low-cost telepresence robot with omnidirectional holonomic locomotion, a cylindrical 7DoF manipulator, and an actuated stereoscopic vision system. This robot is designed to perform a variety of remote mobile manipulation tasks in an everyday environment. Our design can simply execute the basic and complex commands of a human user in an indoor area and provides most of the sensing modalities such as visual and force feedback placed in different parts of the platform.
The structure of this report is as follows: In Section II an overview of the proposed telepresence system is given. Then, in Section III, the MAVI mobile robot hardware architecture is explained. In Section IV, the software framework is discussed and the structure of the node based system is shown. Also, the control packages on the robot side are illustrated. Section V concludes with final remarks.
Ii Telepresence System Overview
The proposed telepresence system consisted of a client station (User) and the server station (Robot), both remotely located and connected to the communication network. The user can control the MAVI platform in an indoor environment by sending the commands from the client side. The software framework for the MAVI platform is based on ROS (Robot Operation System), with the transmission of the data between the robot and the client being defined as ROS based messages .
Server Station (Robot)
The MAVI platform design is based on three main units: omnidirectional locomotion system, cylindrical manipulator, and stereoscopic Pan-Tilt-Roll Unit (PTRU). The robot is able to connect to the 2.4 or 5.0GHz wireless network to transmit and receive the visual, auditory, haptic and motion data. In order to make the teleoperation as accurate and simple as possible, high-level commands are transmitted to control the manipulator and locomotion system while the local control loop on the robot side generates appropriate actuation signals. An actuated PTRU with high-resolution camera can provides a stereoscopic 360
view of the environment and can mimic the user’s head motion. MAVI is equipped with a variety of sensors to estimate its state within the indoor environment and provide high-level feedback to be easily operated. The MAVI locomotion system is an omnidirectional holonomic system which allows the user to easily maneuver. The manipulator ability of the platform supports simple picking and placing of objects by its large payload and working-space.
Client Station (User)
On the user side, our system acquires and sends the high-level commands for locomotion, manipulation, and head motion as well as provides video, audio, and haptic feedback to the user. In order to simplify the user interface to control the robot, we are using a gesture armband from Wearable Technology by Thalmic labs . This armband contains two types of sensors: high rate Inertial Measurement Unit (IMU) and EMG sensors for capturing the muscle activities of the arm and the fingers. Generating commands according to the user’s arm orientation and their gestures allows us to control both locomotion and the manipulation system of the MAVI platform. To provide video and audio feedback, the client side is equipped with a Head-Mounted-Display (HMD) combined with headphones to show the stereo video and play the audio signals. To provide haptic feedback, we are currently using the vibration ability of the armband. An internal IMU of the HMD is used to estimate the users head motion which is sent to the robot side to control the PTRU.
Iii Mavi Architecture
As shown in Fig. 1, MAVI consists of a human-size torso and a mobile base with 1.65 m height and 0.42 m in both width and depth. The platform is designed for an indoor environment and a working time of 4 to 8 hours depending on the different scenarios. MAVI is constructed mostly from light-weight 3D-printed parts. Robotis Dynamixel actuators drive all joints in the robot manipulator, the PTRU, and the locomotion system . This allows for a light-weight and low-cost construction, compared to available telepresence robots. The MAVI hardware structure is divided into the following parts:
Omnidirectional locomotion of MAVI is ensured by a Mecanum drive mechanism to maneuver in the narrow passages found in indoor areas. We developed an omnidirectional driving system based on four Mecanum wheels equipped with a suspension system to be able to smoothly bridge irregularities at the ground (see Fig. 2). An omnidirectional locomotion system based on four Mecanum wheels, compared to other traditional holonomic systems, can carry more weight and provides more efficient movement [14, 15]. On the other hand, one of the problems for this structure is the gap between each freewheel which causes a periodic vibration in the robot. Also critical are bumps or puddles on the ground, which not all the wheels may touch the ground plane to generate the desired movement, as a result, driving of the robot cannot be controlled appropriately .
To address these issues, all the wheels need to be connected to a suspension system to not only absorb the vibration while the robot is moving but also to push all the wheels to touch the ground. Fig. 2(a), shows the structure of the wheels actuation mechanism with the linear vertical suspension system. The Mecanum wheel and the actuator are assembled as one unit and connected to two cylinder slides trough linear bearings from sides, while the absorber spring placed in between the body plate and the linear bearings.
To control the velocity of the robot on a planner surface, the inverse kinematic of the locomotion system needs to be calculated . Fig. 2(b), shows the schematic of the designed system for the platform and illustrates the axes of the locomotion system. By considering the design parameters, the inverse kinematic of the robot locomotion is defined as follow:
where and are the longitudinal and the lateral velocities, and is the angular velocity of the mobile robot. The rotating speed of the i-th wheel is in rad/s. In addition, and are the distance between the wheels in longitudinal and lateral direction respectively, and is the radius of the Mecanum wheel. Meanwhile, according to the rotation feedback from the internal shaft encoder of each wheel , we calculate the odometery of the robot as follows:
where is the time elapsed, and are the positions of the robot in meter along with longitude and latitude, respectively, and is the orientation of the robot in radian relative to the starting point.
The high payload capability of the MAVI manipulator is provided by its particular design. It is a 7DoF cylindrical manipulator, consisting of a vertical linear actuated body mounted on a locomotion base, a 3DoF planar portion followed by a 2DoF wrist (pitch and roll) and equipped with a parallel gripper facilitating a payload of 1.0kg. The planar part of the manipulator provides a mechanical support and increases the amount of force that can applied to the end-effector given the large lever arm (see Fig. 3). The workspace of the manipulator is shown in Fig. 4(a). It is slightly constrained but can be compensated by the robots mobility. We further added another movable gripper to the base to avoid complex motion plannings for trivial obstacle removal or pick and place tasks. The resulting joint manipulation capabilities enable a performant task accomplishment.
The anthropomorphic manipulation control by the user can be realized by mapping the user’s arm gesture, e.g. using the gesture band in , to 6D control commands. To simultaneously control the manipulator and provide real-time force feedback, we implemented a compliance control for the joints. This functionality is enabled by the joints with back-drivable servo actuators. Rather than using a time expensive inverse kinematic (IK) solver to control the custom-designed MAVI manipulator, we resolved the IK problem algebraically. We can determine the revolute and prismatic joint parameters of the manipulator, and respectively, from a desired pose command of the user () with which are the length of the joint links (see Fig. 3 and Fig. 4(a)).
where and are the temporary position for the end-effector and used to calculate the final joint orientations.
A stereo camera system is mounted on a 3DoF actuated PTRU and follows the yaw, pitch, and roll (ZYX) Euler convention with a large working space as shown in Fig. 4(a). The distance between the cameras is adjustable from 60mm about +/-10mm according to the user. Any desired head orientation triggered by the user is converted to quaternions. The correct conversion of an arbitrary head orientation to the angles , , and subject to our custom PTRU design can be obtained with the following equation:
Processing and Sensing
All the control boards (actuators and the sensor drivers) are designed as a distributed module . Each module connected to a system serial bus and communicates over Half-Duplex Serial TTL at 1.0Mbps. MAVI perceives its environment with a variety of sensors. The robot senses the surrounding with a 360 Lidar (RP-LIDAR A2) , along with twelve ultrasonic rangefinders. They are mainly used for 2D mapping and localization as well as obstacle avoidance and providing a virtual feedback of the obstacle proximity on the user side. MAVI is also equipped with two 9-Axis inertial motion sensors (BNO055 IMU) , one mounted on the body frame and the other on the PTRU (next to the stereo camera). Each IMU sensor provides filtered absolute orientation, linear acceleration, and the rotation speed for all three axes and up to 100Hz.
The main processing unit on MAVI is an Intel NUC, a powerful 4x4-inch mini PC with Core-i7 CPU and 32GB of RAM . Two stereo cameras with fish-eye lens are connected through a 1.0Gbps Ethernet switch to the processing unit. The other additional modules like microphone and speaker are connected to the main processor’s sound card.
Iv Software Framework
The software architecture of the MAVI platform is based on the Robot Operating System (ROS) . ROS is an open source framework, including different tools, packages and libraries that make it possible to run and implement different robotic components to communicate with each other. Individual programs, also called nodes, communicate over topics which is a name for a stream of messages on the system. A ROS message consists of a nested structure of variables or objects. The ROS framework also enables the communication between multiple computers running in the same network. This ability of the ROS framework allow us to access all the sensory data from the server station. This means only critical real-time nodes needs to run on the robot’s internal processor while the other non real-time control nodes can be executed in the client station (user side). Fig. 5 shows the structure of the nodes for the MAVI telepresence platform. In the first layer (Driver Layer), a Device Communication Manager (DCM), which implemented as a ROS package, is running at 100Hz reading all the sensory data in real-time. Nodes in this package communicate to all the devices which connected to the serial bus in the platform. As shown in Fig. 4(b), all the sensors and actuators are connected to the serial bus to communicate with DCM. Each module in the bus has a unique id address and this structure allows us to easily add or remove different hardware modules from the platform. In the middle layer (Control Layer) of the software framework, MAVI has four main packages:
Locomotion Controller: Reads the sensory information from the wheels (encoder, voltage, current, and force) along with body IMU data and controls the omnidirectional movement of the robot.
Manipulator Controller: To control the cylindrical manipulator for given trajectory poses. This unit runs at 100Hz and it uses all the sensory data from the manipulator joints (such as: position, speed, force, current, temperature and voltage) to control the manipulator and the parallel gripper.
PTRU Controller: This package uses all the sensory data from the actuators of the PTRU and also the head IMU data and controls the motion of the PTRU actuators at 100Hz. We are using a delay compensation strategy for actuated stereoscopic vision system based on head motion prediction [23, 24]. Publishing the stereo camera information is also integrated in this package and runs at 30Hz.
Localization and Mapping: This unit uses a laser range finder, odometery output, ultrasonic range finder and the body IMU data for SLAM. It utilizes Gmapping package to provide a map and the position of the robot in an indoor environment . This control loop is running under 10Hz and provides the path planning and a 2D map of the indoor environment.
In this report, we present our low-cost MAVI telepresence robot platform and provide a general system description. The proposed telepresence omnidirectional robot is specifically designed for an indoor tele-operation scenario. While typical telepresence robots mostly can not interact within the environment in a multimodal way, we introduce our novel design of a cylindrical manipulator with large payload and customized working space. It is also equipped with the parallel gripper with adaptive fingers to provide easy and reliable remote manipulation. To increase the reliability and performance of the holonomic locomotion system, we adapt the simple four wheel Mecanum drive structure with suspension mechanism. To provide better immersion in the remote environment, we integrate a stereoscopic vision system with actuated Pan-Tilt-Roll Unit. Finally, we discussed about our software architecture which is based on ROS and described the basic control packages and nodes running on different layers.
-  S. Tachi, “Telexistence: Enabling humans to be virtually ubiquitous,” IEEE computer graphics and applications, vol. 36, no. 1, pp. 8–14, 2016.
-  C. Neustaedter, G. Venolia, J. Procyk, and D. Hawkins, “To beam or not to beam: A study of remote telepresence attendance at an academic conference,” in Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing. ACM, 2016, pp. 418–431.
-  D. Anton, G. Kurillo, A. Y. Yang, and R. Bajcsy, “Augmented telemedicine platform for real-time remote medical consultation,” in International Conference on Multimedia Modeling. Springer, 2017, pp. 77–89.
-  S. Koceski and N. Koceska, “Evaluation of an assistive telepresence robot for elderly healthcare,” Journal of medical systems, vol. 40, no. 5, p. 121, 2016.
-  M. Lombard, F. Biocca, J. Freeman, W. IJsselsteijn, and R. J. Schaevitz, Immersed in media: Telepresence theory, measurement & technology. Springer, 2015.
-  M. Jakuba, A. Bowen, C. German, L. Whitcomb, J. Kinsey, D. Yoerger, L. Mayer, C. McFarland, S. Suman, J. Bailey, et al., “Nereid under ice (nui): A hybrid remotely operated vehicle for under ice telepresence,” in American Geophysical Union, Ocean Sciences Meeting 2016, abstract# IS14A-2301, 2016.
-  F. Piltan and A. Badri, Methodology and Tools in Telemanipulation Systems, 1st ed. United Kingdom: Auris Reference, 2015.
-  S. Laniel, D. Létourneau, M. Labbé, F. Grondin, and F. Michaud, “Enhancing a beam+ telepresence robot for remote home care applications,” in Virtual Rehabilitation (ICVR), 2017 International Conference on. IEEE, 2017, pp. 1–2.
-  J. Zhang, E. Langbehn, D. Krupke, N. Katzakis, and F. Steinicke, “A 360 video-based robot platform for telepresent redirected walking,” no. 1, 2018.
-  J. Steuer, “Defining virtual reality: Dimensions determining telepresence,” Journal of communication, vol. 42, no. 4, pp. 73–93, 1992.
-  M. Quigley, K. Conley, B. Gerkey, et al., “Ros: An open-source robot operating system [c/ol],” 2016.
-  “Myo gesture control armband - wearable technology by thalmic labs.” [Online]. Available: https://www.myo.com/
-  “Robotis dynamixel mx-106.” [Online]. Available: http://support.robotis.com/en/product/actuator/dynamixel/mxseries/mx-106
-  G. Ishigami, K. Iagnemma, J. Overholt, and G. Hudas, “Design, development, and mobility evaluation of an omnidirectional mobile robot for rough terrain,” Journal of Field Robotics, vol. 32, no. 6, pp. 880–896, 2015.
-  C. Hendrix and W. Barfield, “Presence within virtual environments as a function of visual display parameters,” Presence: Teleoperators & Virtual Environments, vol. 5, no. 3, pp. 274–289, 1996.
-  M. O. Tatar, C. Popovici, D. Mândru, I. Ardelean, and A. Plesa, “Design and development of an autonomous omni-directional mobile robot with mecanum wheels,” in Automation, Quality and Testing, Robotics, 2014 IEEE International Conference on. IEEE, 2014, pp. 1–6.
-  “Adaptive gripper fingers dhas - festo dhas-gf-120-u-bu.” [Online]. Available: https://www.festo.com/cat/en-us/search?query=DHAS-GF-120-U-BU.html
-  M. Quigley, A. Asbeck, and A. Ng, “A low-cost compliant 7-dof robotic manipulator,” in Robotics and Automation (ICRA), 2011 IEEE International Conference on. IEEE, 2011, pp. 6051–6058.
-  M. Karimi, A. Ahmadi, N. K. Korghond, E. Babaians, and S. S. Ghidary, “Remoro; a mobile robot platform based on distributed i/o modules for research and education,” in Robotics and Mechatronics (ICROM), 2015 3rd RSI International Conference on. IEEE, 2015, pp. 657–662.
-  T. Huang, “Rplidar a2 - slamtec - leading service robot localization and navigation solution provider.” [Online]. Available: https://www.slamtec.com/en/Lidar/A2
-  “Bno55 - 9-axis absolute orientation sensor.” [Online]. Available: https://www.bosch-sensortec.com/bst/products/allproducts/bno055
-  “Intel nuc 7 enthusiast mini pc - nuc7i7bnhxg.” [Online]. Available: https://www.intel.com/content/www/us/en/products/boards-kits/nuc/mini-pcs/nuc7i7bnhxg.html
-  T. Aykut, S. Lochbrunner, M. Karimi, B. Cizmeci, and E. Steinbach, “A stereoscopic vision system with delay compensation for 360° remote reality,” in Proceedings of the on Thematic Workshops of ACM Multimedia. ACM, 2017, pp. 201–209.
T. Aykut, C. Burgmair, M. Karimi, J. Xu, and E. Steinbach, “Delay compensation
for actuated stereoscopic 360 degree telepresence systems with probabilistic
head motion prediction,” in
Winter Conference on Applications of Computer Vision (WACV). IEEE, 2018.
-  G. Grisetti, C. Stachniss, and W. Burgard, “Improved techniques for grid mapping with rao-blackwellized particle filters,” IEEE transactions on Robotics, vol. 23, no. 1, pp. 34–46, 2007.