Domain Adaptive Robotic Gesture Recognition with Unsupervised Kinematic-Visual Data Alignment

by   Xueying Shi, et al.

Automated surgical gesture recognition is of great importance in robot-assisted minimally invasive surgery. However, existing methods assume that training and testing data are from the same domain, which suffers from severe performance degradation when a domain gap exists, such as the simulator and real robot. In this paper, we propose a novel unsupervised domain adaptation framework which can simultaneously transfer multi-modality knowledge, i.e., both kinematic and visual data, from simulator to real robot. It remedies the domain gap with enhanced transferable features by using temporal cues in videos, and inherent correlations in multi-modal towards recognizing gesture. Specifically, we first propose an MDO-K to align kinematics, which exploits temporal continuity to transfer motion directions with smaller gap rather than position values, relieving the adaptation burden. Moreover, we propose a KV-Relation-ATT to transfer the co-occurrence signals of kinematics and vision. Such features attended by correlation similarity are more informative for enhancing domain-invariance of the model. Two feature alignment strategies benefit the model mutually during the end-to-end learning process. We extensively evaluate our method for gesture recognition using DESK dataset with peg transfer procedure. Results show that our approach recovers the performance with great improvement gains, up to 12.91 F1score without using any annotations in real robot.


page 1

page 4

page 5

page 7


Relational Graph Learning on Visual and Kinematics Embeddings for Accurate Gesture Recognition in Robotic Surgery

Automatic surgical gesture recognition is fundamentally important to ena...

Joint Surgical Gesture and Task Classification with Multi-Task and Multimodal Learning

We propose a novel multi-modal and multi-task architecture for simultane...

Unsupervised Domain Adaptation for Device-free Gesture Recognition

Device free human gesture recognition with Radio Frequency signals has a...

Automatic alignment of surgical videos using kinematic data

Over the past one hundred years, the classic teaching methodology of "se...

Gesture Recognition in Robotic Surgery: a Review

Objective: Surgical activity recognition is a fundamental step in comput...

Unsupervised Domain Adaptation through Inter-modal Rotation for RGB-D Object Recognition

Unsupervised Domain Adaptation (DA) exploits the supervision of a label-...

On the Metrics and Adaptation Methods for Domain Divergences of sEMG-based Gesture Recognition

We propose a new metric to measure domain divergence and a new domain ad...