MMTSA: Multimodal Temporal Segment Attention Network for Efficient Human Activity Recognition

10/14/2022
by   Ziqi Gao, et al.
0

Multimodal sensors (e.g., visual, non-visual, and wearable) provide complementary information to develop robust perception systems for recognizing activities. However, most existing algorithms use dense sampling and heterogeneous sub-network to extract unimodal features and fuse them at the end of their framework, which causes data redundancy, lack of complementary multimodal information and high computational cost. In this paper, we propose a new novel multimodal neural architecture based on RGB and IMU wearable sensors (e.g., accelerometer, gyroscope) for human activity recognition called Multimodal Temporal Segment Attention Network (MMTSA). MMTSA first employs a multimodal data isomorphism mechanism based on Gramian Angular Field (GAF) and then applies a novel multimodal sparse sampling method to reduce redundancy. Moreover, we propose an inter-segment attention module in MMTSA to fuse multimodal features effectively and efficiently. We demonstrate the importance of imu data imaging and attention mechanism in human activity recognition by rigorous evaluation on three public datasets, and achieve superior improvements (11.13% on the MMAct dataset) than the previous state-of-the-art methods. The code is available at: https://github.com/THU-CS-PI/MMTSA.

READ FULL TEXT

page 4

page 6

research
08/03/2020

HAMLET: A Hierarchical Multimodal Attention-based Human Activity Recognition Algorithm

To fluently collaborate with people, robots need the ability to recogniz...
research
06/25/2020

DanHAR: Dual Attention Network For Multimodal Human Activity Recognition Using Wearable Sensors

Human activity recognition (HAR) in ubiquitous computing has been beginn...
research
01/25/2016

Egocentric Activity Recognition with Multimodal Fisher Vector

With the increasing availability of wearable devices, research on egocen...
research
11/08/2022

Multi-Stage Based Feature Fusion of Multi-Modal Data for Human Activity Recognition

To properly assist humans in their needs, human activity recognition (HA...
research
11/14/2022

Wearable-based Human Activity Recognition with Spatio-Temporal Spiking Neural Networks

We study the Human Activity Recognition (HAR) task, which predicts user ...
research
05/02/2020

Deep ConvLSTM with self-attention for human activity decoding using wearables

Decoding human activity accurately from wearable sensors can aid in appl...
research
02/28/2023

MateRobot: Material Recognition in Wearable Robotics for People with Visual Impairments

Wearable robotics can improve the lives of People with Visual Impairment...

Please sign up or login with your details

Forgot password? Click here to reset