Multimodal In-bed Pose and Shape Estimation under the Blankets

by   Yu Yin, et al.

Humans spend vast hours in bed – about one-third of the lifetime on average. Besides, a human at rest is vital in many healthcare applications. Typically, humans are covered by a blanket when resting, for which we propose a multimodal approach to uncover the subjects so their bodies at rest can be viewed without the occlusion of the blankets above. We propose a pyramid scheme to effectively fuse the different modalities in a way that best leverages the knowledge captured by the multimodal sensors. Specifically, the two most informative modalities (i.e., depth and infrared images) are first fused to generate good initial pose and shape estimation. Then pressure map and RGB images are further fused one by one to refine the result by providing occlusion-invariant information for the covered part, and accurate shape information for the uncovered part, respectively. However, even with multimodal data, the task of detecting human bodies at rest is still very challenging due to the extreme occlusion of bodies. To further reduce the negative effects of the occlusion from blankets, we employ an attention-based reconstruction module to generate uncovered modalities, which are further fused to update current estimation via a cyclic fashion. Extensive experiments validate the superiority of the proposed model over others.



There are no comments yet.


page 1

page 2

page 3

page 6

page 7

page 8


Simultaneously-Collected Multimodal Lying Pose Dataset: Towards In-Bed Human Pose Monitoring under Adverse Vision Conditions

Computer vision (CV) has achieved great success in interpreting semantic...

Bodies at Rest: 3D Human Pose and Shape Estimation from a Pressure Image using Synthetic Data

People spend a substantial part of their lives at rest in bed. 3D human ...

Multimodal Densenet

Humans make accurate decisions by interpreting complex data from multipl...

A Benchmark for Gait Recognition under Occlusion Collected by Multi-Kinect SDAS

Human gait is one of important biometric characteristics for human ident...

Multimodal Systems: Taxonomy, Methods, and Challenges

Naturally, humans use multiple modalities to convey information. The mod...

Seeing Under the Cover: A Physics Guided Learning Approach for In-Bed Pose Estimation

Human in-bed pose estimation has huge practical values in medical and he...

DC-GNet: Deep Mesh Relation Capturing Graph Convolution Network for 3D Human Shape Reconstruction

In this paper, we aim to reconstruct a full 3D human shape from a single...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.