Efficient 3D Reconstruction, Streaming and Visualization of Static and Dynamic Scene Parts for Multi-client Live-telepresence in Large-scale Environments

11/25/2022
by   Leif Van Holland, et al.
0

Despite the impressive progress of telepresence systems for room-scale scenes with static and dynamic scene entities, expanding their capabilities to scenarios with larger dynamic environments beyond a fixed size of a few squaremeters remains challenging. In this paper, we aim at sharing 3D live-telepresence experiences in large-scale environments beyond room scale with both static and dynamic scene entities at practical bandwidth requirements only based on light-weight scene capture with a single moving consumer-grade RGB-D camera. To this end, we present a system which is built upon a novel hybrid volumetric scene representation in terms of the combination of a voxel-based scene representation for the static contents, that not only stores the reconstructed surface geometry but also contains information about the object semantics as well as their accumulated dynamic movement over time, and a point-cloud-based representation for dynamic scene parts, where the respective separation from static parts is achieved based on semantic and instance information extracted for the input frames. With an independent yet simultaneous streaming of both static and dynamic content, where we seamlessly integrate potentially moving but currently static scene entities in the static model until they are becoming dynamic again, as well as the fusion of static and dynamic data at the remote client, our system is able to achieve VR-based live-telepresence at interactive rates. Our evaluation demonstrates the potential of our novel approach in terms of visual quality, performance, and ablation studies regarding involved design choices.

READ FULL TEXT

page 1

page 4

page 8

page 17

page 18

research
08/08/2019

Efficient 3D Reconstruction and Streaming for Group-Scale Multi-Client Live Telepresence

Sharing live telepresence experiences for teleconferencing or remote col...
research
08/08/2019

A VR System for Immersive Teleoperation and Live Exploration with a Mobile Robot

Applications like disaster management and industrial inspection often re...
research
05/09/2018

SLAMCast: Large-Scale, Real-Time 3D Reconstruction and Streaming for Immersive Multi-Client Live Telepresence

Real-time 3D scene reconstruction from RGB-D sensor data, as well as the...
research
03/24/2020

Removing Dynamic Objects for Static Scene Reconstruction using Light Fields

There is a general expectation that robots should operate in environment...
research
09/06/2023

Reviving Static Charts into Live Charts

Data charts are prevalent across various fields due to their efficacy in...
research
05/07/2019

Robust Dense Mapping for Large-Scale Dynamic Environments

We present a stereo-based dense mapping algorithm for large-scale dynami...
research
11/10/2016

Detecting Moving Regions in CrowdCam Images

We address the novel problem of detecting dynamic regions in CrowdCam im...

Please sign up or login with your details

Forgot password? Click here to reset