DeepLight: Robust Unobtrusive Real-time Screen-Camera Communication for Real-World Displays

by   Vu Tran, et al.

The paper introduces a novel, holistic approach for robust Screen-Camera Communication (SCC), where video content on a screen is visually encoded in a human-imperceptible fashion and decoded by a camera capturing images of such screen content. We first show that state-of-the-art SCC techniques have two key limitations for in-the-wild deployment: (a) the decoding accuracy drops rapidly under even modest screen extraction errors from the captured images, and (b) they generate perceptible flickers on common refresh rate screens even with minimal modulation of pixel intensity. To overcome these challenges, we introduce DeepLight, a system that incorporates machine learning (ML) models in the decoding pipeline to achieve humanly-imperceptible, moderately high SCC rates under diverse real-world conditions. Deep-Light's key innovation is the design of a Deep Neural Network (DNN) based decoder that collectively decodes all the bits spatially encoded in a display frame, without attempting to precisely isolate the pixels associated with each encoded bit. In addition, DeepLight supports imperceptible encoding by selectively modulating the intensity of only the Blue channel, and provides reasonably accurate screen extraction (IoU values >= 83 pipelines. We show that a fully functional DeepLight system is able to robustly achieve high decoding accuracy (frame error rate < 0.2) and moderately-high data goodput (>=0.95Kbps) using a human-held smartphone camera, even over larger screen-camera distances (approx =2m).



There are no comments yet.


page 1

page 2

page 3

page 5

page 6

page 7

page 14


Design and Implementation of a Novel Compatible Encoding Scheme in the Time Domain for Image Sensor Communication

This paper presents a modulation scheme in the time domain based on On-O...

Bringing a Blurry Frame Alive at High Frame-Rate with an Event Camera

Event-based cameras can measure intensity changes (called ` events') wit...

A Feature based Approach for Video Compression

It is a high cost problem for panoramic image stitching via image matchi...

StegaStamp: Invisible Hyperlinks in Physical Photographs

Imagine a world in which each photo, printed or digitally displayed, hid...

A Novel Deep Neural Network Based Approach for Sparse Code Multiple Access

Sparse code multiple access (SCMA) has been one of non-orthogonal multip...

Light Propagation Prediction through Multimode Optical Fibers with a Deep Neural Network

This work demonstrates a computational method for predicting the light p...

DSAC - Differentiable RANSAC for Camera Localization

RANSAC is an important algorithm in robust optimization and a central bu...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.