AirPen: A Touchless Fingertip Based Gestural Interface for Smartphones and Head-Mounted Devices

by   Varun Jain, et al.

Hand gestures are an intuitive, socially acceptable, and a non-intrusive interaction modality in Mixed Reality (MR) and smartphone based applications. Unlike speech interfaces, they tend to perform well even in shared and public spaces. Hand gestures can also be used to interact with smartphones in situations where the user's ability to physically touch the device is impaired. However, accurate gesture recognition can be achieved through state-of-the-art deep learning models or with the use of expensive sensors. Despite the robustness of these deep learning models, they are computationally heavy and memory hungry, and obtaining real-time performance on-device without additional hardware is still a challenge. To address this, we propose AirPen: an analogue to pen on paper, but in air, for in-air writing and gestural commands that works seamlessly in First and Second Person View. The models are trained on a GPU machine and ported on an Android smartphone. AirPen comprises of three deep learning models that work in tandem: MobileNetV2 for hand localisation, our custom fingertip regression architecture followed by a Bi-LSTM model for gesture classification. The overall framework works in real-time on mobile devices and achieves a classification accuracy of 80 of only 0.12 s.


GestARLite: An On-Device Pointing Finger Based Gestural Interface for Smartphones and Video See-Through Head-Mounts

Hand gestures form an intuitive means of interaction in Mixed Reality (M...

Deep Learning for Enhanced Scratch Input

The vibrations generated from scratching and tapping on surfaces can be ...

WristSketcher: Creating Dynamic Sketches in AR with a Sensing Wristband

Restricted by the limited interaction area of native AR glasses (e.g., t...

Fingertip Detection and Tracking for Recognition of Air-Writing in Videos

Air-writing is the process of writing characters or words in free space ...

Guidelines and Benchmarks for Deployment of Deep Learning Models on Smartphones as Real-Time Apps

Deep learning solutions are being increasingly used in mobile applicatio...

Real-time on-device nod and shake recognition

We discuss methods for teaching systems to identify gestures such as hea...

Don't Touch Me! A Comparison of Usability on Touch and Non-Touch Inputs

Public touchscreens are filthy and, regardless of how often they are cle...

Please sign up or login with your details

Forgot password? Click here to reset