Gesture-aware Interactive Machine Teaching with In-situ Object Annotations

08/02/2022
by   Zhongyi Zhou, et al.
0

Interactive Machine Teaching (IMT) systems allow non-experts to easily create Machine Learning (ML) models. However, existing vision-based IMT systems either ignore annotations on the objects of interest or require users to annotate in a post-hoc manner. Without the annotations on objects, the model may misinterpret the objects using unrelated features. Post-hoc annotations cause additional workload, which diminishes the usability of the overall model building process. In this paper, we develop LookHere, which integrates in-situ object annotations into vision-based IMT. LookHere exploits users' deictic gestures to segment the objects of interest in real time. This segmentation information can be additionally used for training. To achieve the reliable performance of this object segmentation, we utilize our custom dataset called HuTics, including 2040 front-facing images of deictic gestures toward various objects by 170 people. The quantitative results of our user study showed that participants were 16.3 times faster in creating a model with our system compared to a standard IMT system with a post-hoc annotation process while demonstrating comparable accuracies. Additionally, models created by our system showed a significant accuracy improvement (Δ mIoU=0.466) in segmenting the objects of interest compared to those without annotations.

READ FULL TEXT

page 1

page 4

page 5

page 7

page 9

page 10

page 13

research
07/03/2022

The Gesture Authoring Space: Authoring Customised Hand Gestures for Grasping Virtual Objects in Immersive Virtual Environments

Natural user interfaces are on the rise. Manufacturers for Augmented, Vi...
research
08/26/2021

Enhancing Model Assessment in Vision-based Interactive Machine Teaching through Real-time Saliency Map Visualization

Interactive Machine Teaching systems allow users to create customized ma...
research
04/04/2015

Preprint Extending Touch-less Interaction on Vision Based Wearable Device

This is the preprint version of our paper on IEEE Virtual Reality Confer...
research
04/20/2023

Segment Anything Model for Medical Image Analysis: an Experimental Study

Training segmentation models for medical images continues to be challeng...
research
07/28/2022

Weakly-Supervised Camouflaged Object Detection with Scribble Annotations

Existing camouflaged object detection (COD) methods rely heavily on larg...
research
05/19/2023

Where does a computer vision model make mistakes? Using interactive visualizations to find where and how CV models can improve

Creating Computer Vision (CV) models remains a complex and taxing practi...
research
03/18/2022

But that's not why: Inference adjustment by interactive prototype deselection

Despite significant advances in machine learning, decision-making of art...

Please sign up or login with your details

Forgot password? Click here to reset