DeepAI AI Chat
Log In Sign Up

Responsible Active Learning via Human-in-the-loop Peer Study

11/24/2022
by   Yu-Tong Cao, et al.
0

Active learning has been proposed to reduce data annotation efforts by only manually labelling representative data samples for training. Meanwhile, recent active learning applications have benefited a lot from cloud computing services with not only sufficient computational resources but also crowdsourcing frameworks that include many humans in the active learning loop. However, previous active learning methods that always require passing large-scale unlabelled data to cloud may potentially raise significant data privacy issues. To mitigate such a risk, we propose a responsible active learning method, namely Peer Study Learning (PSL), to simultaneously preserve data privacy and improve model stability. Specifically, we first introduce a human-in-the-loop teacher-student architecture to isolate unlabelled data from the task learner (teacher) on the cloud-side by maintaining an active learner (student) on the client-side. During training, the task learner instructs the light-weight active learner which then provides feedback on the active sampling criterion. To further enhance the active learner via large-scale unlabelled data, we introduce multiple peer students into the active learner which is trained by a novel learning paradigm, including the In-Class Peer Study on labelled data and the Out-of-Class Peer Study on unlabelled data. Lastly, we devise a discrepancy-based active sampling criterion, Peer Study Feedback, that exploits the variability of peer students to select the most informative data to improve model stability. Extensive experiments demonstrate the superiority of the proposed PSL over a wide range of active learning methods in both standard and sensitive protection settings.

READ FULL TEXT

page 1

page 2

page 3

page 4

05/23/2018

Distribution Aware Active Learning

Discriminative learning machines often need a large set of labeled sampl...
03/06/2018

Multi-class Active Learning: A Hybrid Informative and Representative Criterion Inspired Approach

Labeling each instance in a large dataset is extremely labor- and time- ...
04/05/2021

Stopping Criterion for Active Learning Based on Error Stability

Active learning is a framework for supervised learning to improve the pr...
02/21/2018

Active Learning with Partial Feedback

In the large-scale multiclass setting, assigning labels often consists o...
10/27/2019

Prediction stability as a criterion in active learning

Recent breakthroughs made by deep learning rely heavily on large number ...
01/02/2023

Using Active Learning Methods to Strategically Select Essays for Automated Scoring

Research on automated essay scoring has become increasing important beca...
06/07/2021

Visual Transformer for Task-aware Active Learning

Pool-based sampling in active learning (AL) represents a key framework f...