Adaptive Rule Discovery for Labeling Text Data

by   Sainyam Galhotra, et al.

Creating and collecting labeled data is one of the major bottlenecks in machine learning pipelines and the emergence of automated feature generation techniques such as deep learning, which typically requires a lot of training data, has further exacerbated the problem. While weak-supervision techniques have circumvented this bottleneck, existing frameworks either require users to write a set of diverse, high-quality rules to label data (e.g., Snorkel), or require a labeled subset of the data to automatically mine rules (e.g., Snuba). The process of manually writing rules can be tedious and time consuming. At the same time, creating a labeled subset of the data can be costly and even infeasible in imbalanced settings. This is due to the fact that a random sample in imbalanced settings often contains only a few positive instances. To address these shortcomings, we present Darwin, an interactive system designed to alleviate the task of writing rules for labeling text data in weakly-supervised settings. Given an initial labeling rule, Darwin automatically generates a set of candidate rules for the labeling task at hand, and utilizes the annotator's feedback to adapt the candidate rules. We describe how Darwin is scalable and versatile. It can operate over large text corpora (i.e., more than 1 million sentences) and supports a wide range of labeling functions (i.e., any function that can be specified using a context free grammar). Finally, we demonstrate with a suite of experiments over five real-world datasets that Darwin enables annotators to generate weakly-supervised labels efficiently and with a small cost. In fact, our experiments show that rules discovered by Darwin on average identify 40 positive instances compared to Snuba even when it is provided with 1000 labeled instances.


page 1

page 2

page 3

page 4


PRBoost: Prompt-Based Rule Discovery and Boosting for Interactive Weakly-Supervised Learning

Weakly-supervised learning (WSL) has shown promising results in addressi...

Trove: Ontology-driven weak supervision for medical entity classification

Motivation: Recognizing named entities (NER) and their associated attrib...

Adaptive Multi-view Rule Discovery for Weakly-Supervised Compatible Products Prediction

On e-commerce platforms, predicting if two products are compatible with ...

Towards Automated Real-time Evaluation in Text-based Counseling

Automated real-time evaluation of counselor-client interaction is import...

Leveraging User Engagement Signals For Entity Labeling in a Virtual Assistant

Personal assistant AI systems such as Siri, Cortana, and Alexa have beco...

Demonstration of Panda: A Weakly Supervised Entity Matching System

Entity matching (EM) refers to the problem of identifying tuple pairs in...

Using system context information to complement weakly labeled data

Real-world datasets collected with sensor networks often contain incompl...

Please sign up or login with your details

Forgot password? Click here to reset