Grammar Based Directed Testing of Machine Learning Systems

02/26/2019
by   Sakshi Udeshi, et al.
0

The massive progress of machine learning has seen its application over a variety of domains in the past decade. But how do we develop a systematic, scalable and modular strategy to validate machine-learning systems? We present, to the best of our knowledge, the first approach, which provides a systematic test framework for machine-learning systems that accepts grammar-based inputs. Our OGMA approach automatically discovers erroneous behaviours in classifiers and leverages these erroneous behaviours to improve the respective models. OGMA leverages inherent robustness properties present in any well trained machine-learning model to direct test generation and thus, implementing a scalable test generation methodology. To evaluate our OGMA approach, we have tested it on three real world natural language processing (NLP) classifiers. We have found thousands of erroneous behaviours in these systems. We also compare OGMA with a random test generation approach and observe that OGMA is more effective than such random test generation by up to 489

READ FULL TEXT
research
12/11/2019

Callisto: Entropy based test generation and data quality assessment for Machine Learning Systems

Machine Learning (ML) has seen massive progress in the last decade and a...
research
03/22/2021

Fixes That Fail: Self-Defeating Improvements in Machine-Learning Systems

Machine-learning systems such as self-driving cars or virtual assistants...
research
07/02/2018

Automated Directed Fairness Testing

Fairness is a critical trait in decision making. As machine-learning mod...
research
04/27/2023

Learning to Extrapolate: A Transductive Approach

Machine learning systems, especially with overparameterized deep neural ...
research
05/03/2021

MLCheck- Property-Driven Testing of Machine Learning Models

In recent years, we observe an increasing amount of software with machin...
research
07/29/2021

On the combined effect of class imbalance and concept complexity in deep learning

Structural concept complexity, class overlap, and data scarcity are some...
research
12/05/2022

Continual learning on deployment pipelines for Machine Learning Systems

Following the development of digitization, a growing number of large Ori...

Please sign up or login with your details

Forgot password? Click here to reset