Distribution Awareness for AI System Testing

by   David Berend, et al.

As Deep Learning (DL) is continuously adopted in many safety critical applications, its quality and reliability start to raise concerns. Similar to the traditional software development process, testing the DL software to uncover its defects at an early stage is an effective way to reduce risks after deployment. Although recent progress has been made in designing novel testing techniques for DL software, the distribution of generated test data is not taken into consideration. It is therefore hard to judge whether the identified errors are indeed meaningful errors to the DL application. Therefore, we propose a new OOD-guided testing technique which aims to generate new unseen test cases relevant to the underlying DL system task. Our results show that this technique is able to filter up to 55.44 and is 10.05



There are no comments yet.


page 1

page 2

page 3


Combinatorial Testing for Deep Learning Systems

Deep learning (DL) has achieved remarkable progress over the past decade...

DeepEvolution: A Search-Based Testing Approach for Deep Neural Networks

The increasing inclusion of Deep Learning (DL) models in safety-critical...

Testing Deep Learning Models: A First Comparative Study of Multiple Testing Techniques

Deep Learning (DL) has revolutionized the capabilities of vision-based s...

Towards Characterizing Adversarial Defects of Deep Learning Software from the Lens of Uncertainty

Over the past decade, deep learning (DL) has been successfully applied t...

Quality Monitoring and Assessment of Deployed Deep Learning Models for Network AIOps

Artificial Intelligence (AI) has recently attracted a lot of attention, ...

Hierarchical Distribution-Aware Testing of Deep Learning

With its growing use in safety/security-critical applications, Deep Lear...

DeepHyperion: Exploring the Feature Space of Deep Learning-Based Systems through Illumination Search

Deep Learning (DL) has been successfully applied to a wide range of appl...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.