DeepAI AI Chat
Log In Sign Up

Measuring Discrimination to Boost Comparative Testing for Multiple Deep Learning Models

by   Linghan Meng, et al.

The boom of DL technology leads to massive DL models built and shared, which facilitates the acquisition and reuse of DL models. For a given task, we encounter multiple DL models available with the same functionality, which are considered as candidates to achieve this task. Testers are expected to compare multiple DL models and select the more suitable ones w.r.t. the whole testing context. Due to the limitation of labeling effort, testers aim to select an efficient subset of samples to make an as precise rank estimation as possible for these models. To tackle this problem, we propose Sample Discrimination based Selection (SDS) to select efficient samples that could discriminate multiple models, i.e., the prediction behaviors (right/wrong) of these samples would be helpful to indicate the trend of model performance. To evaluate SDS, we conduct an extensive empirical study with three widely-used image datasets and 80 real world DL models. The experimental results show that, compared with state-of-the-art baseline methods, SDS is an effective and efficient sample selection method to rank multiple DL models.


page 1

page 2

page 3

page 4


An Empirical Study of Challenges in Converting Deep Learning Models

There is an increase in deploying Deep Learning (DL)-based software syst...

DLFuzz: Differential Fuzzing Testing of Deep Learning Systems

Deep learning (DL) systems are increasingly applied to safety-critical d...

A Comparative Measurement Study of Deep Learning as a Service Framework

Big data powered Deep Learning (DL) and its applications have blossomed ...

Testing Deep Learning Models: A First Comparative Study of Multiple Testing Techniques

Deep Learning (DL) has revolutionized the capabilities of vision-based s...

Deep Generative Multimedia Children's Literature

The popularity in Deep Learning (DL) based creative endeavours continues...

Deep Interactive Denoiser (DID) for X-Ray Computed Tomography

Low dose computed tomography (LDCT) is desirable for both diagnostic ima...