Do-AIQ: A Design-of-Experiment Approach to Quality Evaluation of AI Mislabel Detection Algorithm

08/21/2022
by   J. Lian, et al.
0

The quality of Artificial Intelligence (AI) algorithms is of significant importance for confidently adopting algorithms in various applications such as cybersecurity, healthcare, and autonomous driving. This work presents a principled framework of using a design-of-experimental approach to systematically evaluate the quality of AI algorithms, named as Do-AIQ. Specifically, we focus on investigating the quality of the AI mislabel data algorithm against data poisoning. The performance of AI algorithms is affected by hyperparameters in the algorithm and data quality, particularly, data mislabeling, class imbalance, and data types. To evaluate the quality of the AI algorithms and obtain a trustworthy assessment on the quality of the algorithms, we establish a design-of-experiment framework to construct an efficient space-filling design in a high-dimensional constraint space and develop an effective surrogate model using additive Gaussian process to enable the emulation of the quality of AI algorithms. Both theoretical and numerical studies are conducted to justify the merits of the proposed framework. The proposed framework can set an exemplar for AI algorithm to enhance the AI assurance of robustness, reproducibility, and transparency.

READ FULL TEXT
research
10/17/2022

Review of the state of the art in autonomous artificial intelligence

This article presents a new design for autonomous artificial intelligenc...
research
01/05/2023

Corrupted by Algorithms? How AI-generated and Human-written Advice Shape (Dis)honesty

Artificial Intelligence (AI) increasingly becomes an indispensable advis...
research
05/17/2021

Designer-User Communication for XAI: An epistemological approach to discuss XAI design

Artificial Intelligence is becoming part of any technology we use nowada...
research
12/11/2022

Can REF output quality scores be assigned by AI? Experimental evidence

This document describes strategies for using Artificial Intelligence (AI...
research
06/12/2023

Trustworthy Artificial Intelligence Framework for Proactive Detection and Risk Explanation of Cyber Attacks in Smart Grid

The rapid growth of distributed energy resources (DERs), such as renewab...
research
05/18/2022

AI-assisted Optimization of the ECCE Tracking System at the Electron Ion Collider

The Electron-Ion Collider (EIC) is a cutting-edge accelerator facility t...
research
09/11/2020

The AIQ Meta-Testbed: Pragmatically Bridging Academic AI Testing and Industrial Q Needs

AI solutions seem to appear in any and all application domains. As AI be...

Please sign up or login with your details

Forgot password? Click here to reset