Estimating the Brittleness of AI: Safety Integrity Levels and the Need for Testing Out-Of-Distribution Performance

by   Andrew J. Lohn, et al.

Test, Evaluation, Verification, and Validation (TEVV) for Artificial Intelligence (AI) is a challenge that threatens to limit the economic and societal rewards that AI researchers have devoted themselves to producing. A central task of TEVV for AI is estimating brittleness, where brittleness implies that the system functions well within some bounds and poorly outside of those bounds. This paper argues that neither of those criteria are certain of Deep Neural Networks. First, highly touted AI successes (eg. image classification and speech recognition) are orders of magnitude more failure-prone than are typically certified in critical systems even within design bounds (perfectly in-distribution sampling). Second, performance falls off only gradually as inputs become further Out-Of-Distribution (OOD). Enhanced emphasis is needed on designing systems that are resilient despite failure-prone AI components as well as on evaluating and improving OOD performance in order to get AI to where it can clear the challenging hurdles of TEVV and certification.



There are no comments yet.


page 10


Certifiable Artificial Intelligence Through Data Fusion

This paper reviews and proposes concerns in adopting, fielding, and main...

Challenge AI Mind: A Crowd System for Proactive AI Testing

Artificial Intelligence (AI) has burrowed into our lives in various aspe...

Self-explaining AI as an alternative to interpretable AI

The ability to explain decisions made by AI systems is highly sought aft...

Lessons Learned from Designing an AI-Enabled Diagnosis Tool for Pathologists

Despite the promises of data-driven artificial intelligence (AI), little...

Self-explainability as an alternative to interpretability for judging the trustworthiness of artificial intelligences

The ability to explain decisions made by AI systems is highly sought aft...

Recommendations on test datasets for evaluating AI solutions in pathology

Artificial intelligence (AI) solutions that automatically extract inform...

Integrating Testing and Operation-related Quantitative Evidences in Assurance Cases to Argue Safety of Data-Driven AI/ML Components

In the future, AI will increasingly find its way into systems that can p...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.