System Safety and Artificial Intelligence

by   Roel I. J. Dobbe, et al.

This chapter formulates seven lessons for preventing harm in artificial intelligence (AI) systems based on insights from the field of system safety for software-based automation in safety-critical domains. New applications of AI across societal domains and public organizations and infrastructures come with new hazards, which lead to new forms of harm, both grave and pernicious. The text addresses the lack of consensus for diagnosing and eliminating new AI system hazards. For decades, the field of system safety has dealt with accidents and harm in safety-critical systems governed by varying degrees of software-based automation and decision-making. This field embraces the core assumption of systems and control that AI systems cannot be safeguarded by technical design choices on the model or algorithm alone, instead requiring an end-to-end hazard analysis and design frame that includes the context of use, impacted stakeholders and the formal and informal institutional environment in which the system operates. Safety and other values are then inherently socio-technical and emergent system properties that require design and control measures to instantiate these across the technical, social and institutional components of a system. This chapter honors system safety pioneer Nancy Leveson, by situating her core lessons for today's AI system safety challenges. For every lesson, concrete tools are offered for rethinking and reorganizing the safety management of AI systems, both in design and governance. This history tells us that effective AI safety management requires transdisciplinary approaches and a shared language that allows involvement of all levels of society.


page 1

page 2

page 3

page 4


On Safety Assessment of Artificial Intelligence

In this paper we discuss how systems with Artificial Intelligence (AI) c...

Hard Choices in Artificial Intelligence

As AI systems are integrated into high stakes social domains, researcher...

Artificial Intelligence in Concrete Materials: A Scientometric View

Artificial intelligence (AI) has emerged as a transformative and versati...

The AI Index 2022 Annual Report

Welcome to the fifth edition of the AI Index Report! The latest edition ...

A Psychopathological Approach to Safety Engineering in AI and AGI

The complexity of dynamics in AI techniques is already approaching that ...

Trust and Safety

Robotics in Australia have a long history of conforming with safety stan...

What Automated Planning can do for Business Process Management

Business Process Management (BPM) is a central element of today organiza...

Please sign up or login with your details

Forgot password? Click here to reset