DeepAI
Log In Sign Up

Oversight of Unsafe Systems via Dynamic Safety Envelopes

11/22/2018
by   David Manheim, et al.
0

This paper reviews the reasons that Human-in-the-Loop is both critical for preventing widely-understood failure modes for machine learning, and not a practical solution. Following this, we review two current heuristic methods for addressing this. The first is provable safety envelopes, which are possible only when the dynamics of the system are fully known, but can be useful safety guarantees when optimal behavior is based on machine learning with poorly-understood safety characteristics. The second is the simpler circuit breaker model, which can forestall or prevent catastrophic outcomes by stopping the system, without any specific model of the system. This paper proposes using heuristic, dynamic safety envelopes, which are a plausible halfway point between these approaches that allows human oversight without some of the more difficult problems faced by Human-in-the-Loop systems. Finally, the paper concludes with how this approach can be used for governance of systems where otherwise unsafe systems are deployed.

READ FULL TEXT VIEW PDF

page 1

page 2

page 3

page 4

10/17/2022

From Function to Failure

Failure Mode Reasoning (FMR) is a method for formal analysis of system-r...
08/17/2022

Safety Assessment for Autonomous Systems' Perception Capabilities

Autonomous Systems (AS) are increasingly proposed, or used, in Safety Cr...
07/09/2020

Regulating Safety and Security in Autonomous Robotic Systems

Autonomous Robotics Systems are inherently safety-critical and have comp...
03/05/2021

Limits of Probabilistic Safety Guarantees when Considering Human Uncertainty

When autonomous robots interact with humans, such as during autonomous d...
07/20/2020

Reasoning with failures

Safety Instrumented Systems (SIS) protect major hazard facilities, e.g. ...
07/10/2020

AGI Agent Safety by Iteratively Improving the Utility Function

While it is still unclear if agents with Artificial General Intelligence...

References