Understanding Agent Incentives using Causal Influence Diagrams, Part I: Single Action Settings

02/26/2019
by   Tom Everitt, et al.
0

Agents are systems that optimize an objective function in an environment. Together, the goal and the environment induce secondary objectives, incentives. Modeling the agent-environment interaction in graphical models called influence diagrams, we can answer two fundamental questions about an agent's incentives directly from the graph: (1) which nodes is the agent incentivized to observe, and (2) which nodes is the agent incentivized to influence? The answers tell us which information and influence points need extra protection. For example, we may want a classifier for job applications to not use the ethnicity of the candidate, and a reinforcement learning agent not to take direct control of its reward mechanism. Different algorithms and training paradigms can lead to different influence diagrams, so our method can be used to identify algorithms with problematic incentives and help in designing algorithms with better incentives.

READ FULL TEXT
research
08/13/2019

Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective

Can an arbitrarily intelligent reinforcement learning agent be kept unde...
research
02/02/2021

Agent Incentives: A Causal Perspective

We present a framework for analysing agent incentives using causal influ...
research
02/23/2022

A Complete Criterion for Value of Information in Soluble Influence Diagrams

Influence diagrams have recently been used to analyse the safety and fai...
research
02/06/2013

Myopic Value of Information in Influence Diagrams

We present a method for calculation of myopic value of information in in...
research
06/20/2019

Modeling AGI Safety Frameworks with Causal Influence Diagrams

Proposals for safe AGI systems are typically made at the level of framew...
research
11/30/2015

Influence diagrams for the optimization of a vehicle speed profile

Influence diagrams are decision theoretic extensions of Bayesian network...
research
02/08/2013

Complexity distribution of agent policies

We analyse the complexity of environments according to the policies that...

Please sign up or login with your details

Forgot password? Click here to reset