
Reinforcement Learning is not a Causal problem
We use an analogy between nonisomorphic mathematical structures defined...
read it

Conditional Independences and Causal Relations implied by Sets of Equations
Realworld systems are often modelled by sets of equations with exogenou...
read it

Structure Mapping for Transferability of Causal Models
Human beings learn causal models and constantly use them to transfer kno...
read it

Causal Reasoning from Metareinforcement Learning
Discovering and exploiting the causal structure in the environment is a ...
read it

Resolving Spurious Correlations in Causal Models of Environments via Interventions
Causal models could increase interpretability, robustness to distributio...
read it

Feature relevance quantification in explainable AI: A causality problem
We discuss promising recent contributions on quantifying feature relevan...
read it

Structural causal models for macrovariables in timeseries
We consider a bivariate time series (X_t,Y_t) that is given by a simple ...
read it
Causal variables from reinforcement learning using generalized Bellman equations
Many open problems in machine learning are intrinsically related to causality, however, the use of causal analysis in machine learning is still in its early stage. Within a general reinforcement learning setting, we consider the problem of building a general reinforcement learning agent which uses experience to construct a causal graph of the environment, and use this graph to inform its policy. Our approach has three characteristics: First, we learn a simple, coarsegrained causal graph, in which the variables reflect states at many time instances, and the interventions happen at the level of policies, rather than individual actions. Secondly, we use mediation analysis to obtain an optimization target. By minimizing this target, we define the causal variables. Thirdly, our approach relies on estimating conditional expectations rather the familiar expected return from reinforcement learning, and we therefore apply a generalization of Bellman's equations. We show the method can learn a plausible causal graph in a gridworld environment, and the agent obtains an improvement in performance when using the causally informed policy. To our knowledge, this is the first attempt to apply causal analysis in a reinforcement learning setting without strict restrictions on the number of states. We have observed that mediation analysis provides a promising avenue for transforming the problem of causal acquisition into one of costfunction minimization, but importantly one which involves estimating conditional expectations. This is a new challenge, and we think that causal reinforcement learning will involve development methods suited for online estimation of such conditional expectations. Finally, a benefit of our approach is the use of very simple causal models, which are arguably a more natural model of human causal understanding.
READ FULL TEXT
Comments
There are no comments yet.