Learning Causal Models of Autonomous Agents using Interventions

by   Pulkit Verma, et al.
Arizona State University

One of the several obstacles in the widespread use of AI systems is the lack of requirements of interpretability that can enable a layperson to ensure the safe and reliable behavior of such systems. We extend the analysis of an agent assessment module that lets an AI system execute high-level instruction sequences in simulators and answer the user queries about its execution of sequences of actions. We show that such a primitive query-response capability is sufficient to efficiently derive a user-interpretable causal model of the system in stationary, fully observable, and deterministic settings. We also introduce dynamic causal decision networks (DCDNs) that capture the causal structure of STRIPS-like domains. A comparative analysis of different classes of queries is also presented in terms of the computational requirements needed to answer them and the efforts required to evaluate their responses to learn the correct model.



There are no comments yet.


page 1

page 2

page 3

page 4


Differential Assessment of Black-Box AI Agents

Much of the research on learning symbolic models of AI agents focuses on...

Learning User-Interpretable Descriptions of Black-Box AI System Capabilities

Several approaches have been developed to answer specific questions that...

Resolving Spurious Correlations in Causal Models of Environments via Interventions

Causal models could increase interpretability, robustness to distributio...

Causal Analysis of Agent Behavior for AI Safety

As machine learning systems become more powerful they also become increa...

Causal Confusion in Imitation Learning

Behavioral cloning reduces policy learning to supervised learning by tra...

Inducing Causal Structure for Interpretable Neural Networks

In many areas, we have well-founded insights about causal structure that...

Learning of Behavior Trees for Autonomous Agents

Definition of an accurate system model for Automated Planner (AP) is oft...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

The growing deployment of AI systems presents a pervasive problem of ensuring the safety and reliability of these systems. The problem is exacerbated because most of these AI systems are neither designed by their users nor are their users skilled enough to understand their internal working, i.e., the AI system is a black-box for them. We also have systems that can adapt to user preferences, thereby invalidating any design stage knowledge of their internal model. Additionally, these systems have diverse system designs and implementations. This makes it difficult to evaluate such arbitrary AI systems using a common independent metric.

In recent work, we developed a non-intrusive system that allow for assessment of arbitrary AI systems independent of their design and implementation. The Agent Assessment Module (AAM) [26] is such a system which uses active query answering to learn the action model of black-box autonomous agents. It poses minimum requirements on the agent – to have a rudimentary query-response capability – to learn its model using interventional queries. This is needed because we do not intend these modules to hinder the development of AI systems by imposing additional complex requirements or constraints on them. This module learns the generalized dynamical causal model of the agents capturing how the agent operates and interacts with its environment; and under what conditions it executes certain actions and what happens after it executes them.

Causal models are needed to capture the behavior of AI systems as they help in understanding the relationships among underlying causal mechanisms, and they also make it easy to make predictions about the behavior of a system. E.g., consider a delivery agent which delivers crates from one location to another. If the agent has only encountered blue crates, an observational data-based learner might learn that the crate has to be blue for it to be delivered by the robot. On the other hand, a causal model will be able to identify that the crate color does not affect the robot’s ability to deliver it.

The causal model learned by AAM is user-interpretable as the model is learned in the vocabulary that the user provides and understands. Such a module would also help make the AI systems compliant with Level II assistive AI – systems that make it easy for operators to learn how to use them safely [24].

This paper presents a formal analysis of the AAM, presents different types of query classes, and analyzes the query process and the models learned by AAM. It also uses the theory of causal networks to show that we can define the causal properties of the models learned by AAM – in relational STRIPS-like language [5, 19, 6]). We call this network Dynamic Causal Decision Network (DCDN), and show that the models learned by AAM are causal owing to the interventional nature of the queries used by it.

2 Background

2.1 Agent Assessment Module

A high-level view of the agent assessment module is shown in Fig. 1 where AAM connects the agent with a simulator and provides a sequence of instructions, called a plan, as a query. executes the plan in the simulator and the assessment module uses the simulated outcome as the response to the query. At the end of the querying process, AAM returns a user-interpretable model of the agent.

An advantage of this approach is that the AI system need not know the user vocabulary or the modeling language and it can have any arbitrary internal implementation. Additionally, by using such a method, we can infer models of AI systems that don’t have such in-built capability to infer and/or communicate their model. Also, the user need not even know what questions are being asked as long as (s)he gets the correct model in terms of her/his vocabulary. It is assumed that the user knows the names of the agent’s primitive actions. Even when they are not known, without loss of generality, the first step can be a listing of the names of the agent’s actions.

Note that we can have modules like AAM with varying level of capabilities of evaluating the query responses. This results in a trade-off between the evaluation capabilities of the assessment modules and the computational requirements of the AI systems to support such modules. E.g., if we have an assessment module with strong evaluation capabilities, the AI systems can support them easily, whereas we might have to put more burden on AI systems to support modules with weaker evaluation systems. To test and analyze this, we introduce a new class of queries in this work, and study the more general properties of Agent Interrogation Algorithm (AIA) used by AAM. We also present a more insightful analysis of the complexity of the queries and the computational requirements on the agents to answer these queries.

Figure 1: The agent-assessment module uses its user’s preferred vocabulary, queries the AI system, and delivers a user-interpretable causal model of the AI system’s capabilities [26]


2.2 Causal Models

In this work, we focus on the properties of the models learned by AIA, and show that the models learned by AIA are causal. But prior to that, we must define what it means for a model to be causal. Multiple attempts have been made to define causal models [8, 9, 10]. We use the definition of causal models based on halpern_2015_modification [halpern_2015_modification].

Definition 1.

A causal model is defined as a 4-tuple where is a set of exogenous variables (representing factors outside the model’s control), is a set of endogenous variables (whose values are directly or indirectly derived from the exogenous variables), is a function that associates with every variable a nonempty set of possible values for , and is a function that associates with each endogenous variable a structural function denoted as such that maps to .

Note that the values of exogenous variables are not determined by the model, and a setting of values of exogenous variables is termed as a context by Halpern16_causality [Halpern16_causality]. This helps in defining a causal setting as:

Definition 2.

A causal setting is a pair consisting of a causal model and context .

A causal formula is true or false in a causal model, given a context. Hence, if the causal formula is true in the causal setting .

Every causal model can be associated with a directed graph, , in which each variable is represented as a vertex and the causal relationships between the variables are represented as directed edges between members of and [20]. We use the term causal networks when referring to these graphs to avoid confusion with the notion of causal graphs used in the planning literature [12].

To perform an analysis with interventions, we use the concept of do-calculus introduced in pearl_1995_do_calculus [pearl_1995_do_calculus]. To perform interventions on a set of variables , do-calculus assigns values to , and evaluates the effect using the causal model . This is termed as do() action. To define this concept formally, we first define submodels [20].

Definition 3.

Let be a causal model, a set of variables in , and a particular realization of . A submodel of is the causal model where is obtained from by setting (for each ) instead of the corresponding , and setting for each .

We now define what it means to intervene using the action do.

Definition 4.

Let be a causal model, a set of variables in , and a particular realization of . The effect of action do on is given by the submodel .

In general, there can be uncertainty about the effects of these interventions, leading to probabilistic causal networks, but in this work we assume that interventions do not lead to uncertain effects.

The interventions described above assigns values to a set of variables, without affecting any another variable. Such interventions are termed as hard (independent) interventions. It is not always possible to perform such interventions and in some cases other variable(s) also change without affecting the causal structure [18]. Such interventions are termed as soft (dependent) interventions.

We can also derive the structure of causal networks using interventions in the real world, as interventions allow us to find if a variable depends on another variable . We use Halpern16_causality [Halpern16_causality]’s definition of dependence and actual cause.

Definition 5.

A variable depends on variable if there is some setting of all the variables in such that varying the value of in that setting results in a variation in the value of .

Definition 6.

Given a signature , a primitive event is a formula of the form , for and . A causal formula is , where is a Boolean combination of primitive events, are distinct variables in , and . holds if would set to , for .

Definition 7.

Let be the subset of exogenous variables , and be a boolean causal formula expressible using variables in . is an actual cause of in the causal setting , i.e., , if the following conditions hold:

  1. and .

  2. There is a set of variables in and a setting of the variables in such that if , then .

  3. is minimal; there is no strict subset of such that satisfies conditions AC1 and AC2, where is the restriction of to the variables in .

AC1 mentions that unless both and occur at the same time, cannot be caused by . AC2111Halpern16_causality [Halpern16_causality] termed it as AC2(a) mentions that there exists a such that if we change a subset of variables from some initial value to , keeping the value of other variables fixed to , will also change. AC3 is a minimality condition which ensures that there are no spurious elements in .

The following definition specifies soundness and completeness with respect to the actual causes entailed by a pair of causal models.

Definition 8.

Let and

be the vectors of exogenous and endogenous variables, respectively; and

be the set of all boolean causal formulas expressible over variables in .

A causal model is complete with respect to another causal model if for all possible settings of exogenous variables, all the causal relationships that are implied by the model are a superset of the set of causal relationships implied by the model , i.e., s.t. .

A causal model is sound with respect to another causal model if for all possible settings of exogenous variables, all the causal relationships that are implied by the model are a subset of the set of causal relationships implied by the model , i.e., s.t. .

2.3 Query Complexity

In this paper, we provide an extended analysis of the complexity of the queries that AIA uses to learn the agent’s model. We use the complexity analysis of relational queries by vardi82_complexity [vardi82_complexity, vardi_95_on] to find the membership classes for data, expression, and combined complexity of AIA’s queries.

vardi82_complexity [vardi82_complexity] introduced three kinds of complexities for relational queries. In the notion of query complexity, a specific query is fixed in the language, then data complexity – given as function of size of databases – is found by applying this query to arbitrary databases. In the second notion of query complexity, a specific database is fixed, then the expression complexity – given as function of length of expressions – is found by studying the complexity of applying queries represented by arbitrary expressions in the language. Finally, combined complexity – given as a function of combined size of the expressions and the database – is found by applying arbitrary queries in the language to arbitrary databases.

These notions can be defined formally as follows vardi_95_on [vardi_95_on]:

Definition 9.

The complexity of a query is measured as the complexity of deciding if , where is a tuple, is a query, and is a database.

  • The data complexity of a language is the complexity of the sets for queries in , where is the answer set of a query given as:

  • The expression complexity of a language is the complexity of the sets , where is the answer set of a database with respect to a language given as:

  • The combined complexity of a language is the complexity of the set , where is the answer set of a language given as:

vardi82_complexity [vardi82_complexity, vardi_95_on] gave standard complexity classes for queries written in specific logical languages. We show the membership of our queries in these classes based on the logical languages we write the queries in.

3 Formal Framework

The agent assessment module assumes that the user needs to estimate the agent’s model as a STRIPS-like planning model represented as a pair

, where is a finite set of predicates with arities ; is a finite set of parameterized actions (operators). Each action is represented as a tuple , where is the action header consisting of action name and action parameters, represents the set of predicate atoms that must be true in a state where can be applied, is the set of positive or negative predicate atoms that will change to true or false respectively as a result of execution of the action . Each predicate can be instantiated using the parameters of an action, where the number of parameters are bounded by the maximum arity of the action. E.g., consider the action and predicate in the IPC Logistics domain. This predicate can be instantiated using action parameters , , and as , , , , , , , , and . We represent the set of all such possible predicates instantiated with action parameters as .

AAM uses the following information as input. It receives its instruction set in the form of for each from the agent. AAM also receives a predicate vocabulary from the user with functional definitions of each predicate. This gives AAM sufficient information to perform a dialog with about the outcomes of hypothetical action sequences.

We define the overall problem of agent interrogation as follows. Given a class of queries and an agent with an unknown model which can answer these queries, determine the model of the agent. More precisely, an agent interrogation task is defined as a tuple , where is the true model (unknown to AAM) of the agent being interrogated, is the class of queries that can be posed to the agent by AAM, and and are the sets of predicates and action headers that AAM uses based on inputs from and . The objective of the agent interrogation task is to derive the agent model using and . Let be the set of possible answers to queries. Thus, strings denote the information received by AAM at any point in the query process. Query policies for the agent interrogation task are functions that map sequences of answers to the next query that the interrogator should ask. The process stops with the Stop query. In other words, for all answers , all valid query policies map all sequences to Stop whenever is mapped to Stop. This policy is computed and executed online.

Running Example    Consider we have a driving robot having a single action drive (?t ?s ?d), parameterized by the truck it drives, source location, and destination location. Assume that all the locations are connected, hence the robot can drive between any two locations. The predicates available are , representing the location of a truck; and , representing the color of the source location. Instantiating and with parameters of the action drive gives four instantiated predicates , , , and .

4 Learning Causal Models

The classic causal model framework used in Def. 1 lacks the temporal elements and decision nodes needed to express the causality in planning domains.

To express actions, we use the decision nodes similar to Dynamic Decision Networks [16]. To express the temporal behavior of planning models, we use the notion of Dynamic Causal Models [20] and Dynamic Causal Networks (DCNs) [3]. These are similar to causal models and causal networks respectively, with the only difference that the variables in these are time-indexed, allowing for analysis of temporal causal relations between the variables. We also introduce additional boolean variables to capture the executability of the actions. The resulting causal model is termed as a causal action model, and we express such models using a Dynamic Causal Decision Network (DCDN).

A general structure of a dynamic causal decision network is shown in Fig. 2. Here and are states at time and respectively, is a decision node representing the decision to execute action at time , and executability variable represents if action is executable at time . All the decision variables and the executability variables , where , in a domain are endogenous. Decision variables are endogenous because we can perform interventions on them as needed.

4.1 Types of Interventions

To learn the causal action model corresponding to each domain, two kinds of interventions are needed. The first type of interventions, termed , correspond to searching for the initial state in AIA. AIA searches for the state where it can execute an action, hence if the state variables are completely independent of each other, these interventions are hard, whereas for the cases where some of the variables are dependent the interventions are soft for those variables. Such interventions lead to learning the preconditions of an action correctly.

The second type of interventions, termed , are on the decision nodes, where the values of the decision variables are set to true according to the input plan. For each action in the plan , the corresponding decision node with label is set to true. Of course, during the intervention process, the structure of the true DCDN is not known. Such interventions lead to learning the effects of an action accurately. As mentioned earlier, if an action is executed in a state which does not satisfy its preconditions, the variable will be false at that time instant, and the resulting state will be same as state , signifying a failure to execute the action. Note that the state nodes and in Fig. 2 are the combined representation of multiple predicates.

Figure 2: An example of a Dynamic Causal Decision Network (DCDN). and are states at time and respectively, is a decision node representing the decision to execute action at time , and represents if action is executable at time .

We now show that the model(s) learned by AIA are causal models.

Lemma 1.

Given an agent with a ground truth model (unknown to the agent interrogation algorithm AIA), the action model learned by AIA is a causal model consistent with Def. 1.

Proof (Sketch).

We show a mapping between the components of the causal models used in Def. 1 and the planning models described in Sec. 3. The exogenous variables maps to the static predicates in the domain, i.e., the ones that do not appear in the effect of any action; maps to the non-static predicates; maps each predicate to if the predicate is true in a state, or when the predicate is false in a state; calculates the value of each variable depending on the other variables that cause it. This is captured by the values of state predicates and executability variables being changed due to other state variables and decision variables.

The causal relationships in the model learned by AIA also satisfy the three conditions – AC1, AC2, and AC3 – mentioned in the definition for the actual cause (Def. 7). By Thm. 1 in verma2021asking [verma2021asking], AIA returns correct models, i.e., contains exactly the same palm tuples as .

This also means that AC1 is satisfied due to correctness of – a predicate is a cause of only when is precondition of action ; and – a predicate is a caused by and only when is an effect of action . AC2 is satisfied because if any precondition of an action is not satisfied, it will not execute (defining the relationship “state variables ”); or if any action doesn’t execute, it won’t affect the predicates in its effects (defining the relationship “ state variables”). Finally, AC3 is satisfied, as neither spurious preconditions are learned by AIA, nor incorrect effects are learned. ∎

We now formally show that the causal model(s) learned by AIA is(are) sound and complete.

Theorem 1.

Given an agent with a ground truth model (unknown to the agent interrogation algorithm AIA), the action model learned by AIA is sound and complete with respect to .

Proof (Sketch).

We first show that is sound with respect to . Assume that some is an actual cause of according to in the setting , i.e., . Now by Thm. 1 in verma2021asking [verma2021asking], contains exactly the same palm tuples as . Hence any palm tuple that is present in will also be present in , implying that under the same setting according to is an actual cause of .

Now lets assume that some is an actual cause of according to in the setting , i.e., . Now by Thm. 1 in verma2021asking [verma2021asking], contains exactly the same palm tuples as . Hence any palm tuple that is present in will also be present in , implying that under the same setting according to is an actual cause of . Hence the action model learned by the agent interrogation algorithm are sound and complete with respect to the model . ∎

4.2 Comparison with Observational Data based Learners

We compare the properties of models learned by AIA with those of approaches that learn the models from observational data only. For the methods that learn models in STRIPS-like the learned models can be classified as causal, but it is not necessary that they are sound with respect to the ground truth model

of the agent . E.g., in case of the robot driver discussed earlier, these methods can learn a model where the precondition of the action drive is if all the observation traces that are provided to it as input had as true. This can happen if all the source locations are painted blue. To avoid such cases, some of these methods run a pre-processing or a post-processing step that removes all static predicates from the preconditions. However, if there is a paint action in the domain that changes the color of all source locations, then these ad-hoc solutions will not be able to handle that. Hence, these techniques may end up learning spurious preconditions as they do not have a way to distinguish between correlation and causations.

On the other hand, it is also not necessary that the models learned by approaches using only observational data are complete with respect to the ground truth model of the agent . This is because they may miss to capture some causal relationships if the observations do not include all the possible transitions, or contains only the successful actions. E.g., if we have additional predicates , and in the domain, and all the observed transitions are for the transitions within same city, then the model will not be able to learn if the source city and destination city have to be same for driving a truck between them.

Hence, the models learned using only observational data are not necessarily sound or complete, as they can learn causal relationships that are not part of set of actual causal relationships, and can also miss some of the causal relationships that are not part of set of actual causal relationships. Pearl19_Seven [Pearl19_Seven] also points out that it is not possible to learn causal models from observational data only.

4.3 Types of Queries

Plan Outcome Queries    verma2021asking [verma2021asking] introduced plan outcome queries , which are parameterized by a state and a plan . Let be the set of predicates instantiated with objects in an environment. queries ask the length of the longest prefix of the plan that it can execute successfully when starting in the state as well as the final state that this execution leads to. E.g., “Given that the truck is at location , what would happen if you executed the plan , , ?”

A response to such queries can be of the form “I can execute the plan till step and at the end of it truck is at location ”. Formally, the response for plan outcome queries is a tuple , where is the number of steps for which the plan could be executed, and is the final state after executing steps of the plan. If the plan cannot be executed fully according to the agent model then , otherwise . The final state is such that , i.e., starting from a state , successfully executed first steps of the plan . Thus, , where is the set of all the models that can be generated using the predicates and actions , and is the set of natural numbers.

Action Precondition Queries    In this work, we introduce a new class of queries called action precondition queries . These queries, similar to plan outcome queries, are parameterized by and , but have a different response type.

A response to the action precondition queries can be either of the form “I can execute the plan completely and at the end of it, truck is at location ” when the plan is successfully executed, or of the form “I can execute the plan till step and the action failed because precondition was not satisfied” when the plan is not fully executed. To make the responses consistent in all cases, we introduce a dummy action whose precondition is never satisfied. Hence, the responses are always of the form, “I can execute the plan till step and the action failed because precondition was not satisfied”. If is and , then we know that the original plan was executed successfully by the agent. Formally, the response for action precondition queries is a tuple , where is the number of steps for which the plan could be executed, and is the set of preconditions of the failed action . If the plan cannot be executed fully according to the agent model then , otherwise . Also, , where is the set of all the models that can be generated using the predicates and actions , and is the set of natural numbers.

5 Complexity Analysis

Theoretically, the asymptotic complexity of AIA (with plan outcome queries) is , but it does not take into account how much computation is needed to answer the queries, or to evaluate their responses. This complexity just shows the amount of computation needed in the worst case to derive the agent model by AIA. Here, we present a more detailed analysis of the complexity of AIA’s queries using the results of relational query complexity by vardi82_complexity [vardi82_complexity].

To analyze ’s complexity, let us assume that the agent has stored the possible transitions it can make (in propositional form) using the relations , where , , ; and , where , , , and , where is the maximum possible length of a plan in the queries. can be an arbitrarily large number, and it does not matter as long as it is finite. Here, and are sets of grounded states and actions respectively. is if the action was executed successfully, and is if the action failed. is when none of the previous actions had . This stops an action to change a state if any of the previous actions failed, thereby preserving the state that resulted from a failed action. Whenever or , and signifying that applying an action where it is not applicable does not change the state.

Assuming the length of the query plan, , we can write a query in first order logic, equivalent to the plan outcome query as

The output of the query contains the free variables and . Such first order (FO) queries have the expression complexity and the combined complexity in PSPACE [25]. The data complexity class of FO queries is  [15].

The following results use the analysis in vardi_95_on [vardi_95_on]. The query analysis given above depends on how succinctly we can express the queries. In the FO query shown above, we have a lot of spurious quantified variables. We can reduce its complexity by using bounded-variable queries. Normally, queries in a language assume an inifinite supply of individual variables. A bounded-variable version of the language is one which can be obtained by restricting the individual variables to be among , for . Using this, we can reduce the quantified variables in query shown earlier, and rewrite it more succinctly as an query by storing temporary query outputs.

We then write subsequent queries corresponding to each step of the query plan as

Here varies from to , and the value of is 6 because of 6 quantified variables – and . This reduces the expression and combined complexity of these queries to ALOGTIME and PTIME respectively. Note that these are the membership classes as it might be possible to write the queries more succinctly.

For a detailed analysis of ’s complexity, let us assume that the agent stores the possible transitions it can make (in propositional form) using the relations , where , , ; and , where , . contains if a grounded predicate is in state .

Now, we can write a query in first order logic, equivalent to the action precondition query as:

This formulation is equivalent to the queries with . This means that the data, expression and combined complexity of these queries are in complexity classes AC, ALOGTIME, and PTIME respectively.

The results for complexity classes of the queries presented above holds assuming that the agent stores all the transitions using a mechanism equivalent to relational databases where it can search through states in linear time. For the simulator agents that we generally encounter, this assumption almost never holds true. Even though both the queries have membership in the same complexity class, an agent will have to spend more time in running the action precondition query owing to the exhaustive search of all the states in all the cases, whereas for the plan outcome queries, the exhaustive search is not always needed.

Additionally, plan outcome queries place very little requirements on the agent to answer the queries, whereas action precondition queries require an agent to use more computation to generate it’s responses. Action precondition queries also force an agent to know all the transitions beforehand. So if an agent does not know its model but has to execute an action in a state to learn the transition, action precondition queries will perform poorly as agent will execute that action in all possible states to answer the query. On the other hand, to answer plan outcome queries in such cases, an agent will have to execute at most L actions (maximum length of the plan) to answer a query.

Evaluating the responses of queries will be much easier for the action precondition queries, whereas evaluating the responses of plan outcome queries is not straightforward, as discussed in verma2021asking [verma2021asking]. As mentioned earlier, the agent interrogation algorithm that uses the plan outcome queries has asymptotic complexity for evaluating all agent responses. On the other hand, if an algorithm is implemented with action precondition queries, its asymptotic complexity for evaluating all agent responses will reduce to . This is because AAM needs to ask two queries for each action. The first query in a state where it is guaranteed that the action will fail, this will lead AAM to learn the action’s precondition. After that AAM can ask another query in a state where the action will not fail, and learn the action’s effects. This will also lead to an overall less number of queries.

So there is a tradeoff between the computation efforts needed for evaluation of query responses and the computational burden on the agent to answer those queries.

6 Empirical Evaluation

We implemented AIA with plan outcome queries in Python to evaluate the efficacy of our approach. In this implementation, initial states were collected by making the agent perform random walks in a simulated environment. We used a maximum of 60 such random initial states for each domain in our experiments. The implementation is optimized to store the agent’s answers to queries; hence the stored responses are used if a query is repeated.

We tested AIA on two types of agents: symbolic agents that use models from the IPC (unknown to AIA), and simulator agents that report states as images using PDDLGym [23]. All experiments were executed on 5.0 GHz Intel i9-9900 CPUs with 64 GB RAM running Ubuntu 18.04.

The analysis presented below shows that AIA learns the correct model with a reasonable number of queries, and compares our results with the closest related work, FAMA [1]. We use the metric of model accuracy in the following analysis: the number of correctly learned palm tuples normalized with the total number of palm tuples in .

6.1 Experiments with symbolic agents

We initialized the agent with one of the 10 IPC domain models, and ran AIA on the resulting agent. 10 different problem instances were used to obtain average performance estimates.

Table 1 shows that the number of queries required increases with the number of predicates and actions in the domain. We used Fast Downward [13]

with LM-Cut heuristic 

[11] to solve the planning problems. Since our approach is planner-independent, we also tried using FF [14]

and the results were similar. The low variance shows that the method is stable across multiple runs.

Domain (ms) (s)
Gripper 5 3 17 18.0 0.2
Blocksworld 9 4 48 8.4 36
Miconic 10 4 39 9.2 1.4
Parking 18 4 63 16.5 806
Logistics 18 6 68 24.4 1.73
Satellite 17 5 41 11.6 0.87
Termes 22 7 134 17.0 110.2
Rovers 82 9 370 5.1 60.3
Barman 83 17 357 18.5 1605
Freecell 100 10 535 2.24 33.4
Table 1: The number of queries (), average time per query (), and variance of time per query () generated by AIA with FD. Average and variance are calculated for 10 runs of AIA, each on a separate problem. Time in sec.

6.1.1 Comparison with FAMA

We compare the performance of AIA with that of FAMA in terms of stability of the models learned and the time taken per query. Since the focus of our approach is on automatically generating useful traces, we provided FAMA randomly generated traces of length 3 (the length of the longest plans in AIA-generated queries) of the form used throughout this paper ().

Fig. 3 summarizes our findings. AIA takes lesser time per query and shows better convergence to the correct model. FAMA sometimes reaches nearly accurate models faster, but its accuracy continues to oscillate, making it difficult to ascertain when the learning process should be stopped (we increased the number of traces provided to FAMA until it ran out of memory). This is because the solution to FAMA’s internal planning problem introduces spurious palm tuples in its model if the input traces do not capture the complete domain dynamics. For Logistics, FAMA generated an incorrect planning problem, whereas for Freecell and Barman it ran out of memory (AIA also took considerable time for Freecell). Also, in domains with negative preconditions like Termes, FAMA was unable to learn the correct model. We used Madagascar [21] with FAMA as it is the preferred planner for it. We also tried FD and FF with FAMA, but as the original authors noted, it could not scale and ran out of memory on all but a few Blocksworld and Gripper problems where it was much slower than with Madagascar.

Figure 3: Performance comparison of AIA and FAMA in terms of model accuracy and time taken per query with an increasing number of queries.

6.2 Experiments with simulator agents

AIA can also be used with simulator agents that do not know about predicates and report states as images. To test this, we wrote classifiers for detecting predicates from images of simulator states in the PDDLGym framework. This framework provides ground-truth PDDL models, thereby simplifying the estimation of accuracy. We initialized the agent with one of the two PDDLGym environments, Sokoban and Doors. AIA inferred the correct model in both cases and the number of instantiated predicates, actions, and the average number of queries (over 5 runs) used to predict the correct model for Sokoban were 35, 3, and 201, and that for Doors were 10, 2, and 252.

7 Related Work

One of the ways most current techniques learn the agent models is based on passive or active observations of the agent’s behavior, mostly in the form of action traces [7, 27, 4, 28]. jiminez_2012_review [jiminez_2012_review] and arora_2018_review [arora_2018_review] present comprehensive review of such approaches. FAMA [1] reduces model recognition to a planning problem and can work with partial action sequences and/or state traces as long as correct initial and goal states are provided. While FAMA requires a post-processing step to update the learnt model’s preconditions to include the intersection of all states where an action is applied, it is not clear that such a process would necessarily converge to the correct model. Our experiments indicate that such approaches exhibit oscillating behavior in terms of model accuracy because some data traces can include spurious predicates, which leads to spurious preconditions being added to the model’s actions. As we mentioned earlier, such approaches do not feature interventions, and hence the models learned by these techniques do not capture causal relationships correctly and feature correlations.

Pearl19_Seven [Pearl19_Seven] introduce a 3-level causal hierarchy in terms of the classification of causal information in terms of the type of questions each class can answer. He also notes that based on passive observations alone, only associations can be learned, not the interventional or counterfactual causal relationships, regardless of the size of data.

The field of active learning 

[22] addresses the related problem of selecting which data-labels to acquire for learning single-step decision-making models using statistical measures of information. However, the effective feature set here is the set of all possible plans, which makes conventional methods for evaluating the information gain of possible feature labelings infeasible. In contrast, our approach uses a hierarchical abstraction to select queries to ask, while inferring a multistep decision-making (planning) model. Information-theoretic metrics could also be used in our approach whenever such information is available.

Blondel17_identifiability [Blondel17_identifiability] introduced Dynamical Causal Networks which extend the causal graphs to temporal domains, but they do not feature decision variables, which we introduce in this paper.

8 Conclusion

We introduced dynamic causal decision networks (DCDNs) to represent causal structures in STRIPS-like domains; and showed that the models learned using the agent interrogation algorithm are causal, and are sound and complete with respect to the corresponding unknown ground truth models. We also presented an extended analysis of the queries that can be asked to the agents to learn their model, and the requirements and capabilities of the agents to answer those queries.

Extending the empirical analysis to action precondition queries, and extending our predicate classifier to handle noisy state detection, similar to prevalent approaches using classifiers to detect symbolic states [17, 2] are a few good directions for future work. Some other promising extensions include replacing query and response communication interfaces between the agent and AAM with a natural language similar to Lindsay2017 [Lindsay2017], or learning other representations like Zhuo14learnHTN [Zhuo14learnHTN].


This work was supported in part by the NSF grants IIS 1844325, OIA 1936997, and ONR grant N00014-21-1-2045.


  • [1] D. Aineto, S. J. Celorrio, and E. Onaindia (2019) Learning Action Models With Minimal Observability. Artificial Intelligence 275, pp. 104–137. Cited by: §6, §7.
  • [2] M. Asai and A. Fukunaga (2018) Classical Planning in Deep Latent Space: Bridging the Subsymbolic-Symbolic Boundary. In Proc. AAAI, Cited by: §8.
  • [3] G. Blondel, M. Arias, and R. Gavaldà (2017-03) Identifiability and Transportability in Dynamic Causal Networks.

    International Journal of Data Science and Analytics

    3 (2), pp. 131–147.
    Cited by: §4.
  • [4] S. Cresswell, T. McCluskey, and M. West (2009) Acquisition of Object-Centred Domain Models from Planning Examples. In Proc. ICAPS, Cited by: §7.
  • [5] R. E. Fikes and N. J. Nilsson (1971) STRIPS: A New Approach to the Application of Theorem Proving to Problem Solving. Artificial Intelligence 2 (3-4), pp. 189–208. Cited by: §1.
  • [6] M. Fox and D. Long (2003) PDDL2.1: An Extension to PDDL for Expressing Temporal Planning Domains. Journal of Artificial Intelligence Research 20 (1), pp. 61–124. Cited by: §1.
  • [7] Y. Gil (1994) Learning by Experimentation: Incremental Refinement of Incomplete Planning Domains. In Proc. ICML, Cited by: §7.
  • [8] J. Y. Halpern and J. Pearl (2001) Causes and Explanations: A Structural-Model Approach. Part I: Causes. In Proc. UAI, Cited by: §2.2.
  • [9] J. Y. Halpern and J. Pearl (2005) Causes and Explanations: A Structural-Model Approach. Part I: Causes. The British journal for the philosophy of science 56 (4), pp. 843–887. Cited by: §2.2.
  • [10] J. Y. Halpern (2015) A Modification of the Halpern-Pearl Definition of Causality. In Proc. IJCAI, Cited by: §2.2.
  • [11] M. Helmert and C. Domshlak (2009) Landmarks, Critical Paths and Abstractions: What’s the Difference Anyway?. In Proc. ICAPS, Cited by: §6.1.
  • [12] M. Helmert (2004) A Planning Heuristic Based on Causal Graph Analysis. In Proc. ICAPS, Cited by: §2.2.
  • [13] M. Helmert (2006) The Fast Downward Planning System. Journal of Artificial Intelligence Research 26, pp. 191–246. Cited by: §6.1.
  • [14] J. Hoffmann and B. Nebel (2001) The FF Planning System: Fast Plan Generation Through Heuristic Search. Journal of Artificial Intelligence Research 14, pp. 253–302. Cited by: §6.1.
  • [15] N. Immerman (1987) Expressibility as a complexity measure: results and directions. Technical report Technical Report YALEU/DCS/TR-538, Department of Computer Science, Yale University. Cited by: §5.
  • [16] K. Kanazawa and T. Dean (1989) A Model for Projection and Action. In Proc. IJCAI, Cited by: §4.
  • [17] G. Konidaris, L. P. Kaelbling, and T. Lozano-Perez (2014) Constructing Symbolic Representations for High-Level Planning. In Proc. AAAI, Cited by: §8.
  • [18] K. B. Korb, L. R. Hope, A. E. Nicholson, and K. Axnick (2004) Varieties of Causal Intervention. In Proc. PRICAI: Trends in Artificial Intelligence, Cited by: §2.2.
  • [19] D. McDermott, M. Ghallab, A. Howe, C. Knoblock, A. Ram, M. Veloso, D. S. Weld, and D. Wilkins (1998) PDDL – The Planning Domain Definition Language. Technical report Technical Report CVC TR-98-003/DCS TR-1165,

    Yale Center for Computational Vision and Control

    Cited by: §1.
  • [20] J. Pearl (2009) Causality: Models, Reasoning, and Inference. Cambridge University Press. Cited by: §2.2, §2.2, §4.
  • [21] J. Rintanen (2014) Madagascar: Scalable Planning with SAT. In Proc. 8th International Planning Competition., Cited by: §6.1.1.
  • [22] B. Settles (2012) Active Learning. Morgan & Claypool Publishers. External Links: ISBN 1608457257 Cited by: §7.
  • [23] T. Silver and R. Chitnis (2020) PDDLGym: Gym Environments from PDDL Problems. In

    ICAPS Workshop on Bridging the Gap Between AI Planning and Reinforcement Learning (PRL)

    Cited by: §6.
  • [24] S. Srivastava (2021) Unifying Principles and Metrics for Safe and Assistive AI. In Proc. AAAI, Cited by: §1.
  • [25] M. Y. Vardi (1982) The complexity of relational query languages (extended abstract). In

    Proceedings of the Fourteenth Annual ACM Symposium on Theory of Computing

    Cited by: §5.
  • [26] P. Verma, S. R. Marpally, and S. Srivastava (2021) Asking the Right Questions: Learning Interpretable Action Models Through Query Answering. In Proc. AAAI, Cited by: §1, Figure 1.
  • [27] Q. Yang, K. Wu, and Y. Jiang (2007) Learning Action Models from Plan Examples Using Weighted MAX-SAT. Artificial Intelligence 171 (2-3), pp. 107–143. Cited by: §7.
  • [28] H. H. Zhuo and S. Kambhampati (2013) Action-Model Acquisition from Noisy Plan Traces. In Proc. IJCAI, Cited by: §7.