Log In Sign Up

Actions You Can Handle: Dependent Types for AI Plans

Verification of AI is a challenge that has engineering, algorithmic and programming language components. For example, AI planners are deployed to model actions of autonomous agents. They comprise a number of searching algorithms that, given a set of specified properties, find a sequence of actions that satisfy these properties. Although AI planners are mature tools from the algorithmic and engineering points of view, they have limitations as programming languages. Decidable and efficient automated search entails restrictions on the syntax of the language, prohibiting use of higher-order properties or recursion. This paper proposes a methodology for embedding plans produced by AI planners into dependently-typed language Agda, which enables users to reason about and verify more general and abstract properties of plans, and also provides a more holistic programming language infrastructure for modelling plan execution.


page 1

page 2

page 3

page 4


Evolution of artificial intelligence languages, a systematic literature review

The field of Artificial Intelligence (AI) has undoubtedly received signi...

ALPprolog --- A New Logic Programming Method for Dynamic Domains

Logic programming is a powerful paradigm for programming autonomous agen...

Proof-Carrying Plans: a Resource Logic for AI Planning

Recent trends in AI verification and Explainable AI have raised the ques...

Constructing Belief Networks to Evaluate Plans

This paper examines the problem of constructing belief networks to evalu...

Neural Networks in Imandra: Matrix Representation as a Verification Choice

The demand for formal verification tools for neural networks has increas...

Deep Reinforcement Learning for Programming Language Correction

Novice programmers often struggle with the formal syntax of programming ...

Implicit Robot-Human Communication in Adversarial and Collaborative Environments

Users of AI systems may rely upon them to produce plans for achieving de...

1. Introduction

Planning is a research area within AI that studies the automated generation of plans from symbolic domain and problem specifications. AI planners came into existence in the 1970s as an intersection between general problem solvers (ernst1969gps), situation calculus (mccarthy1981some) and theorem proving (green1969theorem).

Typically, the domain is represented by an abstract description of the world and a set of actions that can be used to alter the world states (see Figure 1). Planning problems in the domain consist of the initial state of the world and a goal state (see Figure 2). The planner then produces a plan, i.e. a sequence of actions, moving the world from the initial state to the goal state (see Figure 3). In most domains, the plan produced must not only reach the goal state, but also satisfy other properties such as safety. These properties are encoded via the preconditions of actions. For example, a “rotate” action for a robotic arm might have the precondition that there are no obstacles in the way. The preconditions are taken into account by the planner when creating the plan, and therefore we shall refer to these as intrinsic properties.

Our previous work (SchwaabKHFPWH19; HillKP20) has shown that the operational and declarative semantics of AI planning can be abstractly specified by a simple calculus resembling Hoare Logic (hoare1969axiomatic). Formalisation of this calculus in Agda (bove2009brief) allowed us to prove soundness of the operational semantics. Moreover, in (HillKP20) we showed how the formalisation allowed us to semi-automatically verify that individual plans produced by AI planners are sound with respect to their formal semantics, and therefore that plans produced by the planner really do satisfy the desired intrinsic properties encoded in the action preconditions.

1.1. Verifying extrinsic properties

In this paper, we extend this work to show that our Agda framework can be used to reason about plan properties that the planner itself either cannot or should not reason about. We will refer to these as extrinsic properties.

There are three main classes of extrinsic properties that we have identified:

  1. Inexpressible properties - these are properties that cannot be expressed in the declarative specification language of the planner, for example because they involve high-order functions or unbounded state. A good example of such a property is that the plan produced is fair. Fairness typically involves universally quantifying over all the agents in the problem and keeping track of and comparing state. As discussed in Section 2.2, such global properties are typically impossible to express as pre-conditions of individual actions in the baseline versions of planning languages such as PDDL (mcdermott1998pddl). In contrast, it is easy to express to express and reason about them in a dependently-typed language like Agda.

  2. Unavailable properties

    - these are properties whose evaluation requires world states that are not available at planning time. A good example of such a property is the fuel consumption of a robotic agent. Although the fuel used per action can be estimated at planning time, in practice the amount of the fuel required to carry out an action in the real-world may depend on real-time conditions such as weather, temperature or other local conditions. Therefore, even though it cannot be checked at planning time, it is still desirable to verify that during execution the robotic agent never starts an action that it has insufficient fuel to complete.

  3. Probable properties

    - finally these are properties which plans produced by the planner have a high probability of satisfying. As an example of such a property, we once again consider fairness. Suppose our planner is assigning jobs to workers and we want to verify that the set of assignments does not exhibit gender bias. By default, if the planner does not have access to gender information you would expect the vast majority of plans to be fair. Nonetheless, it is possible that in certain circumstances some other part of the domain may act as a proxy for gender and result in plans that are biased. Such problems are widely known in data science and machine learning 

    (o2016weapons). Even if such a property can be added to the planning domain, the time complexity of planning algorithms is typically super-linear in the size of the domain. Therefore we argue that one should avoid encoding it in the problem domain and only verify the property holds of any produced plans. As the property failure rate is low, one can achieve significant speed-ups at planning time.

We view the distinction between intrinsic and extrinsic properties as mirroring the separation between the search and control components of AI modelling. The intrinsic properties are incorporated into PDDL domains and inform the PDDL search algorithms. In contrast, the extrinsic properties are imposed on the controller and our framework provides a methodology for guaranteeing that the controller does not violate them during the execution of the plan. We envisage this guarantee being used in one of two ways: firstly, prior to execution, the plan can be run in a simulated environment to check for extrinsic property violations. Secondly, during the live execution of the plan it can be used as a form of run-time checking that provides a formally verified guarantee that the controller will never perform an action that violates the extrinsic properties. Note that in this paper we do not address what the controller should do in response to an imminent violation of an extrinsic properties. We discuss work on how one might provide feedback about the violation to the planner in Section 5.

1.2. The technical approach

The work presented in this paper builds upon the earlier work of (SchwaabKHFPWH19), in which the Agda fomalisation of PDDL was first given. However, we substantially clarify and simplify that initial formalisation here.

Our novel technical contribution is the use of action handlers as a means of integrating rich extrinsic properties expressed in the proof and programming environment of Agda with our previous PDDL formalisation. An action handler is a function that, given a state and an action, executes the action by applying the action (seen as a function) to the state. The handlers were introduced in (SchwaabKHFPWH19) as an auxiliary means of establishing a correspondence between the declarative and the operational semantics of AI planning.

In this paper, action handlers become the central tool for building richer program and proof infrastructure around the plans produced by AI planners. In particular we use dependent-types to enrich the handlers with additional constraints representing extrinsic properties that should hold during plan execution. As a result, we obtain enriched action handlers in which we can incorporate additional safety, security, fairness or other checks of arbitrary complexity which are then formally verified by Agda. Crucially, these extrinsic properties can be expressed and verified without altering either the native PDDL problem domain or its formal semantics. Notably, the richer properties we seek to define and prove are specified at the type level. From this point of view, this paper presents a non-trivial exercise in dependently-typed programming.

With regards to future applications, this paper can be seen as a prototype for embedding existing automated reasoning tools within dependently-typed modelling environments. For example, we can perform higher-order reasoning (in Agda’s interactive style) on top of the first-order proof search already performed by the AI planner. This substantially extends the modelling power of the AI planners, as in Agda we can encode many properties that PDDL cannot. This includes function definitions, universal and existential quantification, action dependencies and higher-order quantification. We argue that this approach promises to play an important role in verification of complex AI applications.

1.3. Road map

We proceed as follows. Section 2 contains a brief summary of the PDDL language that is used for planners, illustrated by using a classic taxi planning problem. We then recap the Agda formalisation of plans first developed in (SchwaabKHFPWH19; HillKP20), including the notion of the canonical action handler as motivated by the running example. Section 3 introduces the novel method of enriched handlers by illustrating how to model and incorporate rich extrinsic verification properties into the type level of handlers. Section 5 discusses future work, including on the handling of failure of the extrinsic properties and how this work relates to Explainable AI.

2. PPDL, Plans, Action Handlers & Agda

In this section, we provide an introduction to the PDDL planning language and the essential parts of the Agda formalisation accompanying (HillKP20), thereby providing some clarification and simplification of that formalisation. This will then pave the way to Section 3 in which we explain how to extend the formalisation to allow the embedding of extrinsic properties. We refer the reader directly to (HillKP20) for more theoretical aspects of the previous work.

[frame=single,rulecolor=,label= 1. The notion of domain] (define (domain taxi) (:requirements :strips :typing)

[frame=single,rulecolor=,label= 2. Types] (:types taxi location person)

[frame=single,rulecolor=,label= 3. Predicates] (:predicates (taxiIn ?obj1 - taxi ?l1 - location) (personIn ?obj1 - person ?l1 - location))

[frame=single,rulecolor=,label= 4. Actions] (:action drive_passenger :parameters (?t1 - taxi ?p1 - person ?l1 - location ?l2 - location)

[frame=single,rulecolor=,label= 5. Action preconditions and effects] :precondition (and (taxiIn ?t1 ?l1) (personIn ?p1 ?l1)) :effect (and (not (taxiIn ?t1 ?l1)) (not (personIn ?p1 ?l1)) (taxiIn ?t1 ?l2) (personIn ?p1 ?l2))) (:action drive :parameters (?t1 - taxi ?l1 - location ?l2 - location) :precondition (taxiIn ?t1 ?l1) :effect (and (not (taxiIn ?t1 ?l1)) (taxiIn ?t1 ?l2))))

Figure 1. The PDDL Taxi Domain, with main logical blocks outlined in boxes.
(define (problem taxi)
  (:domain taxi)
      taxi1 taxi2 taxi3 - taxi
      person1 person2 person3 - person
      loc1 loc2 loc3 - location)
  (:init  (taxiIn taxi1 loc1)
          (taxiIn taxi2 loc2)
          (taxiIn taxi3 loc3)
          (personIn person1 loc1)
          (personIn person2 loc2)
          (personIn person3 loc3))
  (:goal (and  (taxiIn taxi1 loc2)
               (personIn person1 loc3)
               (personIn person3 loc1))))
Figure 2. A Taxi planning problem expressed in PDDL. Initial state: There are three taxis with taxi1 being in loc1, taxi2 in loc2 and taxi3 in loc3. There are also three people with person1 being in loc1, person2 in loc2 and person3 in loc3. Goal state: taxi1 is in loc2, person1 is in loc3 and person3 is in loc1.

plan = (drive_passenger taxi3 person3 loc3 loc1); (drive taxi1 loc1 loc2); (drive_passenger taxi3 person1 loc1 loc3)

Figure 3. One possible solution to the Taxi planning problem in Figure 2

2.1. PDDL Syntax

Many versions of planning languages were proposed, and the Planning Domain and Definition Language (PDDL) (mcdermott1998pddl) aimed to standardise them. One notable design decision of PDDL is the splitting of the planning problem into domain and problem descriptions. The domain describes the predicates and admissible actions (as shown in Figure 1), while the problem description defines specific initial and goal states (Figure 2).

We begin by explaining how each part of a planning domain and a planning problem are translated from PDDL into our dependently typed framework. In general, we maintain two kinds of Agda files. The first are the files which hold general definitions of the PDDL syntax, contexts and inference rules and are parametrised by an abstract domain. The second are example files which contain concrete encodings of the planning example in question, which then instantiate the generic modules with the corresponding parts of the encoding. The user only has to interact with the latter to specify the properties over their PDDL domain, whilst former can be hidden from their view.

An abstract planning domain

Types, predicates and actions (blocks 2, 3 & 4 in Figure 1) are the basic components of any PDDL domain definition, and abstractly these are represented as three Agda sets Type, Predicate and Action.

To indicate whether a predicate is true or false we map it to a Polarity, a set that contains two elements, and . We then have a notion of state that is a list of polarities mapped to predicates. The State type is used to represent preconditions and effects of actions in a domain as well as the goal state in the problem specification. We will refer to specific instances of the State type using the types Precondition, Effect and GoalState.

[0]State=List(Polarity × Predicate)

The notion of actions’ preconditions and effects (block 5 in Figure 1) are defined generically as the ActionDescription record: [0]recordActionDescription:Setwhere
[4]effects[10]:Effects and a context maps every Action to an ActionDescription: [0]Context:Set
[0]Context=Action→ActionDescription This then allows us to represent an abstract planning domain (block 1 in Figure 1) as the following record: [0]recordDomain:Set₁where

The taxi planning domain

We can then instantiate the taxi domain as follows. To describe PDDL types one simply needs to create a data type in Agda with the required types as constructors. [0]dataType:Setwhere
[0][@l@0][2]taxilocationperson:Type In PDDL, objects are first introduced implicitly as typed variables within the block that defines predicates, and only later does the planning problem give an explicit set of objects that can be used to instantiate these variables. For convenience, in our formalisation we combine these two separate notions of objects and variables into a single Object data type whose constructors are indexed by Types. The number of objects for each constructor are given by a finite number indicated by Fin. For example if numberOfTaxis was equal to 3 then we can construct taxis: taxi 0, taxi 1, taxi 2. Thus, the second block of Figure 1 boils down to the following data declarations: [0]dataObject:Type-¿Setwhere
[2]person[10]:FinnumberOfPeople[20]→Objectperson In practice, developing this prototype system further may require one to properly distinguish between objects and variables, but we leave this for future work. In (SchwaabKHFPWH19) objects were defined simply by naming them as constructors. However, in this paper we need to add additional information about the number of available taxis in order to reason about extrinsic properties of plans, such as fairness.

We can now define predicates over typed objects, closely mimicking the PDDL syntax in block 3 of Figure 1. [0]dataPredicate:Setwhere
[2]personIn[10]:Objectperson[20]→Objectlocation[30]→Predicate Actions (see block , and in Figure 1) are defined as another data type. [0]dataAction:Setwhere
[0][@l@0][2]drivePassenger:… The context that details each action’s preconditions and effects can be easily instantiated in a manner that is close to the PDDL syntax:


The planning problem

The specific planning problem (see Figure 2) needs to be defined concretely, by providing an initial World and a GoalState. Intuitively, a World is a logical description of the predicates that are true. We operate under the same closed world assumption as PDDL where all predicates that are not contained in a world are assumed to be false. [0]World:Set
[0]World=ListPredicate Whereas we use a World to represent the initial state, we need to be able to talk about specific negative predicates in the GoalState so it is a simple alias for State.



One of the most popular early planners was the Stanford Research Institute Problem Solver (STRIPS) (strips) which was created to address the problems faced by a robot in rearranging objects and in navigating. The STRIPS planner will perform an automatic search for a plan that moves from the initial world to the goal state defined in the domain. One such plan that it might find for the problem outlined so far is shown in Figure 3.

We define a Plan as a list of actions, (renaming the empty list to halt to improve readability). [0]Plan:Set
[0]Plan=ListAction The plan shown in Figure 3 can then be defined as: [0]plan:Plan

These are the main building blocks that we expect to receive from the given AI planner.

2.2. Expressivity of PDDL

PDDL is a very expressive language with many extensions. PDDL 1.2 usually operates under a closed world assumption and expresses domains using the STRIPS assumption where actions effects are applied by adding and deleting predicates to a given world. The closed world requirement implies the use of first-order logic without function symbols (which guarantees finite domains when defining the models). The problem with functions, especially with recursive functions, is that they can make domains infinite. For example, it only takes one nullary and one unary function to generate the set of natural numbers.

PDDL 1.2 also allows for the expression of types with type hierarchy, equalities over objects, existential and universal quantification over preconditions and conditional effects. Conditional effects are effects that will only be applied when a list of preconditions hold true. In PDDL 2.1 there is also a definition of numeric fluents that allow for the reasoning about numbers such as comparing and adding numbers. PDDL 2.1 also introduces negative preconditions and durative actions. Durative actions add the concept of time to actions. Finally PDDL 3 adds strong and soft constraints that can be applied across a planning problem. Strong constraints can allow for the statement of certain implications to hold across every state during the execution of a plan. Soft constraints, also known as preferences, introduce soft goals that a user would prefer a planner to satisfy but are not necessary to satisfy for a valid plan. In this paper we will mainly focus on a subset of PDDL 1.2 under the closed world assumption.

Two of the above restrictions in particular are the subject of the syntactic (type-driven) extensions we propose in this paper: we do rely on arbitrary functions in our development, and we open ways to surpass the closed world assumption, by embedding the plans in a wider programming and modelling environment. We also use higher-order functions and predicates to express some more sophisticated properties, for example calculating the number of taxi’s that satisfy a certain property as discussed in Section 3.2.

2.3. Validating intrinsic properties of a plan

We now briefly overview the calculus in which Agda validates the plans given by AI planners. The calculus is very simple, consisting of just two rules: action sequencing and halting. The main intuition behind the rules is that the goal state as well as the actions defined in describe minimal preconditions and effects, whereas plans are executed on potentially larger states.

We define a satisfaction relation between a world and a state , denoted , where a world satisfies a state when all positively mapped predicates in a state are in the world and all negatively mapped predicates are not in the world. [0]∈⟨⟩:World→State→Set

The rules of our calculus then say that it is safe to halt a plan if our current world satisfies the goal state, and it is safe to sequence (seq) another action to a plan if the current world satisfies the action’s precondition. As expected, the rules of inference are then defined as an inductive relation plan initialWorld goalState:

[0]data⊢∶↝:Context→Plan→World → GoalState→Set
[4] →world∈⟨goal⟩
[2]seq:∀{ αworldgoalΓf}

The actual sequencing is performed by the updateWorld function, that applies the effects of an action by adding all the positively mapped predicates to the world and removing all the negatively mapped predicates. It should be noted that we do not need check the preconditions of an action in the updateWorld function as we only apply actions on a world after ensuring that the world satisfies the preconditions of the action in the typing relation.


In (SchwaabKHFPWH19), these rules were proven sound and complete relative to the possible world semantics of PDDL. Compared to (SchwaabKHFPWH19) we have simplified the rules so that we use a World rather than a State to represent the initial state of the problem. Technically speaking, this is all we need to validate the intrinsic properties of a PDDL plan. Since the rules are so simple, it is possible to generate Agda proofs automatically from PDDL plans, which we implemented as a function solver in (HillKP20). Since the rules are defined generically, a user who works on a specific plan validation does not have to do anything, except for supplying a generic validation command (in which they insert the given plan as well as initial world and goal state):


The initial motivation behind this work was in giving Curry-Howard, or computational interpretation to AI plans, with a view of opening the way to a certified code extraction. With that in mind, the inference rules defining

model plans as functions that inhabit the type


In Section 5 we also discuss how this approach differs from modelling PPDP plans within linear logic.

2.4. Plan Execution: Action Handlers

In (SchwaabKHFPWH19) we introduced the notion of a canonical action handler, that can take a plan validated as in previous section, and turn it into an executable function over the possible worlds, as defined in PDDL semantics.

In order to discuss our approach of verifying extrinsic properties, only the notion of the possible World is relevant. We refer interested readers to  (SchwaabKHFPWH19) for a complete definition of the possible world semantics.

A handler executes Actions on Worlds: [0]ActionHandler:Set

We now define a canonical handler which applies the effects of an action according to the context by using the updateWorld function.


To be able to evaluate an entire plan we define an execute function that takes in a plan, action handler and initial world as its arguments and recursively applies all actions in the plan using the given action handler to the world until the end of the plan. [0]execute:Plan→ActionHandler→World→World
[0]execute(α∷f)[20]σw=executefσ(σαw) Note that in this case execute could simply be defined as a fold over the list of actions. We have left it in this explicit form, as in the next section we will alter the definition of ActionHandler to use dependent types in order to encode rich extrinsic properties, which means that expressing this as a fold is no longer possible.

We can now evaluate the taxi example by applying the execute function to the canonical handler and initial world. [0]evaluationResult:World
[0]evaluationResult=executeplan(canonical-σΓ)initialWorld As we execute the already validated plan on the initialWorld, we expect to see, as an output, a world that corresponds to the goal state goalState. In fact, we get: [0]Output:
[0] taxiIn taxi3 location3 ∷
[0] personIn person1 location3 ∷
[0] personIn person3 location1 ∷
[0] taxiIn taxi1 location2 ∷
[0] taxiIn taxi2 location2 ∷
[0] personIn person2 location2 ∷ [] That is, the world that the function evaluationResult returns is larger than the world the goalState directly entails, but this is expected, as long as the information contained in goalState is preserved.

Note that generally, given a state, there may be many worlds that satisfy it. For example, the following world also satisfies our goalState. [0] taxiIn taxi3 location3 ∷
[0] personIn person1 location3 ∷
[0] personIn person3 location1 ∷ []

The central soundness result of (SchwaabKHFPWH19) states that, whenever we can prove that plan  :  initialWorld goalState then executing the plan on initialWorld will result in a world that satisfies conditions in the goalState.

We rely on our proof of plan soundness to establish that the plan, and therefore each action in it, is valid. As a consequence we do not require action handlers to check the validity of each action with respect to the current world before applying it. This allows us to simplify the definition of the enriched handlers, described in the next section.

This finishes the recap of canonical handlers from (SchwaabKHFPWH19), we are now ready to tackle extrinsic properties in the next section.

3. Verifying extrinsic properties

So far we have introduced two out of three components of our proposed framework:

  • Plan generation via a PDDL planner which takes a PDDL domain and problem definition as in Figures 1 & 2, and performs an automated search for plan that takes the system from the initial to the goal state.

  • Validation of the resulting plan via Agda which compiles the planning domain, planning problem and plan received from the planner into a compact DSL. The plan is then validated relative to the formalised operational and possible world semantics of PDDL. We have shown in  (HillKP20) that this validation stage is capable of eliminating state inconsistencies that are otherwise admitted by the STRIPS planner.

We are now ready to introduce a third, and perhaps the most intriguing component of the framework:

  • Dependently-typed verification of extrinsic properties of the execution of the plan via Agda, in which we formally verify that during execution of the plan the controller will never execute an action that violates extrinsic properties. As discussed previously, extrinsic properties are those which are either undesirable or impossible to encode in PDDL at planning time.

To achieve this, we augment the ActionHandler type with the desired property and then ensure that the execute function has the correct type. Note that this third stage lacks the generality of the stage (II), as these higher level properties are necessarily specific to the particular domain being modelled. Nonetheless we argue it is a powerful and flexible technique for verifying properties that cannot be checked at planning time. A notable advantage of our approach is that we can verify a property holds without altering the semantics of PDDL specification or the shape of the plans produced by the planner.

As discussed in the introduction, the verified controller could be used directly during execution. Alternatively, it could be run in a simulated environment prior to the execution of the plan by a non-verified controller. If the plan satisfies the extrinsic properties during simulation, then the environment can be monitored during execution and as long as it doesn’t deviate from the simulation then the actual execution of the plan will not violate the extrinsic properties either.

3.1. Example 1: Fuel Consumption

One very simple property that we might be interested in verifying is that the agent never runs out of fuel while executing a plan. Although fuel is often used in an abstract sense in functional programming to limit the number iterations a function may perform before termination, in planning fuel often has a very real interpretation as it represents a resource (e.g. electrical energy) that an agent uses to perform actions. Typically, before an agent runs out of fuel it must return to its base and recharge.

In many domains, fuel levels cannot be taken into account by the planner in stage (I) because it is unknown what the exact fuel level will be at a given point in the plan. For example while the plan of driving from location1 to location2 and then from location2 to location3 may be valid at planning time, it can subsequently be invalidated by unexpectedly high fuel consumption during the first leg of the trip (e.g. due to a diversion caused by road-works) that leaves the taxi unable to complete the second leg.

We will now show how such a constraint can be incorporated into our framework. For simplicity and pedagogical purposes we will assume that all taxis share a single fuel source and that applying any action uses unit of fuel. To add this property to an action handler we first model the property using types in Agda and then add that property to the type level of the action handler. To allow us to reason about natural numbers at the type level we create a Fuel data type that is indexed by a natural number:

[0][@l@0][2]fuel:(n:Nat)→Fueln We can now enrich the definition of an action handler, by encoding the fact that applying an action reduces the fuel level from suc to where suc is the successor of the natural number . [0]FuelAwareActionHandler=∀{n}→Action
[22]→World×Fueln This encodes at the type level that the agent cannot begin to execute an action without having sufficient energy and that each action uses one unit of fuel.

Now we can define an enriched handler, by changing the canonical handler’s return type to FuelAwareActionHandler: [0]enriched-σ:Context→FuelAwareActionHandler
[0]enriched-σΓα=updateWorld’(effects(Γα)) The auxiliary function updateWorld’ above is an enriched version of updateWorld we used in the Section 2.4. It takes care of checking the additional fuel constraint on the type of the action handler is satisfied during the execution: [0]updateWorld’:Effect→[24]World×Fuel(sucn)
[0]updateWorld’s(w,fuel(sucn)) =updateWorldSw,fueln One advantage of defining at the type level that the fuel goes from suc to is that we are forced to supply an energy of exactly in the return type of this function.

To execute plans with the FuelAwareActionHandler and check the constraints during execution, we need to further enrich the evaluation function. The evaluation function must check the fuel level and if it is suc we handle the action and if it is zero whilst there are still actions to apply then the plan fails in which case we return an error message with the failure. One simple way to implement this is to introduce a disjunction in the return type where the function can either return a world or an error if the execution fails. To do this we define a OutOfFuelError data type that is constructed by passing in the current world and the failed action. [0]dataOutOfFuelError:Setwhere

[0][@l@0][12]executeWithFuelfσ(σα(w,fuel(sucn))) We can now execute the same plan that we validated in the previous section, only this time we have the enriched (rather than canonical) handler and evaluation function:


This section used a simple fuel consumption example to explain the general approach of reasoning about meta-properties of already validated plans and demonstrated how enriched handlers allow us to introduce arbitrary additional constraints at execution time without interfering with either the native (sound) semantics of PDDL, or the shape of the native plans produced by STRIPS. In a realistic system, fuel levels might be better implemented as a monadic evaluation function that performs real-time measurement of the current fuel level.

3.2. Example 2: Fairness

We will now look at a more complex constraint, in particular that the assignment of taxi drivers to trips exhibits no significant gender bias. Unlike the fuel consumption example, the gender information could be made available to the planner at Stage (I). However it is infeasible and undesirable to do so for the following two reasons. Firstly, any non-trivial fairness property is unlikely to be expressible in standard PDDL syntax. Secondly and perhaps more subtly, statistically speaking we would expect there to be no gender bias in the output of the planner in the first place. The time complexity of planning algorithms are normally non-linear in the size of the domain description, so we why complicate the planning stage to enforce something that should be normally true most of the time? Verifying that the property holds only at execution time significantly reduces the cost.

To encode this property in Agda we first need to define a model of gender in Agda. [0]dataGender:Setwhere
[0][@l@0][2]malefemaleother:Gender We then define a TripCount type which is used to store the number of trips each gender has taken so far. [0]TripCount:Set
[0]TripCount=Gender→ℕ We will define the code associated with the enriched handler in a separate Agda module. The advantage of this choice is that we can pass in static functions representing data that we do not intend to change during the evaluation of a given domain. [20]driverGender:Objecttaxi→Gender
[20]margin:Nat In particular we pass in two functions driverGender and one natural number called margin. The driverGender function maps all taxi drivers to a Gender. The margin is used to allow for some leeway for statistical fluctuations when enforcing our fairness constraint.

We simply calculate the total trips taken by adding the trips taken for all genders.


The percentage of trips assigned to a given gender is then calculated via the following function: [0]calculateGenderAssignment:Gender→TripCount→ℕ
[0][@l@0][2](tripCountg*100)/₀totalTripsTakentripCount To calculate a fair percentage of assignments we first need to calculate the number of drivers of each gender. Note that this uses a higher order function filter which, as discussed in Section 2.2, are not supported by the PDDL language. Neither are we aware of any alternative way of expressing this calculation in PDDL short of providing the taxis of each gender manually, an approach which scale extremely poorly as the domain grew in size. [0]noGender:Gender→ℕ
[0][@l@0][2]length(filter(λt→decGenderg(driverGendert))allTaxis) Using this we can then calculate the percentage of drivers of a given gender: [0]percentage:Gender→ℕ
[0]percentageg=(noGenderg*100)/₀totalDrivers The lowest acceptable threshold that is deemed to be fair, which is controlled by a margin parameter, is then calculated as follows: [0]calculateLowerbound:Gender→ℕ
[0][@l@0][3]percentageg∸(percentageg/₀margin) We can now express the property that a trip count is unbiased for a particular gender as follows: [0]IsFair:Gender→TripCount→Set
[0][@l@0][2]calculateGenderAssignmentgf[33]≥calculateLowerboundg We have defined a fairness property for a single gender we want to enrich an action handler so that applying an action is fair for all genders not just one. This is modelled by adding a IsFairForAll type that is the product of the type for all genders. [0]IsFairForAll:TripCount→Set
[0]IsFairForAllf=∀(g:Gender)→IsFairgf There are still two problems with implementing the action handler just using the type. The first problem is that it is unreasonable to assume that after the assignment of one or just a few trips that the trips will be fairly assigned. To model this we add the following predicate: [0]UnderMinimumTripThreshold:TripCount→Set
[0][@l@0][2]totalTripsTakentripCount¡totalDrivers*10 The second problem is that there are two actions drive and drivePassenger and only the latter should count as a paying trip for the purpose of fairness. Again this is represented by another predicate: [0]TripAgnostic:Action→Set
[0]TripAgnostic(drivetl1l2)=⊤ We now have sufficient definitions to describe the fairness property in detail, in which an action is fair if it satisfies any of the three predicates defined above: [0]dataActionPreservesFairness
[2][@l@0][4]→ActionPreservesFairnessαtripCount The type of enriched action handlers that enforce this property can then be defined as follows: [0]GenderAwareActionHandler:Set
[2]→World→World One thing to note is that the form of this definition is slightly different from that of the FuelAwareActionHandler defined in the previous section. Instead of adding TripCount as a part of a product with the World, we add it as an implicit argument. This is because, unlike fuel, we’ve chosen not to enforce any type-level relationships between the trip count before an after applying the action. Instead we will rely on our enriched execute function to update the trip count correctly. The disadvantage of this approach is that one cannot enforce relationships between actions and the additional enriched state at the type-level, however the advantage of this is that it allows us to use exactly the same form for the enriched handler and canonical handler instances: [0]enriched-σ:Γ→GenderAwareActionHandler
[0]enriched-σΓα=updateWorld(effects(Γα)) Another advantage of working in a rich dependently-type language such as Agda is that our execution function can return error messages containing proofs in them explaining exactly why the execution of the function failed. Currently a failed execution just returns a precise error for why an execution has failed however we envision that we could use these precise errors for plan repair in future work. In this case our error contains a proof of why the action is not fair: [0]dataGenderBiasError:Setwhere
[2][@l@0][4]→¬(ActionPreservesFairnessaf)→GenderBiasError The enriched execute function can be then be defined to check for fairness and can only execute in action if it can generate a proof that the action will not result in any gender bias: [0]execute’:[747I]Plan→

4. Implementation, Code Extraction, Further Applications

The accompanying Agda library (DFHKS21) is arranged in a way that is friendly to users from the AI planning community. This is the summary of the general methodology to set up, verify and execute a PDDL problem using an enriched handler in our Agda library:

  1. Import the Semantics and Plan files from the Plan folder.

  2. Create and import a Domain file for your problem.

  3. Define an initial world, goal state and a plan.

  4. Use the typing derivation to check that the plan is valid for the initial world and goal state provided.

  5. Create an enriched handler and evaluation function:

    1. Model the additional properties as types.

    2. Show that the additional properties are decidable if necessary.

    3. Create the relevant error types.

    4. Define an action handler that includes the additional properties.

    5. Define an evaluation function for the action handler.

    6. Define an enriched canonical handler.

  6. Import the enriched handler that you want to use.

  7. Use the relevant evaluation function to execute your handler on the initial world.

Although the primary purpose of the presented work is to test the limits of type-driven code development in AI, we have put some thought into future extensions and applications of this work.

In this paper we have manually performed steps 2, 3 and 4, however in previous work  (HillKP20) these steps were fully automated to relieve the burden on the user. The automation is not immediately transferable to the paper due to changes in the Agda formalisation mentioned in Section 2, however it would be relatively easy to update it.

As for applications to AI planning, we envisage several. Firstly, one can use our implementation of the plan validator to verify plans using the typing relation. In  (HillKP20) we showed a few examples of when this exercise can reveal surprising (and often undesirable) properties of plans produced by STRIPS.

A second use for this methodology is suggested by Agda’s infrastructure for code extraction. It is easy enough to extract the examples that we implemented either to Haskell or to binaries, with the repository (DFHKS21) contains some detailed description of the extracted files we obtain as a result (accompanying previous papers on this topic). Thus, one can imagine future deployment of such verified code directly on to robots.

Finally, and as discussed, there may be use cases when software and hardware requirements, or indeed legal regulations, do not permit the direct deployment of code extracted from Agda. For example in the autonomous car industry, the set of usable languages is strictly regulated. In such cases, the methodology we proposed can be used as part of a broader modelling and simulation environment. In fact, we believe this to be the most promising avenue for applications of these ideas. The enriched handlers proposed in this paper enhance exactly this modelling aspect, by opening a way for lightweight and flexible modelling of arbitrary properties of plans separately from (and in addition to) the automated plan search performed by an AI planner such as STRIPS.

5. Conclusions, Related and Future Work

We have presented a novel methodology of using enriched handlers for embedding AI plans into a richer programming and modelling environment in Agda. Our main focus was to show that the idea of a verification framework combining automated solvers and planners on the one hand and richer type-driven programming environments on the other hand has its merits, and can be implemented in an interesting, natural and even user-friendly way. We hope that this line of work inspires more applications in AI verification in the future.

Apart from our own work (HillKP20; SchwaabKHFPWH19), we are not aware of any other approaches to (dependent) type driven methodology for AI plans. However, more broadly the logic and programming language communities have paid attention to AI planning in the past.

AI Planning and Linear Logic. There is a long history of modelling AI planning in Linear logic, that dates back to the 90s (jacopin1993classical), and was investigated in detail in the 2000s, see e.g. (chrpa2007encoding; steedman2002plans). In fact, AI planning is used as one of the iconic use-cases of Linear logic (polakow2001ordered). The main idea behind using Linear logic for AI planning is treating action descriptions as linear implications:

where and

are given by tensor products of atoms:

. We could incorporate information about polarities inside the predicates, as follows: . Then, the linear implication and the tensor products model the resource semantics of PDDL rather elegantly.

The computational (Curry-Howard) interpretation of AI plans was not the focus of study in the above mentioned approaches, yet it plays a crucial role in this paper, from design all the way to implementation, verification and proof extraction.

AI Planning and (Linear) Logic Programming.

The above syntax also resembles linear logic programming Lolli, introduced by Miller et al 

(HodasM94). Lolli was applied in speech planning in (DixonST09).

Our previous work (SchwaabKHFPWH19) in fact takes inspiration from Curry-Howard interpretation of Prolog (0001KSP16; FuK17). In our previous work and in general, logic programming does not work well with PDDL negation. In PDDL, we have to work with essentially three-valued logic: a predicate may be declared to be absent or present in a world. But if neither is declared, we assume a “not known” or “either” situation. Logic programming usually uses the approach of “negation-as-failure” that does not agree with this three-valued semantics. A solution is to introduce polarities as terms, as shown in the example above. This merits further investigation.

Curry-Howard view on Linear Logic. Curry-Howard semantics of Linear logic also attracted attention of logicians first in the 90s (AlbrechtCJ97), and then in the 2000s in connection with research into Linear Logical Frameworks (Schack-NielsenS08; CervesatoP02).

The work that we do relates to that line of work, and can be seen as a DSL for AI planning. It is simpler and less expressive than Linear logic generally but makes up for it in simplicity and close correspondence to PDDL syntax. Transformations between PDDL domain and problem descriptions to Agda syntax are straightforward by design of the DSL. This enables us to automate the generation of Agda proofs from PDDL plans.

Origins of the Frame Rule. The “frame problem” that inspired the frame rule of Separation logic actually has origins in AI (hayes1981frame; dennett2006cognitive). Initially, the problem referred to the difficulty in local reasoning about problems in a complex world. In AI planning specifically, this problem consisted of keeping track of the consequences of applying an action on a world. Intuitively, one understands that driving one passenger in one taxi would have no effect on a journey time of another passenger in another taxi. The frame problem deals with the way to represent this intuition formally.

One way to deal with the frame problem is to declare “frame axioms” for every action explicitly. This is an inefficient way to deal with this problem as defining these frame axioms becomes infeasible the larger the system gets (dennett2006cognitive). Since most actions in AI planning only make small local changes to the world, a more general representation would be more suitable. STRIPS deals with this problem by introducing an assumption that every formula in a world that is not mentioned in the effect list of an action remains the same after execution of the action. This is known as the “STRIPS assumption” and it is an assumption that PDDL also uses.

The logic of Bunched Implications (o2001local; ishtiaq2001bi) and Separation Logic (o2007resources) took inspiration from this older notion of the frame problem, and introduced more abstract formalism, which is now known as a “frame rule”, into the resource logics (pym2019resource). This family of logics has brought many theoretical and practical advances to modelling of complex systems, and is behind many lightweight verification projects (calcagno2015moving).

Outside of logic and semantics communities, AI planning researchers recently started to invest more effort into explaing and validating plans, as well as in modelling extrinsic properties. We highlight two approaches in particular.

Formalisation of planning in other theorem provers. Much of the work we proposed here could be replicated in another dependently-typed prover, such as Coq, Idris or Lean. In addition, there has been an impressive line of work by Abdulaziz and co-authors on formal verification of plan validators in Isabelle/HOL, see e.g. (abdulaziz2018formally; AbdulazizB21; AbdulazizL20). One advantage of using Isabelle/HOL for PDDL formalisations is availability of extensive mathematical libraries that can support proof development. On the other hand, dependently-typed provers are particularly attractive for “lightweight verification” via type-driven program development. This is exactly the feature we wanted to show-case in this paper.

Explainable AI. Extrinsic tools that introduce meta properties over PDDL are already being used in the field of Explainable AI. In (cashmore2019towards) a wrapper over PDDL was created so that users can express “contrastive questions” to better understand and explore why a planner has chosen certain actions over others. An example contrastive question could be ”Why did you choose action A rather than B?”. To accomplish this, users give questions in natural language which are then converted into formal constraints that are then compiled down into PDDL. These additional constraints force the planner to choose different actions which the wrapper will use to generate a contrastive explanation by comparing the original plan to the new plan generated from the additional constraints. The user can then add additional constraints by asking further contrastive questions. This ability to ask further questions is particularly useful as it allows a user to build complex constraints to gain a deeper understanding of a plan.

Plan-property Dependencies. There is also work (eifler2020new; eifler2020plan) that introduces plan-property dependencies which impose boolean functions over plans which allows a user to query why a plan satisfies certain properties over others. These properties are equivalent to soft goals in PDDL (gerevini2005plan). This work explains plans by showing the cost of satisfying certain properties over others by computing the minimal unsolvable goal subsets of a planning problem. An example question in this work could be ”Why does the plan not satisfy the property X?” and a potential reply could be ”because then we would have to forgo property Y and property Z”. To be able to do this, they compile plan properties into goal facts and then compute the minimal unsolvable goal subsets to produce plan explanations. This work can also reason about plan properties in linear temporal logic.

In comparison to our work, both of the previous approaches define extrinsic properties in a domain-independent manner. Whilst the verification and execution of plans in our system is domain-independent, the enriched handlers are not necessarily domain independent. For example, the more generic properties of FuelAwareActionHandler could be used in any domain, however the GenderAwareActionHandler

is defined specifically for the taxi domain. The benefit of our approach is that we can define complex properties that would be undefinable in either of the previous systems. However, at the current moment we have no way to compile our properties into PDDL when a plan fails.

One area of future work that we would like to focus on is plan repair. In our current system we can verify additional properties of plans using our enriched handlers but we have no obvious course of action for what to do once a plan fails. In this paper we have tried to address this by choosing additional constraints that will most likely be satisfied by a planner without any additional replanning. We believe that we could address this issue by compiling down additional constraints to PDDL based on the extrinsic properties of the enriched handler. Since the extrinsic properties can not be easily expressed in PDDL we can create compilation strategies based on the errors produced by failed evaluations to force the planner to pick different actions. For example, if we have a plan that fails in our taxi domain because it has disproportionately picked men over women in a plan we could fix this by removing a certain number of male taxi drivers from the planning problem so that the planner no longer has the option to choose them. This could be further enhanced by modelling partial plans where a new PDDL problem can be created at a failure point in a plan. This would potentially reduce the amount of replanning needed.

In previous work (HillKP20) we fully automated our system so that verification and execution of plans can be generated from PDDL domain and problem descriptions. This should ensure that there is a low barrier for entry for new users in terms of Agda and programming language knowledge. Because the extrinsic properties (modelled by the enriched handlers) are not part of the PDDL domain or problem, we cannot provide the same level of automation for generating these. In future work we intend to address this by creating a more user-friendly infrastructure for defining the extrinsic properties. For example, a DSL for enriched handlers is an option worth considering. This would mean that a user would only have to learn how to use the DSL. Implementing such a DSL may even open opportunities for automating the feedback loop from Agda to PDDL. A drawback of this approach would be that we will have to restrict the expressibility of enriched handlers.

The first author acknowledges support of the EPSRC Doctoral Training scheme; the second and third author acknowledge generous support of the EPSRC grant AISEC: AI Secure and Explainable by Construction, EP/T026952/1, The authors also thank the TyDe’21 reviewers for their valuable input, which lead to substantial clarification and simplifaction of some of the code accompanying this paper.