Structural and semantic pattern matching analysis in Haskell

by   Pavel Kalvoda, et al.

Haskell functions are defined as a series of clauses consisting of patterns that are matched against the arguments in the order of definition. In case an input is not matched by any of the clauses, an error occurs. Therefore it is desirable to statically prove that the function is defined for all well-typed inputs. Conversely, a clause that can never be matched also indicates a likely defect. Analyzing these properties is challenging due to presence of GADT and guards as well as due to Haskell's lazy evaluation. We implement a recently proposed algorithm that unifies and extends the related analyses implemented in version 7 of the Glasgow Haskell Compiler. By using an SMT solver to handle the semantic constraints arising from pattern guards, we achieve a further improvement in precision over the existing GHC 8.0.1 implementation. We present a tool that uses the analysis to give sound, more precise, and actionable warnings about program defects.


page 1

page 2

page 3

page 4


Unification in Matching Logic - Extended Version

Matching Logic is a framework for specifying programming language semant...

Unification in Matching Logic

Matching Logic is a framework for specifying programming language semant...

Set Constraints, Pattern Match Analysis, and SMT

Set constraints provide a highly general way to formulate program analys...

The Improved GP 2 Compiler

GP 2 is a rule-based programming language based on graph transformation ...

Source Matching and Rewriting

A typical compiler flow relies on a uni-directional sequence of translat...

Identifying Overly Restrictive Matching Patterns in SMT-based Program Verifiers

Universal quantifiers occur frequently in proof obligations produced by ...

1 Introduction

In Haskell, functions are defined as one or more clauses that consist of one pattern for each of the formal parameters. When a function application is evaluated, the program attempts to match the arguments against the clauses in the order of definition and the right hand side of the first clauses that matched is evaluated. If no clause was matched, the program terminates with an error.

For example, consider the following function: haskell pairs :: [a] -¿ [(a, a)] pairs [] = [] pairs (x:y:zz) = (x, y):pairs zz Calling haskellpairs with haskell[1] will result in a pattern matching error and ultimately an erroneous termination. This is synonymous to the return value being haskellundefined or bottom, a special value that is a member of every type and indicates an unsuccessful computation.

A Haskell function that terminates with non-bottom value for all non-bottom inputs is called total. A necessary but not sufficient111For example, consider haskell_ -¿ undefined. condition for totality is exhaustiveness, the ability of the clauses to match any non-bottom input.

While there are a number of non-total functions in the standard library (e.g. haskelltail is not defined for the empty list partial), the general trend is to prefer total functions avoidPartial. This is because a programmer should ideally not need to familiarize themselves with the implementation of a function to discover on which parts of its corange it is total before using that function.

Since totality in a Turing-complete language is trivially undecidable and a reduction in power is often impractical DBLP:journals/jucs/ATurner04, it is useful to examine exhaustiveness as a proxy to proving partiality, as it reveals common programming errors. It is especially useful when adding new constructors to existing types because then the compiler will notify the programmer of functions that have become partial as a consequence.

Apart from regular patterns, Haskell also supports guards, arbitrary Boolean expressions that are evaluated when all patterns are matched. If the guard expression evaluates to haskellTrue, the clause is selected, otherwise matching falls through to the next clause.

This makes checking for exhaustiveness challenging, as illustrated by the following example:

[tabsize=4]haskell abs :: Int -¿ Int abs x — x ¡ 0 = -x — x ¿= 0 = x

Proving that haskellabs is exhaustively defined cannot be achieved by structural manipulation with type definitions alone. Semantic insight (knowledge that textx ¡ 0 —— x ¿= 0 holds for all textx) is necessary. While generally undecidable, guards tend to be simple, rendering a limited semantic analysis realistic.

Furthermore, a clause can also be unreachable because all the values that would be matched by it are matched by the preceding clauses, as demonstrated by the second clause of haskellf in the example below. Such clauses are called redundant and are another likely indicator of an error.

The lazy evaluation of Haskell means that a function may be evaluated when its result is to be matched against a pattern, even if not used later. This leads to unexpected subtleties in the semantics of pattern matching. For instance, consider functions haskellf and haskellf’:

haskell f :: Bool -¿ Bool -¿ Int f _True = 1 f True True = 2 f _False = 3

haskell f’ :: Bool -¿ Bool -¿ Int f’ _True = 1 f’ _False = 3

Although the second clause of haskellf is clearly redundant in the sense that it is never matched, removing it changes the semantics, as demonstrated by the difference in evaluation of the following expressions: haskell f undefined False haskell f’ undefined False

The evaluation of haskellf undefined False will terminate with an error, whereas haskellf’ undefined False will evaluate to haskell3 because evaluation of the first tuple element was not forced by the removed clause. This is an unusual behavior that the programmer might not have introduced intentionally.

Moreover, the example illustrates that reasoning about which parts of the input will be evaluated is non-trivial, especially for recursive data types. Analyzing the depth of evaluation with respect to input values is thus another related topic deserving attention.

Despite their practical value, until very recently, all of the aforementioned issues were addressed only in specific cases in the GHC marlow2004glasgow; DBLP:conf/icfp/KarachaliasSVJ15.

1.1 Our contributions

In this report, we present a static analysis tool that can give accurate warnings and information for the aforementioned properties. We implement the recent algorithm by Karachalias, Schrijvers, Vytiniotis, and Jones DBLP:conf/icfp/KarachaliasSVJ15 that enables us to to overcome the challenges posed by laziness and guards while also being easily extensible to GADT.

We simplify the existing work, provide complexity bounds, and extend it with a proof-of-concept term-constraint oracle based on an SMT solver that enables semantic insight into guards at compile time. To our best knowledge, ours is the first practical implementation of this technique. We show that a similar approach is applicable to virtually all languages with similar semantics and could improve the completeness of type checking.

We also introduce the concept of evaluatedness of a function. This is a comprehensive overview of how, when and how deeply each argument to a function will be evaluated during pattern matching.

The information our tool provides can thus be used to prevent defects, debug existing code, and help gain insight to those unfamiliar with a particular code base or Haskell in general.

Figure 1: The uncovered values function and helper functions.

denotes the empty vector;

is the set of existentially quantified type variables.

2 Background

There exists a body of literature on the analysis of pattern matching and related problems. Initially, the problem has been examined from an efficiency perspective, since knowing the covered and uncovered values can lead to generating more specialized, performant code Augustsson1985; DBLP:conf/icfp/FessantM01; wadler1987efficient. Follow-up work addressed the challenge of lazy semantics Maranget1992 as well as a limited analysis of redundancy thiemann1993avoiding.

Maranget Maranget2007 introduced an algorithm for exhaustiveness and redundancy checking for the ML language, heavily borrowing from the previous compilation techniques DBLP:conf/cc/Pettersson92. The algorithm was formulated in terms of matrices of values and includes a limited provisions for Haskell semantics and laziness while disregarding guards and GADT.

Mitchel and Runciman Mitchell2008 gave a more sophisticated analysis for Haskell that captures all information as constraints, which enables them to precisely characterize the values, although solving these constraints has proven to be challenging.

Recently, Karachalias et al. DBLP:conf/icfp/KarachaliasSVJ15 proposed a Haskell-specific algorithm that unifies all the previous work and accounts for laziness, guards, and GADT (see Section 3). Their independent parallel work resulted in an implementation of the algorithm that became a part of GHC 8 during the course of our work.

3 Algorithm

In this section, we describe our adaptation of the aforementioned algorithm by Karachalias et al. DBLP:conf/icfp/KarachaliasSVJ15. We start by providing a general intuition for the algorithm, with a precise description following in Section 3.2.

3.1 Intuition

For the sake of simplicity, consider a well-typed program with a finite number of types and a finite number of values for each of the types. In such a program, we can show that a function defined using clauses 1 through of arguments each is exhaustive. Given a definition


where through are the types of the respective arguments and is the return type, it is easy to show that the function is total.

In order to do so, we will keep track of two sets of values, and , for each clause, where is intuitively the set of values that will matched by the clause and is the set of values that have not been matched by this point. Starting with the set of all well-typed inputs , we compute the refined sets of values that have not been covered yet for clauses to by just removing all value tuple covered by the respective clause: , where is the set of values denoted by the pattern , as defined by denotational semantics of Haskell [DBLP:conf/icfp/KarachaliasSVJ15, Figure 4].

In order to check that is exhaustive, it suffices to check that is empty. Furthermore, we can define the set of covered values for each clause as . Checking whether a clause is redundant then amounts to showing that is empty.

While this approach is not feasible since all recursive types (e.g. lists) have infinitely many values, it can be refined by using value abstraction in the place of explicit sets of values. The presented intuition forms the basis of the algorithm proposed by Karachalias, et al. DBLP:conf/icfp/KarachaliasSVJ15.

Their algorithm takes advantage of the fact that all values of a given user-defined type are created using the data constructors of the type. The set of constructors provides a natural abstract domain for the set of concrete values, which in turn yields a compact representation for the sets of value tuples.

3.2 Outline

The actual algorithm follows the structure suggested in the previous section: it processes clauses in the order of appearance, gradually refining the abstraction of values. Specifically, it computes three sets for each of the clauses:

  • , the set of covered values. For these values, the right-hand side is evaluated.

  • , the set of uncovered values. These values will not be matched and will fall through to the next clause.

  • , the set of divergent values. Evaluating these values will fail, therefore neither this nor any subsequent clause will be matched.

The values in these sets are represented by triples of the form , where:

  • is a typing environment that keeps tracks of variables and type variables.

    For variables, it is a map from variables that occur in to types. We denote that has type by .

    For type variables, it simply records their existence in the context, written as .

    Let denote that a variable or a type variable does not occur in .

  • is a vector of patterns, where pattern can be

    • A variable, as in haskellid x = x.

    • A guard where is a pattern and is a Boolean expression.

    • A data constructor pattern , for example haskellJust a of haskellMaybe a and is a vector of patterns. Note that may be empty, e.g. for haskellFalse.

  • is a set of term and type equality constrains. The can be of form

    • , where is a variable and is an expression; and may be of any Haskell type.

    • , where is a variable and represents a divergent computation.

    • , a type equality.

The algorithm defines three functions , , that take a vector of patterns and an abstraction and compute the set of covered, uncovered, and divergent value abstraction respectively.

The first iteration starts with the most general value abstraction, . For clause , we compute the abstractions from the fall-through values and filter out those that are not plausible:


The denotes satisfiability over-approximation for set of constraints, i.e. , provided by an oracle, as discussed in section 3.4. The algorithm is independent of a particular oracle or its properties.

3.3 Uncovered values

We now describe in more detail. Covered and divergent values are analogous. Figure 1 gives its definition as a Haskell-style function from a pattern vector and a single value abstraction that represents one possible input to a set of possibly uncovered abstraction refinements.

The UNil rule states that for an empty pattern vector and an empty value value abstraction vector, there are no uncovered values. This is only useful for constants and to terminate recursion.

The UConCon rule applies when both the pattern and the abstraction are constructor patterns. When the constructors are equal,222Observe that they must be constructors of the same type, otherwise the program is not well-typed. then the arguments ( and ) are extracted and the computation continues on the flattened list. Mapping kcon then reconstructs the structure by applying to the appropriate number of resulting abstractions.

In case the constructors do not match, the value abstraction is definitely uncovered because it will not be matched, so it is returned unchanged.

In contrast, UConVar is matching a constructor pattern against a variable value abstraction. To find the possible values of for which will not match, each possible constructor of the type is substituted and the constraints are recorded, and a union of recursive solutions is taken. For all , the recursive computation will return a non-empty set due to UConCon. The recorded constraints can then filter out implausible abstractions.

UVarVar assumes equality of the two variables and computes the remainder recursively. This describes all the abstractions that are unmatched due to constructor inequality at a subsequent position.

The UGuard shows how the Boolean expression is added to the set of constraints and substituted by a fresh unique variable, which prevents aliasing of possibly unrelated values in the SMT constraints.

3.4 Oracle

An oracle is a Boolean function of value abstraction triples. For a triple , it serves to over-approximate whether the constraints in are satisfiable.

Since is consists of value equalities as well as type equalities, this can be implemented using a separate term-level constraint solver and a type-level constraint solver respectively, both of which must then over-approximate factual satisfiability.

In particular, a trivial oracle that declares any input to be satisfiable is sound with respect to the properties given in Section 3.7 and only decreases precision.

Note that for non-GADT or GADT with trivial constraints, all type equalities are trivial, i.e. of form , thus do not have any effect on precision.

3.5 Recommendations

Recommendations are generated from the results of the analysis as follows:

  • If the uncovered value abstraction set for the last clause is non-empty, then all its value abstractions represent missing clauses.

  • For every clause that has an empty set of covered value abstractions , there are no values that can be matched, thus the clause may be redundant. If the set of divergent value abstractions is also empty, then the clause is redundant, otherwise it has an inaccessible right-hand side.

3.6 Evaluatedness

After running the discussed analysis for a function with clauses, we obtain the analysis trace, a list of 3-tuples


containing the values abstractions corresponding to the definition in Section 3.2 after each of the iterations. The trace is used to compute the evaluatedness of the function. The value abstraction patterns in represent the divergent values of each of the arguments in clause . We then assert that each argument (or subexpression thereof) that gets a constraint of the form will be evaluated during pattern matching. This corresponds to an evaluation of .

The result of this post-processing is a trace of tuples of value abstraction vectors. The first element in this tuple specifies the form of the input and the second indicates how and which arguments are evaluated.

For example, in the case of our running example haskellf, the evaluatedness is as follows: raw f a b a: _b: b

raw f a False a: a False: False

It is to be read as: When haskellf is evaluated with input of the form haskellf a b, where haskella and haskellb are not further specified, only haskellb’s first constructor is evaluated during pattern matching. Further, when haskellf is evaluated with input of the form haskella False, haskella will be evaluated.

Not all divergent values need necessarily cause the pattern matching evaluation to diverge. Consider the following function that takes an arbitrary tuple as an argument:

haskell fst :: (a, b) -¿ a fst (x, _) = x The evaluatedness of this function is simple:

raw fst a a: a It states that the first argument to haskellfst is always evaluated to its first constructor. Indeed, when haskellfst undefined is evaluated, the evaluation diverges during pattern matching. It is, however, not evaluated beyond the first constructor, which in this case is the tuple constructor haskell(,). Conversely, when haskellfst (undefined, undefined) is evaluated, the tuple is matched but the arguments are not further evaluated and therefore the evaluation does not diverge during pattern matching.

3.7 Soundness properties

The over-approximation of satisfiability results in the following properties:

  • If there are no value abstractions in , then there exist no concrete values that are not covered. In other words, the non-exhaustiveness warnings are sound.

  • Similarly, is an over-approximation. Therefore any clauses reported as redundant indeed are redundant.

  • By the same token, reported inaccessible RHSs are indeed inaccessible.

3.8 Complexity

Algorithms for manipulating patterns are known to be predominantly exponential. For example, determining the set or redundant clauses has been shown to be NP-complete sekar1992adaptive.

It is easy to see that the algorithm runs in time where is number of clauses, is the maximum number of patterns occurring in any clause, and is the maximum number of constructors of any data type occurring among parameters.

This is because UConVar of Figure 1 establishes the upper bound on the number of value abstractions at any given point (maximum -fold increase relative to the input abstraction) and the abstraction size is constant in . This also implies space usage of and satisfiability queries of size .333Assuming the constraints are solved incrementally, as outlined in [DBLP:conf/icfp/KarachaliasSVJ15, Section 6.2].

4 Implementation

Our implementation is publicly available444 under an open source license. In this section, we discuss some additional technical considerations; Section 5 showcases the tool in practice.

4.1 Overview

Each function is analyzed separately. The clauses, guards, and other relevant information is extracted directly from the function’s AST; no interaction with the GHC interface is required. Minimal desugaring is performed (see Section 4.2).

This AST passed to the main analysis algorithm, which produces the analysis trace, which is not yet filtered using at this point. Each value abstraction in the trace is fed into the oracle (Section 4.3) to query satisfiability of the recorded constraints. This gives the final analysis result that contains plausible value abstractions only. All the warnings are generated from the filtered trace; the initial AST is used to augment the output so that it closely resembles the input source code (Section 3.5).

Since GHC interfaces are not required for the structural analysis or the guard exhaustiveness issues we focus on, we avoid the laborious integration with GHC altogether. This entails some limitations on the language features we support. In particular, without an access to the GHC type-constraint solver DBLP:conf/icfp/SchrijversJSV09, we can only provide a rudimentary GADT support.

4.2 Desugaring and special types

Throughout preceding sections, we have assumed that all types are defined uniformly using standard definition of constructors. Since our tool operates predominantly on the AST level, built-in types and values that require special syntactical support have to be addressed before the analysis.

For lists, we simply define the empty list constructor haskell[] :: a -¿ [a] and the infix concatenation constructor haskell(:) :: a -¿ [a] -¿ [a] A list of the form haskell[x, y, …, z] is then translated into haskellx:(y:(…:(z:[]))

In the same manner, tuples are defined as using a single constructor

haskell(, … ,) :: a -¿ … -¿ z -¿ (a, …, z)

So as to increase clarity, both tuples and lists are translated into their original syntactic forms before output.

Integers and other numerals are also a challenge, since they conceptually have an infinite number of constructors.555Haskell integers are implemented using GMP arbitrary precision arithmetic. Conceptually, however, haskelldata Integer = … — -1 — 0 — 1 — … is a valid data type. This limitation is overcome by replacing integer literals with a variable pattern and a guard pattern that asserts equality. For example, haskellg 42 = 1 would be translated to (e.g. haskellg x — x == 42 = 1).

Wildcard patterns (haskellf _) and user-provided guards are also subject to desugaring, as outlined by Karachalias [DBLP:conf/icfp/KarachaliasSVJ15, Figure 7].

Generating correct SMT formulae also involves adding reconciliation of type definitions. For example, values of numeric types such as haskellWord8 must be postulated to be within their range in order to capture this property within the formula.

4.3 Oracle

Term-constraints, in our simplified context, can only be variable equalities: , bottom assertions: or Boolean equalities: , where is any Boolean Haskell expression The Boolean equalities come from guards in the function under analysis. 666See the UGuard part of Figure 1.

4.3.1 Translating expressions

When querying the oracle, the set of constraints is checked for satisfiability. Boolean expressions as they appear in Haskell, however, code cannot simply be fed to a satisfiability solver.

An expression is first broken down into a simple abstract syntax tree. This tree is then translated into a SMT representation. There is support for certain functions like haskell&&, haskell+, haskellnot, and other Boolean and numerical functions that are supported by the solver, but not all Haskell expressions can be fully translated.

For example, a Boolean function like haskellisPrime :: Int -¿ Bool is not broken down at all. Instead, the sub-expression haskellisPrime x is treated as a Boolean variable by itself, because it can be either True or False depending only on haskellx.

4.3.2 Resolution of term-constraints

To judge satisfiability of term-constraints, first all variable constraints are resolved by replacing the variable by in all other constraints.

The next step handles all sets of term constraints with bottom assertions. Any bottom assertions makes the set of constraints unsatisfiable if the variable occurs in another constraint. This means we can entirely discard a set of term constraints as unsatisfiable when we find such a bottom assertion.

Once the bottom assertions are dealt with, the only expressions that are left are Boolean equalities. These are then passed directly into Z3 theorem prover DBLP:conf/tacas/MouraB08 via the SVB library sbv.

4.3.3 SMT results

The results from this solver are then converted into an over-approximation of satisfiability. Any unsatisfiability is interpreted as such, but any other result, whether it be “satisfiable”, “timeout” or “unknown”, is interpreted as “satisfiable”. Finally, the unsatisfiable value abstractions are removed from the analysis traces to complete the analysis.

5 Evaluation

As explained in a previous section, our implementation does not depend on the sophisticated GHC infrastructure, focusing on the term equalities instead. Since we also omit most language extensions and thus are unable to process most real-world code bases, we perform the evaluation on a qualitative basis.

For functions with non-GADT data types and no guards, our implementation gives exactly the same results as the GHC version 8.0.1 (released May 21, 2016) with -Wall flag, which we use as baseline for all subsequent comparisons. This already constitutes a major improvement over the GHC 7 implementation [DBLP:conf/icfp/KarachaliasSVJ15, Section 7, Table 1].

For functions with guards, we see an improvement in precisions, i.e. see warnings that GHC misses due to its coarse over-approximation.

5.1 Integer constraints

Consider the following (erroneous) implementation of the absolute value function:

haskell abs :: Int -¿ Int abs x — x ¡ 0 = - x — x ¿ 0 = x The code compiles without any warnings, even though haskellabs 0 is undefined. Our tool indeed does detect the non-exhaustive definition and provides a counterexample:

The patterns may not be exhaustive, the following
clauses are missing:
abs x
x = 0 :: Integer

5.2 Boolean constraints

In the same manner, our tool also improves precision for Boolean guards. Consider the following example of a redundant guard: haskell bguard :: Bool -¿ Int bguard x — x = 0 — not x = 1 — otherwise = 2 – redundant Even though is unsatisfiable, the GHC solver cannot reveal the inconsistency (it can only discover inconsistencies of the form ), thus failing to report the redundancy, whereas our tool reports the following recommendation.

raw The following clause is redundant: bguard x — otherwise

5.3 Mixed constraints

Apart from from improving the precision for integers and Booleans, our approximation of other Haskell fragments can improve precision when guards contain e.g. function applications.

For example, the following function guards contain an unknown function haskellisPrime: haskell isPrimeAndSmall :: Int -¿ Bool isPrimeAndSmall x — isPrime x && x ¡ 10 = True — not (isPrime x) = False Nevertheless, we can still show that the definition is not exhaustive by treating haskellisPrime x as a symbolic expression and give a counterexample: raw The patterns may not be exhaustive, the following clauses are missing: isPrimeAndSmall  a Constraints:  f == False  f == not (isPrime x)  c == False  c == isPrime x  a ¡ 10

Satisfiable. Model: […]

5.4 Future work

A surprising but straightfoward applications of our work lies in increasing the typechecking precision in languages that enforce totality constraints in a sound but incomplete way. For example, consider the following Rust DBLP:conf/sigada/MatsakisK14 implementation of the signun function:

rust fn sgn(x: i32) -¿ i32 match x y if y ¡ 0 =¿ -1, y if y == 0 =¿ 0, y if y ¿ 0 =¿ 1,

The Rust compiler777As of version 1.23. will refuse this function as the pattern is possibly incomplete. Using the analysis techniques we present, such imprecisions can be eliminated up to the level afforded by the oracle. Similarly, the value-level constrains we generate could also be useful during program optimization as it is reasonable to expect that they are more precise than commonly used dataflow analyses.

The next step in extending the analysis of constraints would be to also fully process guards that are defined in terms of functions and other data types.

For functions, it remains unclear whether re-formulation in e.g. uninterpreted functions theory is feasible. In particular, all functions used in the constraints would have to be total, which cannot be enforced in Haskell as of now, but the area is a subject of active research liquid.

Proving properties of general data structures is, within a limited scope, viable. Zeno DBLP:conf/tacas/SonnexDE12 and HipSpec DBLP:conf/cade/ClaessenJRS12 have demonstrated implementation of the concept for Haskell, but both are no longer maintained and do not support the current language ecosystem.

Finally, with the shift towards dependent typing, many properties will become a part of the type system and a significant portion of the work might thus be offloaded to the type-level constraints solver.

The authors would like to thank Tom Schrijvers for the supplementary material to his paper as well as his encouragement.

=0mu plus 1mu