Ready, Set, Verify! Applying hs-to-coq to real-world Haskell code

Good tools can bring mechanical verification to programs written in mainstream functional languages. We use hs-to-coq to translate significant portions of Haskell's containers library into Coq, and verify it against specifications that we derive from a variety of sources including type class laws, the library's test suite, and interfaces from Coq's standard library. Our work shows that it is feasible to verify mature, widely-used, highly optimized, and unmodified Haskell code. We also learn more about the theory of weight-balanced trees, extend hs-to-coq to handle partiality, and -- since we found no bugs -- attest to the superb quality of well-tested functional code.


page 1

page 2

page 3

page 4


Active Learning of Points-To Specifications

When analyzing programs, large libraries pose significant challenges to ...

Auto-active Verification of Graph Algorithms, Written in OCaml

Functional programming offers the perfect ground for building correct-by...

Verifying Verified Code

A recent case study from AWS by Chong et al. proposes an effective metho...

Embracing a mechanized formalization gap

If a code base is so big and complicated that complete mechanical verifi...

Total Haskell is Reasonable Coq

We would like to use the Coq proof assistant to mechanically verify prop...

Online Verification of Deep Neural Networks under Domain or Weight Shift

Although neural networks are widely used, it remains challenging to form...

Verifying Probabilistic Specifications with Functional Lagrangians

We propose a general framework for verifying input-output specifications...

1. Introduction

What would it take to tempt functional programmers to verify their code?

Certainly, better tools would help. We see that functional programmers who use dependently-typed languages or proof assistants, such as Coq (Coq:manual), Agda (agda), Idris (idris), and Isabelle (isabelle), do verify their code, since their tools allow it. However, adopting these platforms means rewriting everything from scratch. What about the verification of existing code, such as libraries written in mature languages like Haskell?

Haskell programmers can reach for LiquidHaskell (Vazou:2014:RTH:2628136.2628161) which smoothly integrates the expressive power of refinement types with Haskell, using SMT solvers for fully automatic verification. But some verification endeavors require the full strength of a mature interactive proof assistant like Coq. The hs-to-coq tool, developed by hs-to-coq-cpp, translates Haskell types, functions and type classes into equivalent Coq code – a form of shallow embedding – which can be verified just like normal Coq definitions.

But can this approach be used for more than the small, textbook-sized examples it has been applied to so far? Yes, it can! In this work, we use hs-to-coq to translate and verify the two set data structures from Haskell’s containers package.111Specifically, we target version, which was released on January 22, 2018 and was the most recent release of this library at the time of publication; it is available at This codebase is not a toy. It is decades old, highly tuned for performance, type-class polymorphic, and implemented in terms of low-level features like bit manipulation operators and raw pointer equality. It is also an integral part of the Haskell ecosystem. We make the following contributions:

  • We demonstrate that hs-to-coq is suitable for the verification of unmodified, real-world Haskell libraries. By “real-world”, we mean code that is up-to-date, in common use, and optimized for performance. In Section 2 we describe the containers library in more detail and discuss why it fits this description.

  • We present a case study not just of verifying a popular Haskell library, but also of developing a good specification of that library. This process is worth consideration because it is not at all obvious what we mean when we say that we have “verified” a library. LABEL:sec:specs discusses the techniques that we have used to construct a rich, two-sided specification; one that draws from diverse, cross-validated sources and yet is suitable for verification.

  • We extend hs-to-coq and its associated standard libraries to support our verification goal. In particular, in LABEL:sec:translating we describe the challenges that arise when translating the Data.Set and Data.IntSet modules, and our solutions. Notably, we drop the restriction in previous work (hs-to-coq-cpp)

    that the input of the translation must be intrinsically total. Instead, we show how to safely defer reasoning about incomplete pattern matching and potential nontermination to later stages of the verification process.

  • We increase confidence in the translation done of hs-to-coq. In one direction, properties of the Haskell test suite turn into Coq theorems that we prove. In the other direction, the translated code, when extracted back to Haskell, passes the original test suite.

  • We provide new implementation-agnostic insight into the verification of the weight-balanced tree data structure, as we describe in LABEL:sec:additional. In particular, we find the right precondition for the central balancing operations needed to verify the particular variant used in Data.Set.

Our work provides a rich specification for Haskell’s finite set libraries that is directly and mechanically connected to the current implementation. As a result, Haskell programmers can be assured that these libraries behave as expected. Of course, there is a limit to the assurances that we can provide through this sort of effort. We discuss the verification gap and other limitations of our approach in LABEL:sec:limitations.

We would like to have been able to claim the contribution of findings bugs in containers, but there simply were none. Still, our efforts resulted in improvements to the containers library. First, an insight during the verification process led to an optimization that makes the Data.Set.union function 4% faster. Second, we discovered an incompleteness in the specification of the validity checker used in the test suite.

The tangible artifacts of this work have been incorporated into the hs-to-coq distribution and are available as open source tools and libraries.222’t forget to submit the tarball

2. The containers library

We select the containers library for our verification efforts because it is a critical component of the Haskell ecosystem. With over 4000 publicly available Haskell packages using on containers, it is the third-most dependent-up package on the Haskell package repository Hackage, after base and bytestring.333

The containers library is both mature and highly optimized. It has existed for over a decade and has undergone many significant revisions in order to improve its performance. It contains seven container data structures, covering support for finite sets (Data.Set and Data.IntSet), finite maps (Data.Map and Data.IntMap), sequences (Data.Sequence), graphs (Data.Graph), and trees (Data.Tree). However most users of the containers library only use the map and set modules;444We calculated that 78% of the packages on Hackage that depend on containers use only sets and maps.moreover, the map modules are essentially analogues of the set modules. Therefore, we focus on Data.Set and Data.IntSet in this work.

2.1. Weight-balanced trees and big-endian Patricia trees

B@>l<@6@>l<@13@>l<@14@>l<@16@>l<@36@>l<@41@>l<@45@>l<@E@>l<@[B]-- ------------------------------------------------------------------ [E]
[B]-- Sets are size balanced trees [E]
[B]-- ------------------------------------------------------------------ [E]
[B]data Set a [13][13]= [16][16]Bin  {-# UNPACK #-}  !Size !a !(Set a) !(Set a) [E]
[B]   [13][13]| [16][16]Tip [E]
[B]type Size [13][13]= Int [E]
[B]-- | . Is the element in the set? [E]
[B]member :: Ord a => a -> Set a -> Bool [E]
[B]member = go [E]
[B]   [6][6]where [14][14]go !_ Tip = False [E]
[B]   [14][14]go x (Bin _ y l r) = [36][36]case compare x y of [E]
[B]   [36][36]   [41][41]LT [45][45]-> go x l [E]
[B]   [41][41]GT [45][45]-> go x r [E]
[B]   [41][41]EQ [45][45]-> True[E]

Figure 1. The Set data type and its membership function555From
All code listings in this paper are manually reformatted and may omit module names from fully qualified names.

The Data.Set module implements finite sets using weight-balanced binary search trees. The definition of the Set datatype in this module, along with its membership function, is given in Figure 1. These sets and operations are polymorphic over the element type and require only that this type is linearly ordered, as expressed by the Ord constraint on the member function. The member function descends the ordered search tree to determine whether it contains a particular element.

The Size component stored with the Bin constructor is used by the operations in the library to ensure that the tree stays balanced. The implementation maintains the balancing invariant

where and are the sizes of the left and right subtrees of a Bin constructor. This definition is based on the description by adams-tr, who modified the original weight-balanced tree proposed by nievergelt. Thanks to this balancing, operations such as insertion, membership testing, and deletion take time logarithmic in the size of the tree.

This type definition has been tweaked to improve the performance of the library. The annotations on the Bin data constructor instruct the compiler to unpack the size component, removing a level of indirection. The ! annotations indicate that all components should be strictly evaluated.

The Data.IntSet module also provides search trees, specialized to values of type Int to provide more efficient operations, especially union. This implementation is based on big-endian Patricia trees, as proposed in Morrison:1968’s work on PATRICIA (Morrison:1968) and described in a pure functional setting by okasakigill.

The definition of this data structure is shown in Figure 2. The core idea is to use the bits of the stored values to decide in which subtree of a node they should be placed. In a node Bin p m s1 s2, the mask m has exactly one bit set. All bits higher than the mask bit are equal in all elements of that node; they form the prefix p. The mask bit is the highest bit that is not shared by all elements. In particular, all elements in s1 have this bit cleared, while all elements in s2 have it set. When looking up a value x, the mask bit of x tells us into which branch to descend.

B@>l<@14@>c<@14E@l@15@>l<@17@>l<@E@>l<@[B]data IntSet [14][14]= [14E][17]Bin  {-# UNPACK #-}  !Prefix  {-# UNPACK #-}  !Mask !IntSet !IntSet [E]
[B]-- Invariant: Nil is never found as a child of Bin. [E]
[B]-- Invariant: The Mask is a power of 2. It is the largest bit position at [E]
[B]-- Invariant: which two elements of the set differ. [E]
[B]-- Invariant: Prefix is the common high-order bits that all elements share to [E]
[B]-- Invariant: the left of the Mask bit. [E]
[B]-- Invariant: In Bin prefix mask left right, left consists of the elements [E]
[B]-- Invariant: that don't have the mask bit set; right is all the elements [E]
[B]-- Invariant: that do. [E]
[B]   [14][14]| [14E][17]Tip  {-# UNPACK #-}  !Prefix  {-# UNPACK #-}  !BitMap [E]
[B]-- Invariant: The Prefix is zero for the last 5 (on 32 bit arches) or 6 bits [E]
[B]-- Invariant: (on 64 bit arches). The values of the set represented by a tip [E]
[B]-- Invariant: are the prefix plus the indices of the set bits in the bit map. [E]
[B]   [14][14]| [14E][17]Nil [E]
[B]-- A number stored in a set is stored as [E]
[B]-- * Prefix (all but last 5-6 bits) and [E]
[B]-- * BitMap (last 5-6 bits stored as a bitmask) [E]
[B]-- * Last 5-6 bits are called a Suffix. [E]
[B]type Prefix [15][15]= Int [E]
[B]type Mask   [15][15]= Int [E]
[B]type BitMap [15][15]= Word [E]
[B]type Key   [15][15]= Int[E]

Figure 2. The IntSet data type666From

Instead of storing a single value at the leaf of the tree, this implementation improves time and space performance by storing the membership information of consecutive numbers as the bits of a machine-word-sized bitmap in the Tip constructor.

The Nil constructor is the only way to represent an empty tree, and will never occur as the child of a Bin constructor. Every well-formed IntSet is either made of Bins and Tips, or a single Nil.

2.2. A history of performance tuning

The history of the Data.Set module can be traced back to 2004, when a number of competing search tree implementations were debated in the “Tree Wars” thread on the Haskell libraries mailing list. Benchmarks showed that Daan Leijen’s implementation had the best performance, and it was added to containers in 2005 as Data.Set.777

In (straka), Milan Straka thoroughly evaluated the performance of the containers library and implemented a number of performance tweaks (straka). This change888 replaced a fairly readable balance and several small and descriptive helper functions with a single dense block of code. A later change999 by straka created two copies of this scary-looking balance, each specialized and optimized for different preconditions.

adams-tr describes two algorithms for union, intersection, and difference: “hedge union” and “divide and conquer”. Originally containers used the former, but in 2016 its maintainers switched to the latter,101010 again based on performance measurement.

The module Data.IntSet (and Data.IntMap) has been around even longer. okasakigill mention in their (okasakigill) paper (okasakigill) that GHC had already made use of IntSet and IntMap for several years. In 2011, the Data.IntSet module was re-written to use machine-words as bit maps in the leaves of the tree, as discussed at the end of Section 2.1.111111 This moved the containers library further away from the literature on Patricia trees and introduced a fair amount of low-level bit twiddling operations (e.g., highestBitMask, lowestBitMask, and revNat).

3. Overview of our verification approach

In order to verify Set and IntSet, we use hs-to-coq to translate the unmodified Haskell modules to Gallina and then use Coq to verify the translated code. For example, consider the excerpt of the implementation of Set in Figure 1. The hs-to-coq tool translates this input to the following Coq definitions.121212In the file examples/containers/lib/Data/Set/Internal.v The strictness and unpacking annotations are ignored, as they do not make sense in Coq, and the type name Set is renamed to Set_ to avoid clashing with the Coq keyword.

B@>l<@6@>l<@10@>l<@11@>c<@11E@l@15@>l<@20@>l<@41@>l<@46@>l<@52@>l<@E@>l<@[B]Definition Size := GHC.Num.Int%type. [E]
[B]Inductive Set_ a : Type [E]
[B]   [6][6]:= [10][10]Bin : Size -> a -> (Set_ a) -> (Set_ a) -> Set_ a [E]
[B]   [6][6]| [10][10]Tip : Set_ a. [E]
[B]Definition member {a} `{GHC.Base.Ord a} : a -> Set_ a -> bool := [E]
[B]   [6][6]letfix go arg_0__ arg_1__ [E]
[B]   [6][6]   [11][11]:= [11E][15]match arg_0__, arg_1__ with [E]
[B]   [15][15]   [20][20]| _, Tip => false [E]
[B]   [20][20]| x, Bin _ y l r => [41][41]match x y with [E]
[B]   [41][41]   [46][46]| Lt [52][52]=> go x l [E]
[B]   [46][46]| Gt [52][52]=> go x r [E]
[B]   [46][46]| Eq [52][52]=> true [E]
[B]   [41][41]end [E]
[B]   [15][15]end [E]
[B]   [6][6]in go.[E]

These definitions depend on hs-to-coq’s pre-existing translated version of GHC’s standard library base. Here, we use the existing translation of Haskell’s Int type, the Ord type class, and Ord’s compare method.

We carry out this translation for the Set and IntSet along with their attendant functions, and then verify the resulting Gallina code. In LABEL:sec:specs we discuss the properties that we prove about the two data structures.

To further test that the translation from Haskell to Coq, we also used Coq’s extraction mechanism to translate the generated Gallina code, like that seen above, back to Haskell. This process converts the implicitly-passed type class dictionaries to ordinary explicitly-passed function arguments, but otherwise preserves the structure of the code. By providing an interface that restores the type-class based types, we can run the original containers test suite against this code. This process helps us check that hs-to-coq preserves the semantics of the original Haskell program during the translation process.