--- title: "`mist`: Artifact for the ECOOP21 Paper _Refinements of Futures Past_" abstract: | `mist` is a tiny language for teaching and experimenting with refinement types, in the style of [LiquidHaskell](https://github.com/ucsd-progsys/liquidhaskell). We use it as a platform for experimenting with and as a demonstration of implicit refinement types as presented in the ECOOP21 paper _Refinements of Futures Past: Higher-Order Specification with Implicit Refinement Types_. We start with the parser and AST we use to teach our undergradute compilers class, and layer upon it a refinement type checker directly translated from the typing rules presented in that paper, which produces constraints that are solved with the `liquid-fixpoint` horn clause solver. We present source code and binaries for `mist` in a container image that includes installations of the competing tools we compare to. --- # Initial build, install, and running all tests You can use the Docker image or install `mist` manually. The Docker image also includes the tools we compare against. ## Using `docker` > **Windows and Mac users:** Make sure your docker container has at least 4GB of RAM The following command will download an image containing `mist`, `fstar`, and `mochi`, run the full `mist` test suite, and then drop you into an interactive shell at the root of the `mist` code repository. $ docker run -it atondwal/mist If you want to skip the test suite, instead run $ docker run -it atondwal/mist /bin/bash You can then (re)run all of the tests in the `tests/` directory (perhaps after editing some) at any time by running $ stack test ### Juggling containers You can use `docker ps` to see the running container and open another shell to it using `docker exec`, e.g.: $ docker ps CONTAINER ID IMAGE STATUS NAMES 696b2221e3ad atondwal/mist Up 45 seconds vibrant_leavitt $ docker exec -it vibrant_leavitt bash ecoop21@696b2221e3ad:~/mist$ You can use `docker start` to restart exited containers $ docker ps -a CONTAINER ID IMAGE STATUS NAMES 696b2221e3ad atondwal/mist Exited (137) 5 seconds ago vibrant_leavitt $ docker start vibrant_leavitt vibrant_leavitt $ docker exec -it vibrant_leavitt bash ecoop21@696b2221e3ad:~/mist$ ## Manually You'll need git, [z3 version 4.8.10](https://github.com/Z3Prover/z3/releases), and [stack](https://docs.haskellstack.org/en/stable/README/). $ git clone -b ecoop21 --recursive https://github.com/ucsd-progsys/mist $ cd mist $ stack install $ export PATH=$HOME/.local/bin/:$PATH You can then run the full `mist` test suite (which is located in the `tests/` directory). $ stack test # Running specific tests You can run a specific test by calling mist on the test file, e.g. $ mist tests/pos/incrState.hs If you're using the docker image, you can also run tests for `fstar`: $ fstar fstar-tests/incrState.fst # Benchmarks from the paper Here's a table of where you can find each of the tests described in the paper: | Name | Mist test (tests/pos/) | Mochi (mochi-tests/) | Fstar (fstar-tests/) | | ------------- | ---------------------- | -------------------- | -------------------- | | incr | [incr00.hs](tests/pos/incr00.hs) | [incr00.ml](mochi-tests/incr00.ml) | [incr.fst](fstar-tests/incr.fst) | | sum | [sum.hs](tests/pos/sum.hs) | [sum.ml](mochi-tests/sum.ml) | [sum.fst](fstar-tests/sum.fst) | | repeat | [repeat.hs](tests/pos/repeat.hs) | [repeat.ml](mochi-tests/repeat.ml) | x | | d2 | [mochi-app-lin-ord2.hs](tests/pos/mochi-app-lin-ord2.hs) | [d2.ml](mochi-tests/d2.ml) | [mochi-d2.fst](fstar-tests/mochi-d2.fst) | | | | | | | incrState | [incrStatePoly.hs](tests/pos/incrStatePoly.hs) | [incrState.ml](mochi-tests/incrState.ml) | [incrState.fst](fstar-tests/incrState.fst) | | accessControl | [acl.hs](tests/pos/acl.hs) | [acl.ml](mochi-tests/acl.ml) | [accessControl.fst](fstar-tests/accessControl.fst) | | tick | [tick-append.hs](tests/pos/tick-append.hs) | x | [tick.fst](fstar-tests/tick.fst) | | linearDSL | [linearTypes.hs](tests/pos/linearTypes.hs) | x | [linearDSL.fst](fstar-tests/linearDSL.fst) | | | | | | | pagination | [paginationTokens.hs](tests/pos/paginationTokens.hs) | x | x | | login | [idr_login.hs](tests/pos/idr_login.hs) | x | x | | twophase | [twoPhaseCommit.hs](tests/pos/twoPhaseCommit.hs) | x | x | | ticktock | [ticktock3.hs](tests/pos/ticktock3.hs) | x | x | | tcp | [tcp_client.hs](tests/pos/tcp_client.hs) | x | x | As in the paper, an `x` indicates that the specification cannot be directly expressed with that tool. Unfortunately, the version of MoCHi we compare against was made available as a [web demo](http://www.kb.is.s.u-tokyo.ac.jp/~uhiro/relcomp/) that is no longer functional. Since then, there has been a source release of MoCHi, but it does not support the `-relative-complete` verification mode that we compared against in our paper. We include a build of the latest version of MoCHi anyways, in case you want to play with it yourself and get an idea of how it works: $ mochi mochi-tests/incrState.ml # A quick tutorial in writing mist > **A note about UX:** We demonstrate the ability of our type system to localize error messages in this prototype, but when it comes to the parser, we favor an easy to modify and understand grammar over one that provides the best user experience. As such... When experimenting with `mist`, we recommend starting with one of the known working test cases, and then expanding on it to achieve the desired result, rather than starting from scratch in an empty text file. In this short tutorial we will take the same approach, starting from a minimal test case and building up to the pagination example from the ECOOP21 paper that demonstrates both implicit refinement function types and pair types. ## How to read this tutorial We recommend reading the pdf version of this tutorial as it is the easiest to read, but we also recommend keeping open a copy of the markdown source (`~/mist/README.md`) in your text editor as you follow along, as the markdown source includes the location of each snippet as range of lines in a test file, so you can open, edit, and rerun those tests yourself. We recommend running a continuous build in a terminal while you experiment with a mist file. For example, this sets up `entr` to run `mist` on a file every time it gets written (whether or not it's a `mist` file). ```{.console} $ find tests | entr mist /_ (in another window - see above for docker instructions) $ vim tests/.../mytest.hs ``` > Bits of syntax that are potential sources of confusion or frustration > are highlighted in grey boxes. If you're struggling to make your code parse, > checking to see if you've stepped on one of these Legos is a good place to start. ## Refinement Types We start from an extremely simple example that demonstrates the concrete semantics of mist's refinement type system. ```{include=tests/pos/Int00.hs .haskell .numberLines} twelve :: { v : Int | v == 12 } twelve = 12 ``` Here, we have a top-level binder for the constant `int`. Each top level binder includes a type signature (line 1), and a body (line 2). The body of `int` simply states that it's equal to the integer constant `12`. This type signature is a minimal example of a refinement type: we refine the base type `Int`, binding its values to `v`, and taking the quotient of this type by the proposition `v == 12`. This results in a singleton type that checks against the body of `twelve`. $ mist tests/pos/Int00.hs SAFE If we had used a different value in the type and body: ```{include=tests/neg/Int01.hs .haskell .numberLines} twelve :: { v : Int | v == 14 } twelve = 12 ``` We'd see a type error: ```{.console} $ mist tests/neg/Int01.hs Working 150% [=================================================================] Errors found! tests/neg/Int01.hs:2:7-9: Expected (VV##0 == 14) : 2| int = 12 ^^^ ``` ## Functions and polymorphism We can extend this to writing functions in `mist`: ```{include=tests/pos/Inc02.hs .haskell .numberLines startLine=1 endLine=5} incr :: x:Int -> {v:Int | v == x + 1} incr = \x -> x + 1 moo :: {v:Int | v == 8} moo = incr 7 ``` This program checks that `incr`ementing 7 results in 8. Here, the binder `x:Int` binds `x` in the type on the right-hand side of `->`. Similarly, at the value level, `\` denotes a lambda. Functions can also be polymorphic: ```{include=tests/pos/Inc02.hs .haskell .numberLines startLine=7 endLine=11} id :: rforall a. {v:a | True} -> {v:a | True} id = \x -> x bar :: {v:Int | v == 8} bar = incr (id 7) ``` $ mist tests/pos/Inc02.hs SAFE > All function applications that are not directly under a binder or a function abstraction should be enclosed in parentheses. Here, `rforall` denotes that the function `id` is _refinement polymorphic_ in the type variable `a`. That is, `a` stands in for any _refined_ type, so we know that the result of applying `id` to any value will always result in a value of the same refinement type; i.e. one for which all the same propositions are true. The only function of this type is `id`. Later we will also see `forall`, which allows functions to be polymorphic over base types. ## Implicit function types We're ready for our first example of a feature introduced in this paper! We write an implicit function type the same way as a normal function, but using the squiggly arrow `~>` instead of the straight arrow `->`: ```{include=tests/pos/incr00.hs .haskell .numberLines} incr :: n:Int ~> (Int -> { v : Int | v == n }) -> { v : Int | v == n + 1 } incr = \f -> (f 0) + 1 test1 :: { v : Int | v == 11 } test1 = incr (\x -> 10) test2 :: m:Int -> { v : Int | v == m+1 } test2 = \mv -> incr (\x -> mv) ``` $ mist tests/pos/incr00.hs SAFE > Note the parentheses around `(f 0)` --- there are no precedence rules for infix primitives. Given a constant function, `incr` increments the result. This is straightforward at the value level, but encoding it at the type level requires the use of implicit parameters. Here, `n` in bound at the type level, but has no corresponding binder at the value level in the surface syntax. The body of the function must typecheck for all values of `n`, but each call to the function need only be valid for some particular choice of `n`. `n` is picked at the call site by the implicit instantiation algorithm for refinement types described in the paper, such that the function application typechecks. Here, for the call to `incr` on line 5 inside `test1`, `n` takes on the value 10, and on line 8, it takes on the value `mv`. ## Datatypes, axioms, and measures Mist supports user-defined datatypes by axiomatising their constructors. In this section we're going to demonstrate specification and verification with the `List` datatype, which in Haskell one might write: ```{.haskell} data List a = Nil | Cons a (List a) ``` ### Datatypes In mist, `List a` is spelled `List >a` There are two things of note here: 1. As in Haskell, Mist datatypes are written in TitleCamelCase. 2. Unlike Haskell, datatypes carry _variance annotations_ with them that tell you if they're co- or contra-variant in a given argument. Having these around can be helpful when you're debugging or reading code with complex subtyping relationships. Here, the variance annotation `>` indicates that `a` appears covariantly in `List` (that is, `List` contains things that are subtypes of `a`). If it appeared contravariantly, we would have written `List a` behaves like a `List` that might produce `a`s when it is used up, whereas a `List `: Just as you can use a function that _returns_ any subtype of the type you need, and that _accepts_ any supertype of the arguments you have, if you're a type variable on the pointy end of the variance annotation (or function arrow) you're a covariant type variable, and if you're on the other end you're contravariant. If you try to pass a `List >a` as a `List a exFalsoQuodlibet = ... ``` Whatever we put for ... is taken to be the witness of the axiom, and executed when the axiom is used in code that is run. > You need to provide a body for every binding, axioms or otherwise. To use the `List` datatype, we need constructors, and projections from these constructor (or induction principles, but let's keep it simple for the tutorial). To introduce axioms for each of these, we write something like ```{.haskell} nil as forall a. List >a nil = ... cons as forall a. a -> List >a -> List >a cons = ... first as forall a. List >a -> a first = ... rest as forall a. List >a -> List >a rest = ... ``` where `...` can be e.g. Boehm-Beraraducci (1985) encoding of constructors and projection operators, but since we're focused on testing the typechecker here, we generally set them equal to 0 as the witnesses to axioms don't matter so far as the typechecker is concerned. We can use axiomatized constructors to define a datatype `Lin` which is the type of terms of a linear DSL. Here, we use `Set` primitives. ```{include=tests/pos/linearTypes.hs .haskell .numberLines startLine=4 endLine=4} var as x:Int -> (Lin >{v:Set >Int | v = setPlus emptySet x}) ``` `var` constructs a term that is a variable mention. It checks that the variable is in the environment. $\frac{x \in \Gamma}{\Gamma \vdash \texttt{var } x}$ ```{include=tests/pos/linearTypes.hs .haskell .numberLines startLine=7 endLine=10} fun as env:(Set >Int) ~> n:{v:Int | (v ∈ env) ≠ True} -> (Lin >{v:Set >Int | v = setPlus env n}) -> (Lin >{v:Set >Int | v = env}) ``` `fun` constructs a lambda term, while checking that the variable it binds is fresh. $\frac{x\not\in\Gamma \quad \Gamma \cup \{x\} \vdash e}{\Gamma \vdash \texttt{fun } x\, e}$ > It may not always be obvious when you need parenthesis around type constructor applications. Some rules of thumb: parenthesize them on both sides of `->`, but in the left-hand side of a refinement type (the binder), "`(`" may not follow "`:`" > Note that the not-equals operator is the unicode symbol, not a multi-character sigil --- infix operators in Mist are single characters. ```{include=tests/pos/linearTypes.hs .haskell .numberLines startLine=13 endLine=16} app as env1:(Set >Int) ~> env2:{v:Set >Int | env1 ∩ v = emptySet} ~> (Lin >{v:Set >Int | v = env1}) -> (Lin >{v:Set >Int | v = env2}) -> (Lin >{v:Set >Int | v = env1 ∪ env2}) ``` `app` applies a function to a value, checking that no variable is used more than once in either argument. $\frac{\Gamma_1 \cap \Gamma_2 = \emptyset\quad \Gamma_2 \vdash e \quad \Gamma_1 \vdash f}{\Gamma_1 \cup \Gamma_2\vdash \texttt{app } f\, e}$ ```{include=tests/pos/linearTypes.hs .haskell .numberLines startLine=19 endLine=19} typecheck as (Lin >{v:Set >Int | v = emptySet}) -> (Lin >(Set >Int)) ``` `typecheck` simply checks that a term is closed $\frac{\emptyset\vdash e}{\vdash e}$ ```{include=tests/pos/linearTypes.hs .haskell .numberLines startLine=25 endLine=100} program2 :: Lin >(Set >Int) program2 = typecheck (fun 1 (fun 2 (app (var 1) (var 2)))) ``` ```{.console} $ mist tests/pos/linearTypes.hs SAFE ``` ## State We can define a State Monad datatype! See section 2.3 of the paper for more explanation. `put` takes a world (called `wp`), `put` updates the state to one where the state of the world is now `wp`. ```{include=tests/pos/incrState.hs .haskell .numberLines startLine=20 endLine=20} put as wp:Int -> ST {p:Int|p==wp} >Unit ``` `get` takes a boolean and ignores it, then leaves the state of the world unchanged, but returns its value in the `ST` monad. ```{include=tests/pos/incrState.hs .haskell .numberLines startLine=17 endLine=17} get as wg:Int ~> Bool -> ST <{gi:Int|gi==wg} >{go:Int|go==wg} >{gr:Int|gr==wg} ``` And then we have the standard monadic interface: ```{include=tests/pos/incrState.hs .haskell .numberLines startLine=1 endLine=2} -- Monadic Interface ret as rforall a. wr:Int ~> x:a -> ST <{ri:Int|ri==wr} >{ro:Int|ro==wr} >a ``` ```{include=tests/pos/incrState.hs .haskell .numberLines startLine=11 endLine=14} bind as rforall a, b. w1:Int ~> w2:Int ~> w3:Int ~> (ST <{v:Int|v==w1} >{v:Int|v==w2} >a) -> (unused:a -> ST <{v:Int|v==w2} >{v:Int|v==w3} >b) -> ST <{v:Int|v==w1} >{v:Int|v==w3} >b ``` Using this, we can verify a more stateful version of the incr example from before. ```{include=tests/pos/incrState.hs .haskell .numberLines startLine=24 endLine=25} incr :: i:Int ~> ST <{v:Int|i==v} >{w:Int|w==i+1} >Unit incr = bind (get True) (\x -> put (x+1)) ``` ```{.console} $ mist tests/pos/incrState.hs SAFE ``` Going forward, however, we're going to use a more polymorphic definition of state: ```{include=tests/pos/paginationTokens.hs .haskell .numberLines startLine=13 endLine=32} bind :: rforall a, b, p, q, r. ST

q >a -> (x:a -> ST r >b) -> ST

r >b bind = undefined pure :: rforall a, p. x:a -> ST

p >a pure = undefined thenn :: rforall a, b, p, q, r. ST

q >a -> ST r >b -> ST

r >b thenn = \f g -> bind f (\underscore -> g) fmap :: rforall a, b, p, q. (underscore:a -> b) -> ST

q >a -> ST

q >b fmap = \f x -> bind x (\xx -> pure (f xx)) ``` > If a function type signature is failing to parse, try assigning a name to the argument (e.g. `x:Int ->` instead of `Int ->`). ## Implicit pair types Next, we demonstrate how to use another core feature unique to Mist: implicit pair types as described in the paper. In the paper the syntax is $[n:Int].\text{---}$, but in the implementation, we use `exists n. -`. Consider iterating through an infinite stream of token handlers. The API for getting the next token is: ```{include=tests/pos/paginationTokens.hs .haskell .numberLines startLine=37 endLine=42} nextPage as token:{v: Int | v ≠ done} -> (exists tok:Int. (ST <{v:Int | v = token} >{v:Int | (v = tok) /\ (v ≠ token)} >{v:Int | v = tok})) ``` > Remember to parenthesize both sides of conjunctions (`/\`)! That is, given a token, `nextPage` give you a state action where it picks a new token that's not equal to the old token, and updates the state of the world to reflect the new token. ```{include=tests/pos/paginationTokens.hs .haskell .numberLines startLine=52 endLine=57} client :: token:Int -> (ST <{v:Int | v = token} >{v:Int | v = done} >Int) client = \token -> if token == done then pure 1 else bind (nextPage token) (\tok -> client tok) ``` $ mist tests/pos/paginationTokens.hs SAFE And that concludes our short tutorial on `mist`. Go forth and verify! \appendix # Appendix : Measures But these `List` constructors are all a bit boring --- what good are user datatypes if we can't say anything about them at the type level?! Until now, we've only been able to form refinements out of variables and primitive functions such as `+` and `∩` on special types such as `Int` and `Set`. We use (purely) refinement-level functions called measures (Vazou et al, ICFP 2014) to extend the language of refinements and enrich the types of our constructors. ```{include=tests/pos/recursion.hs .haskell .numberLines startLine=1 endLine=1} measure mNil :: List [>Int] -> Bool ``` > Measures use a different syntax for types --- type applications have a list of parameters in `[`square`,` brackets`,` separated`,` by`,` commas`]`, unlike applications of type constructors that produce refinement types, which use the usual space-separated syntax. This declares a measure `mNil` that takes a `List` of `Int`s and returns a `Bool`. Measure have unrefined types. > If you get an error message about free vars that implicates the start of the file, you probably tried to use a measure you didn't declare. This will cause the solver print a list of unbound measures and crash with some mumbo-jumbo about `--prune-unsorted`. Measures always go at the top of the file. We can use these measures in constructor axioms to effectively define structurally recursive functions over a datatype. ```{include=tests/pos/recursion.hs .haskell .numberLines startLine=10 endLine=10} nil as {v: List >Int | (mNil v) /\ (mLength v = 0) /\ (not (mCons v))} ``` ```{include=tests/pos/recursion.hs .haskell .numberLines startLine=13 endLine=14} cons as x:Int -> xs:(List >Int) -> {v: List >Int | (mCons v) /\ (mLength v = mLength xs + 1) /\ (not (mNil v))} ``` ```{include=tests/pos/recursion.hs .haskell .numberLines startLine=17 endLine=17} first as {v: List >Int | mCons v} -> Int ``` ```{include=tests/pos/recursion.hs .haskell .numberLines startLine=20 endLine=22} rest as rs:{v: List >Int | mCons v} -> {v: List >Int | mLength v + 1 == mLength rs } ``` and we can then use them in verification! ```{include=tests/pos/recursion.hs .haskell .numberLines startLine=24 endLine=100} append :: xs:(List >Int) -> ys:(List >Int) -> {v: List >Int | mLength v = (mLength xs) + (mLength ys)} append = \xs -> \ys -> if empty xs then ys else cons (first xs) (append (rest xs) ys) ``` ```{.console} $ mist tests/pos/recursion.hs SAFE ```