02157FunctionalProgram-

ming

Michael R. Hansen02157 Functional ProgrammingProperty-based testing: An appertizer

Michael R. Hansen

1 DTU Compute, Technical University of Denmark Property-based testing: An appertizer MRH 07/11/2019

02157FunctionalProgram-

ming

Michael R. Hansen

Overview

QuickCheck: A Lightweight Tool for Random Testing of HaskellPrograms, Claessen and Hughes, 2000

• Random generation of values of arbitrary types• Properties are expressed as Boolean-valued functions

let rec sort xs = .....let rec ordered xs = ...

// Test that: for all lists xs: ordered(sort xs)let sortProp (xs: int list) = ordered(sort xs)

let _ = Check.Quick sortPropOk, passed 100 tests.

The tool has been ported to many languages. We look at FsCheckfor the .Net platform

Consult https://fscheck.github.io/FsCheck/ concerninginstallation and resources

2 DTU Compute, Technical University of Denmark Property-based testing: An appertizer MRH 07/11/2019

02157FunctionalProgram-

ming

Michael R. Hansen

Testing for correctness wrt. a reference model (I)

#r ".......FsCheck.dll"open FsCheck

let rec sumA xs acc = match xs with| [] -> 0| x::xs -> sumA xs (x+acc);;

Correctness property wrt. the built-in function: List.sum:

for all xs: List.sum xs = sumA xs 0

let sumRefProp xs = List.sum xs = sumA xs 0;;let _ = Check.Quick sumRefProp;;Falsifiable, after 2 tests (2 shrinks) (StdGen ...... :Original:[-2; -1]Shrunk:[1]

• uses built-in generators for lists• tool provides a short counterexample

3 DTU Compute, Technical University of Denmark Property-based testing: An appertizer MRH 07/11/2019

02157FunctionalProgram-

ming

Michael R. Hansen

Testing for correctness wrt. a reference model (II)

let rec sumA xs acc = match xs with| [] -> acc| x::xs -> sumA xs (x+acc);;

Correctness property wrt. the built-in function: List.sum:

for all xs: List.sum xs = sumA xs 0

let sumRefProp xs = List.sum xs = sumA xs 0;;let _ = Check.Quick sumRefProp;;Ok, passed 100 tests.

• default is 100 random tests• can be configured

4 DTU Compute, Technical University of Denmark Property-based testing: An appertizer MRH 07/11/2019

02157FunctionalProgram-

ming

Michael R. Hansen

Testing for correctness wrt. a reference model (III)

Test is exposed using Check.Verbose as follows:

let sumRefProp xs = List.sum xs = sumA xs 0;;let _ = Check.Verbose sumRefProp;;

0:[-2].....99:[-1; 0; -1; -1; 2; 1; -1; 0; 0; 5; -1; 1; -1; 0; 0; -1; 2; -1; -2; 0; 1; 0; -1;1; -1; 1; -1; 0; -1; -1; -1; -1; 1; -1; 1; 1; 1; 0; -2; 1; 2; -2; 1; 0; 0; -2;1; 0; -1; 0; -1; -1; -2; 2; 0; 1; -1; -1; 1; 1; 0; 0; -1; 0; 1; 0; 0; 1; 1; 1;0]Ok, passed 100 tests.

5 DTU Compute, Technical University of Denmark Property-based testing: An appertizer MRH 07/11/2019

02157FunctionalProgram-

ming

Michael R. Hansen

Checking invariants: Search trees

Type for search trees with associated insertion function:

type Tree = | Lf | Br of Tree*int*Tree;;

let rec insert i t =match t with| Lf -> Br(Lf,i,Lf)| Br(t1,j,t2) -> match compare i j with

| 0 -> t| n when n<0 -> Br(insert i t1 ,j,t2)| _ -> Br(t1,j, insert i t2);;

The function insert should preserve the search tree invariant:

for all trees t, integers 3: invariant(t) implies invariant(insert i t)

where invariant is a Boolean-valued function see handout

6 DTU Compute, Technical University of Denmark Property-based testing: An appertizer MRH 07/11/2019

02157FunctionalProgram-

ming

Michael R. Hansen

Testing invariants: Search trees

The property can be checked as follows:

let insertInvProp i t = not (invariant t)|| invariant(insert i t);;

let _ = Check.Quick insertInvProp;;Ok, passed 100 tests.

Observe:• The invariant for insert i t is only tested when the invariant holds

for t short-circuit semantics of ||• Many randomly generated trees do not satisfy the invariant, and

hence more that 100 tests may be needed

7 DTU Compute, Technical University of Denmark Property-based testing: An appertizer MRH 07/11/2019

02157FunctionalProgram-

ming

Michael R. Hansen

Configuring the number of tests

Suppose we want 400 random tests:

let _ = Check.One ( Config.Quick with MaxTest = 400; ,insertInvProp)

Ok, passed 400 tests.

8 DTU Compute, Technical University of Denmark Property-based testing: An appertizer MRH 07/11/2019

02157FunctionalProgram-

ming

Michael R. Hansen

Summary

Property-based testing supports testing at a high level of abstraction• Focus is on fundamental properties – not on concrete test cases• You write programs for properties – not concrete test cases• Properties are tested automatically• Short counterexamples are found — when properties are

falsified

The examples given here are just appetizers.

Next week we will have a look at• more kinds of properties• user-defined random generators• ...

9 DTU Compute, Technical University of Denmark Property-based testing: An appertizer MRH 07/11/2019