18
import org.scalacheck._ BeScala – February 2012 Gilles Scouvart

ScalaCheck

  • Upload
    bescala

  • View
    3.521

  • Download
    0

Embed Size (px)

Citation preview

Page 1: ScalaCheck

import org.scalacheck._

BeScala – February 2012Gilles Scouvart

Page 2: ScalaCheck

Once upon a time…

• Three scientists (an astrophysicist, a physicist and a mathematician) are in a bus to a conference in Scotland

• Shortly after the border, they see a black sheep on a hillside.

• The astrophysicist declares: « In Scotland, all sheep are black! »

• The physicist then says: « Hm, all we can say is that in Scotland, some sheep are black. »

• The mathematician sighs and concludes: « All we can tell is that in Scotland, there exists one sheep with at least one black side! »

Page 3: ScalaCheck

The moral of the story

• We’re often relying on a limited set of simple examples to validate our assumptions

• But our implementation might totally overfit our test set (particularly in TDD)

• And edge cases might go completely unnoticed…

Can we do better?

Page 4: ScalaCheck

How can we be sure?

• We would like to check our assumptions for a « general case » (intension)– This can only be done through formal verification of the code– This cannot be automatized for Turing-complete languages (cf.

halting problem)• The alternative is then to generate explicitly all possible

configurations (extension) and test each of them– But the number of configurations can be huge or infinite

• Now if our code passes the test on 100 randomly chosen examples, we might still get some good level of confidence in our implementation– This is just what ScalaCheck proposes

Page 5: ScalaCheck

Main idea

In ScalaCheck, we check properties on random datasets created by generators.

Page 6: ScalaCheck

Properties

• Logical statements that the function must satisfy, e.g.

• Elements– Quantifier: forall, exists, atLeast, iff, imply…– Assertion: boolean function

• org.scalacheck.Prop

i , 2*i == i+i i , such that 2*i==2 i,j , (i*j) == 0 i == 0 || j == 0

forAll((i:Int) => 2*i == i+i)exists((i:Int) => 2*i == 2)forAll{(i:Int,j:Int)=>

(i*j)==0 ==> (i==0 || j==0)}

Math Scala

Try those, you might have some surprises…

Page 7: ScalaCheck

Generators

• org.scalacheck.Gen• Basically a function Params => Option[T]– Params: size + random number generator

• Built-in generators– choose: integer in range– oneOf : element from sequence of T or Gen[T]– frequency : oneOf taking weights into account– arbitrary : arbitrary instance (incl. edge cases)

• Gen is a Monad!– map, flatMap, filter

Page 8: ScalaCheck

Not «?», but Unicode!

Generator example

• Suppose we havecase class Person(name:String,age:Int)

• We can construct a Gen for Personval personGen = for (n<-arbitrary[String];a<-choose(1,125))

yield Person(n,a)

• We can test it using the sample functionpersonGen.sampleres: Option[Person] = Some(Person(???????????????,85))

• We can use it to test properties on PersonProp.forAll(personGen)((p:Person) => p.age>0)

• If we want to use it implicitly we can declare it as an arbitrary generatorimplicit val arbPerson = Arbitrary(personGen)Prop.forAll((p:Person) => p.age>0)

Page 9: ScalaCheck

Generator example (cont’d)

• Now if we only want young peopleval youngPeople = personGen.filter(_.age<31)

• We can use it for some specific testsProp.forAll(youngPeople)((p:Person) => p.age<50)

• If we want a list of 3 young people, we can writeval trio = Gen.listOfN(3,youngPeople)trio.sampleres: Option[List[Person]] =

Some(List(Person(????????????????????????,4), Person(???????????????,9), Person(???????????,20))))

Page 10: ScalaCheck

Complex failures

• Suppose you have a property that relies on complex data (e.g. a list)(l:List[Int]) => l.size==l.distinct.size

• Now this obviously fails for list with duplicates• ScalaCheck could give the first failure

! Falsified after 5 passed tests.> ARG_0: List("-2147483648", "1", "1933529754", "-726958561", "-

2147483648”, "750300922", "841716922", "-2147483648", "1671473995")

• But it gives instead! Falsified after 8 passed tests.> ARG_0: List("-1", "-1")

Page 11: ScalaCheck

How did this happen?

• ScalaCheck applies shrinking strategies to gradually simplify the problem to a simplest failing case

• Standard shrinking strategies are provided for most common cases– List– Tuple– String– Int– Container

Page 12: ScalaCheck

Integration with Scala frameworks

• ScalaTest–prop.Checkers trait• check method

(but cannot use the should/must matcher syntax)

• Specs2–check function

Page 13: ScalaCheck

Example

• Scampi– Library for

Operations Research

– Abstract algebra using GADTs

– Very mathematical

– Perfect fit

class AlgebraTest extends FeatureSpec with Checkers with Algebra { // [Generators for E…] def associativity(op: (E, E) => E) = (a: E, b: E, c: E) => simplify(op(op(a, b), c)) == simplify(op(a, op(b, c)))

def commutativity(op: (E, E) => E) = (a: E, b: E) => simplify(op(a, b)) == simplify(op(b, a))

feature ("Addition") { val addition = (e1: E, e2: E) => e1 + e2 scenario ("scalars") { check((i: Int, j: Int) => Const(i) + Const(j) == Const(i + j)) } scenario ("associativity") { check(associativity(addition)) } scenario ("commutativity") { check(commutativity(addition)) } } //…}

hg clone https://bitbucket.org/pschaus/scampi

Page 14: ScalaCheck

Other nice stuff

• collect to store generated values• classify to make histograms• Stateful testing– Finite State Machines– Types• State• Command

Page 15: ScalaCheck

Pros

• Enhance readability– Very declarative, appeals to mathematically-inclined– Domain experts can read and comment

• Focuses on properties of input and output– Instead of concrete examples

• Automatically explores edge cases– Instead of tedious, incomplete and error-prone code

• Shrinks to simplest failing examples

Domain knowledge Active documentation

Page 16: ScalaCheck

Cons• It can be difficult to

– Find « good » properties• Not trivially satisfied• Not too complex to observe

– Create « good » generators• Relevant edge cases

– Write checking functions• Implementing the check might be as error-prone as writing the function!

• Requires functional code

• Beware of shrinking– If your property relies on the size of your instances

Reflection on the domain

Reflection on the specifications

Focus on variations/set of properties

Good programming practice

Page 17: ScalaCheck

Credits

• Inspiration– Haskell library QuickCheck (Koen Claessen, John Hughes)

• Main contributors– Ricky Nilsson – Tony Morris – Paul Phillips – Yuvi Masonry

• Links– GitHub: https://github.com/rickynils/scalacheck– User guide: http://code.google.com/p/scalacheck/wiki/UserGuide– Articles:

http://code.google.com/p/scalacheck/wiki/ScalaCheckArticles

Page 18: ScalaCheck

Thank you!