How Fast and Fat IsYour Probabilistic Model

Checker?an experimental performance

comparison

David N. Jansen3,1, Joost-Pieter Katoen1,2, Marcel Oldenkamp2,

Mariëlle Stoelinga2, Ivan Zapreev1,2

1 MOVES Group, RWTH Aachen University2 FMT Group, University of Twente, Enschede3 ICIS, Radboud University, Nijmegen

ProbabilisticModel Checker

ProbabilisticModel Checking

Probabilistic System Probabilistic Requirement

≤Probabilistic Model Probabilistic Formula

Yes No

Probability

ETMCC

MRMCPRISM(sparse)

PRISM(hybrid)

YMER

VESTA

Why This Work?• Used more often

– applications: distributed systems, security, biology, quantum computing...

• Powerful tools

• Problem: Which tool to choose?

ProbabilisticModel Checker

ProbabilisticModel Checkers

Probabilistic System Probabilistic Requirement

Probabilistic Model Probabilistic Formula

Yes No

Probability

Choices made

Four examples

Overall evaluation

ToolsETMCC

numerical

CTMC Java

MRMC DTMC + CTMC C

PRISM hybrid

DTMC + CTMCMTBDD for transition matrix

C(++) and JavaPRISM

sparse

DTMC + CTMCsparse transition matrix

VESTAstatis

tCTMC, no S Java

YMER CTMC, only U ≤t C(++)

Experiment Relevance

• Repeatable• Verifiable• Significant• Encapsulated

Selected Benchmarks

Synchronous Leader Election

discrete time

Randomized Dining Philosophers

discrete time

Birth–Death Processdiscrete time

Tandem Queuing Networkcontinuous time

Cyclic Server Pollingcontinuous time

SynchronousLeader Election

• nodes in a ring elect a leader– each node selects random number as id– passes it around the ring (synchronously)

– if unique id,node with maximum unique id is leader

• [Itai & Rodeh, 1990]

SynchronousLeader Election

1

3

2

1

5

5

42

5

4

1

2 2

3

5

14

1

2

2 3

5

1

51

2

2

3 5

1

5

4

SynchronousLeader Election

1

3

2

1

5

5

42

RandomizedDining Philosophers

• Dining Philosophers– pick up chopsticks in random order

• Deadlocks resolved– if there is no second chopstick,give up eating

• [Pnueli & Zuck, 1986]

Birth–Death Process

• Models a waiting queue• Standard modelin performance evaluation

• Limit queue size to get finite model

Tandem Queueing Network

• Two queues after each other

• [Hermanns, Meyer-Kayser & Siegle, 1999]

checkincounter

two-phase

security check

exponential

Cyclic Polling System• server cycles over n stationsand serves each one in turn– e.g. teacher walks through class,each pupil may ask a question

• [Ibe & Trivedi, 1990]

Modelling

PRISMmodel

.tra formatmodel

PRISM

YMERmodel

ETMCC MRMC YMER VESTA

VESTAmodel

adaptsyntax

informaldescription

Experiment 1Qualitative Properties• unbounded reachability with prob 1

• Cyclic Polling System:

busy1 P≥1(true U poll1)

If station 1 is busy,the server will poll it eventually

1

10

100

1000

10000

100000

1000000

10000000

3 6 9 12 15 18number of stations n

log.scale VSZ memory (Kbytes)

mrmcprism hybridprism sparseetmccymervesta

0,00001

0,0001

0,001

0,01

0,1

1

10

100

1000

10000

3 6 9 12 15 18number of stations n

log.scale model check time (sec.)

PRISM: MTBDD Size

• Multi-Terminal BDD =data structure for transition matrix

• size heavily depends on model• large MTBDD slow

Model#

states# MTBDD nodes

Synchronous leader election

500.0001.000.000

.

Cyclic polling

7.000.000

< 3.000 .

CPS versus SLE runtime

0,0001

0,001

0,01

0,1

1

10

100

1000

10000

100000

4_4 4_8 4_12 4_16 8_2 8_4number of processes_random set size n_k

log.scale model check time (sec.)

0,00001

0,0001

0,001

0,01

0,1

1

10

100

1000

10000

3 6 9 12 15 18number of stations n

log.scale model check time (sec.)

mrmcprism hybridprism sparseetmccymervesta

7.077.888 states2.745 MTBDD nodes

458.847 states1.131.806 MTBDD nodes

VESTA:simulation problem

• actual probability close to bound P≥p(...)

• estimate is almost always in [p–,p+]

• some irregularity stops the simulation

• 0.95 Prob(yes actual Prob≥p) Prob(actual Prob≥p yes)

P≥1(... U ...)Timing Overview

1E-6

1E-3

1E+0

1E+3

1E+0 1E+3 1E+6 1E+9

number of states

time [sec]

ETMCCMRMCPRISM hybrid

PRISM sparseVESTA

Analysis

ETMCC slow, out of memory

MRMC fastest tool for small models

PRISM only symbolic sparse=hybriddepends on MTBDD size

VESTAexcessive number of samples, slow

YMER not implemented

depends heavilyon MTBDD sizedepends heavilyon MTBDD sizedepends heavilyon MTBDD size

Result Overview: Timing

ETMCC MRMC PRISM hybri

d

PRISM spars

e

VESTA YMER

unb’nded U – +

+/++

+/++

–/0 N/A

Result Overview: Memory

ETMCC MRMC PRISM hybri

d

PRISM spars

e

VESTA YMER

unb’nded U – +

+/++

+/++

0/+ N/A

MTBDD sizevaries heavily

almost independentfrom model size

Experiment 2Bounded Reachability

• Tandem Queueing Network:

P<0.01(true U ≤2 full )

Is the probabilitythat the system gets full in 2 time unitssmall?

1

10

100

1000

10000

100000

1000000

10000000

10 50 100 255 511 1023queue size n

log.scale VSZ memory (Kbytes)

0,00001

0,0001

0,001

0,01

0,1

1

10

100

1000

10000

10 50 100 255 511 1023queue size n

log.scale model check time (sec.)

mrmcprism hybridprism sparseetmccymervesta

Analysis

ETMCC slow, out of memory

PRISMsparse=faster, hybrid=smaller

MRMC fast for small models

VESTAokif you can afford statistical errors

YMERbest choiceif you can afford statistical errors

Result Overview: Timing

ETMCC MRMC PRISM hybri

d

PRISM spars

e

VESTA YMER

unb’nded U – +

+/++

+/++

–/0 N/A

bounded U – + 0/++/++

+ ++

Result Overview: Memory

ETMCC MRMC PRISM hybri

d

PRISM spars

e

VESTA YMER

unb’nded U – +

+/++

+/++

0/+ N/A

bounded U – ++/++

+ + ++

Experiment 3Steady State Property

• Tandem Queuing Network

S>0.2( P>0.1(X 2nd queue full

) )

In equilibrium,the probability to satisfy

is > 0.2

P>0.1(X 2nd queue full )

P>0.1(X ... )

1

10

100

1000

10000

100000

1000000

10000000

10 50 100 255 511 1023queue size n

log.scale VSZ memory (Kbytes)

0,00001

0,0001

0,001

0,01

0,1

1

10

100

1000

10000

10 50 100 255 511 1023queue size n

log.scale model check time (sec.)

mrmcprism hybridprism sparseetmccymervesta

Simulating Steady State?

• simulation of bounded reachabilityhas clear stopping criterion

• simulation of unbounded reachability reachability with very large bound

• simulation of steady state? never stops

Analysis

ETMCC slow, out of memory

PRISMsparse=fasterhybrid=slightly smaller

MRMC fastest

VESTA not implemented

YMER not implemented

Result Overview: Timing

ETMCC MRMC PRISM hybri

d

PRISM spars

e

VESTA YMER

unb’nded U – +

+/++

+/++

–/0 N/A

bounded U – + 0/++/++

+ ++

steady state – ++ 0/+ + N/A N/A

Result Overview: Memory

ETMCC MRMC PRISM hybri

d

PRISM spars

e

VESTA YMER

unb’nded U – +

+/++

+/++

0/+ N/A

bounded U – ++/++

+ + ++

steady state – +

+/++

+ N/A N/A

Nested Formulas• Birth–Death Process:

P≥0.8(P≥0.9(true U ≤100 n70) U n50)

The probability to reach n50

(while the probability to reach n70

in 100 steps never drops <0.9)is ≥0.8

0.00001

0.0001

0.001

0.01

0.1

1

10

100

1000

10000

100 1000 10000 100000maximum population size m

log.scale model check time (sec.)

1

10

100

1000

10000

100000

1000000

10000000

100 1000 10000 100000maximum population size m

log.scale VSZ memory (Kbytes)

mrmcprism hybridprism sparseetmccymervesta

Result Overview: Timing

ETMCC MRMC PRISM hybri

d

PRISM spars

e

VESTA YMER

unb’nded U – +

+/++

+/++

–/0 N/A

bounded U – + 0/++/++

+ ++

steady state – ++ 0/+ + N/A N/A

nested – ++ 0/+ + – – N/Adid not terminate

Result Overview: Memory

ETMCC MRMC PRISM hybri

d

PRISM spars

e

VESTA YMER

unb’nded U – +

+/++

+/++

0/+ N/A

bounded U – ++/++

+ + ++

steady state – +

+/++

+ N/A N/A

nested – ++/++

+ N/A N/A

ConclusionsETMCC worst, only small models

MRMC fastest for small models

PRISM hybrid

fast if MTBDD is small

PRISM sparse

fast

VESTArather slow, no S , statistical errors

YMERslim & very fast, only U ≤t,few statistical errors