Formal testing of timed and probabilistic systemsantares.sip.ucm.es/manolo/papers/InvTalk_ICTSS11.pdfMotivation The F o rmalism Confo rmance Relations Application and Derivation of

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationFormal testing of timed and probabilisti systemsManuel NúñezUniversidad Complutense de Madrid23rd Int. Conf. on Testing of Software and Systems

Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 1

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationOutline1 Motivation2 The Formalism3 Conforman e Relations4 Appli ation and Derivation of Test Cases5 Other Time Domain: Sto hasti time6 Timed Systems with Probabilisti InformationFormal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 2

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationFormal testing is urrently a big areaIn fa t, there is a ton of workshops, onferen es, symposia,spe ial issues on formal testing!Fo us of this talk: Relations between pro esses and appli ationof tests to systems.Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 3

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationFormal testing is urrently a big areaIn fa t, there is a ton of workshops, onferen es, symposia,spe ial issues on formal testing!Fo us of this talk: Relations between pro esses and appli ationof tests to systems.Testing an be informal, too...Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 3

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationFormal testing is urrently a big areaIn fa t, there is a ton of workshops, onferen es, symposia,spe ial issues on formal testing!Fo us of this talk: Relations between pro esses and appli ationof tests to systems.Testing an be informal, too... But this is out of the s ope of thistalkFormal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 3

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationIntrodu tionThis talk in a nutshellHaving a model/spe and a (bla k-box) SUT, try to establishthe onforman e of the SUT with respe t to the spe byapplying experiments.Add probabilities and/or time to the onsidered systems.Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 4

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationTemporal RequirementsThe temporal behavior of real-time systems is riti al.Di�erent time domains for representing onditions over thetime onsumed by the systems while it performs a tions.Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 5

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationTemporal RequirementsThe temporal behavior of real-time systems is riti al.Di�erent time domains for representing onditions over thetime onsumed by the systems while it performs a tions.Fix TimeA time value spe i�es the amount of time needed to performan a tion.Not always is easy to pre isely establish time requirements.Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 5

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationTemporal RequirementsThe temporal behavior of real-time systems is riti al.Di�erent time domains for representing onditions over thetime onsumed by the systems while it performs a tions.Time intervalsA time interval spe i�es the range of time values that thesystem an take to perform an a tion.Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 5

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationTemporal RequirementsThe temporal behavior of real-time systems is riti al.Di�erent time domains for representing onditions over thetime onsumed by the systems while it performs a tions.Sto hasti timeSpe ify that with probability p an a tion must be performedbefore t time units have passed.Sometimes the spe i�er either does not have su h probabilisti information or onsiders it unne essarily ompli ate the model.Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 5

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationFormal testing methodologyThe formalismRepresentation of systems that present non-standardrequirements.Notion of onforman eWhat it means for an implementation to onform to aspe i� ation.Derivation and appli ation of testsAlgorithms for derivation of tests from the spe i� ation.Relation between appli ation of tests and onforman e.Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 6

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationCommon onditionsSpe i� ation and Implementations an be expressed using thesame formalism.Implementations are input-enabled.Both spe i� ations and implementations are observable (wesometimes remove this restri tion).Deterministi 12 3a1/b1 a2/b1 Observable12 3a1/b1 a1/b2 Non-Deterministi 12 3a1/b1 a1/b1Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 7

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationFinite State Ma hinesM = (S ,I,O,Tr , sin)S is a �nite set of states.I is the set of input a tions.O is the set of output a tions.Tr is the set of transitions.sin is the initial state.


MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationFinite State Ma hiness1 s2s4 s3

a/xb/y a/x b/z /yb/yb/x

a/z /y

a/z


MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationFSMs extended with Time Valuess1 s2s4 s3

a/x/3b/y/5a/x/1b/z/2 /y/3b/y/2b/x/1

a/z/1 /y/5

a/z/3


MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationConforman e RelationsFun tional Implementation RelationAn implementation I onforms to a spe i� ation S if for all possibleevolution of S the outputs that the implementation I may performafter a given input are a subset of those of the spe i� ation.I onfnt S , if for all e = (i1/o1, . . . , ir−1/or−1, ir/or ) ∈ NTEvol(S),with r ≥ 1, we have thate ′ = (i1/o1, . . . , ir−1/or−1, ir/o ′r ) ∈ NTEvol(I ) =⇒ e ′ ∈ NTEvol(S)Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 11

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationExamples of fun tional onforman eS 1

2 a2/b3a1/b4a1/b4 I 12 a2/b3a1/b4a1/b4

a2/b4 I onfnt SS 1

3 2a3/b3 a1/b1a2/b2I 1

3 2a3/b3 a1/b1a2/b1 a3/b1Not I onfnt SFormal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 12

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationTimed onforman e relations: Right hoi e?We require I onfnt S plus something else...


MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationTimed onforman e relations: Right hoi e?We require I onfnt S plus something else...I onfa S : All evolutions in I take time = to an equivalentevolution in S .Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 13

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationTimed onforman e relations: Right hoi e?We require I onfnt S plus something else...I onfa S : All evolutions in I take time = to an equivalentevolution in S .I onfw S : All evolutions in I take time ≤ the slowestequivalent evolution in S .I onfb S : All evolutions in I take time ≤ to the fastestequivalent evolution in S .I onfsw S : Some evolution in I takes time ≤ to the slowestequivalent evolution in S .I onfsb S : Some evolution in I takes time ≤ to the fastestequivalent evolution in S .Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 13

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationRelations among relationsI onfb S ⇒ I onfsb S⇓ ⇓I onfa S ⇒ I onfw S ⇒ I onfsw S


MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationRelations among relationsI onfb S ⇒ I onfsb S⇓ ⇓I onfa S ⇒ I onfw S ⇒ I onfsw Stime-deterministi spe i� ations onfw = onfb and onfsw = onfsbtime-deterministi implementations onfw = onfsw and onfb = onfsb.time-deterministi spe i� ations and implementations onfa is still di�erent from onfb.Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 14

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationTimed Test CasesWhat is a test ase?Tests represent sequen es of inputs applied to an implementation.Che king fun tional behaviorOn e an output is re eived, the tester he ks whether it belongs tothe set of expe ted ones or not.Che king temporal behaviorTests in lude time stamps to he k the orre t performan e of a -tions.Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 15

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationApplying testss1s2 a/y , 3a/x , 9a/z , 10

I T1apass, 3 x yapass, 9x faily pass, 7z

failzT2b

pass x yafail x faily pass, 7z

failz

I ‖ T1X but I ‖ T2 6XFormal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 16

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationAppli ation of Tests to ImplementationsChe king fun tional behaviorTerminal states rea hed by the omposition of implementation andtest belong to the set of passing states.After we know that the fun tional behavior of the implementationis orre t with respe t to the suite, we he k time onditions.Che king time onditionsDi�erent notions of passing tests orresponding to:Domain for representing time requirements.Underlying onforman e relation.Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 17

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationTest DerivationAlgorithm for generating a test suite from a spe S.


MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationTest DerivationAlgorithm for generating a test suite from a spe S.Generating one test for ea h possible tra e in S.Non-deterministi hoi e with two possibilities.Close a bran h with a pass/fail state.Continue testing in the bran h.A time stamp is atta hed to ea h pass state: Time umulatedso far to perform the sequen e (values are taken from S).Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 18

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationTest DerivationAlgorithm for generating a test suite from a spe S.Generating one test for ea h possible tra e in S.Non-deterministi hoi e with two possibilities.Close a bran h with a pass/fail state.Continue testing in the bran h.A time stamp is atta hed to ea h pass state: Time umulatedso far to perform the sequen e (values are taken from S).By onsidering all hoi es we get a sound and omplete test suite.Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 18

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationUn omplete CompletenessIn general, generated test suites are in�nite.Completeness in the limitGenerating all test of length n.Completeness is a hieved when n tends to in�nite.


MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationLimitations of �xed time valuesSometimes we annot simply use a value to represent a time onstraint.Consider more expressive frameworks: Sto hasti time


MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationWhat is Sto hasti Time?Combination of deterministi time and probabilities.


MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationWhat is Sto hasti Time?Combination of deterministi time and probabilities.What is Deterministi Time?The message will arrive before 2 se onds.Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 21

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationWhat is Sto hasti Time?Combination of deterministi time and probabilities.What is Deterministi Time?The message will arrive before 2 se onds.How do we use Probabilities?The message will arrive with probability 0.95.Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 21

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationWhat is Sto hasti Time?Combination of deterministi time and probabilities.What is Deterministi Time?The message will arrive before 2 se onds.How do we use Probabilities?The message will arrive with probability 0.95.So, sto hasti time is...Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 21

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationWhat is Sto hasti Time?Combination of deterministi time and probabilities.What is Deterministi Time?The message will arrive before 2 se onds.How do we use Probabilities?The message will arrive with probability 0.95.So, sto hasti time is...The message will arrive before 2 se onds with probability 0.95.Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 21

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationFSMs extended with Sto hasti TimeFξ1(x) =

0 x < 0x2 0 ≤ x < 21 2 ≤ xUniform distribution in [0, 2) s1 s2s4 s3

a/x/ξ1b/y/ξ2a/x/ξ3b/z/ξ1 /y/ξ3b/y/ξ2b/x/ξ2

a/z/ξ1 /y/ξ1

a/z/ξ2Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 22

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationRandom VariablesA random variable ξ takes values a ording to a probabilitydistribution fun tion Fξ.Fξ(x) = p means the probability that ξ takes a value smallerthan or equal to x is p.Random variables an be added. We may de�ne ξ distributedas ξ1 + ξ2.In our framework, random variables always denote time:Fξ(x) = 0, for x < 0.We write ξ1 ∼ ξ2 if for all x we have Fξ1(x) = Fξ2(x).Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 23

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationSimulating Deterministi TimeDira distributions: The message arrives exa tly at time 2F (x) = { 0 x < 21 2 ≤ xUniform distributions: The message arrives in the interval [0, 2],being times equiprobableF (x) =

0 x < 0x2 0 ≤ x < 21 2 ≤ xFormal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 24

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationA First Implementation RelationSto hasti tra es of M: STra es(M)A tra e plus the random variable denoting time.I onfs SI sto hasti ally onforms to S if I onfnt S and the randomvariables asso iated with a given tra e in S and in I are equivalent.(ρ, ξ) ∈ STra es(I )

∧ρ ∈ NSTra es(S)

=⇒ (ρ, ξ′) ∈ STra es(S) ∧ ξ ∼ ξ′Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 25

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationProperties of onfsFrom a theoreti al point of view, onfs is an appropriate onforman e relation.


MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationProperties of onfsFrom a theoreti al point of view, onfs is an appropriate onforman e relation.UnfortunatelySpe i� ationWe know everything about its probability distribution fun tions.ImplementationUnder bla k-box testing, we don't have a ess to them.Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 26

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationProperties of onfsTime Intervals and Sto hasti Time((i1/o1, . . . , in/on), t) is a timed exe ution of I if theobservation of I shows (i1/o1, . . . , in/on) performed in time t.H = {|(e ′1, t1), . . . , (e ′n, tn)|} be a multiset of timed exe utions:Sampling(H)(e) = {|t | (e, t) ∈ H|}


MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationSto hasti TimeHypothesis ContrastWe take from the spe i� ation the expe ted performan e ξS .We apply the test several times and obtain a set of timevalues H.We determine whether these are similar enough up to a on�den e level.Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 28

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationHypothesis ContrastWe take from the spe i� ation the expe ted performan e FS .We apply the test several times and obtain time values:At time Frequen y0.75 21 observations1 16 observations1.5 25 observations2.15 12 observations2.3 48 observations2.8 4 observationsFrom these time values we obtain a fun tion FI .Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 29

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationHypothesis ContrastWe determine whether these two fun tions are similar enough(up to a level of on�den e).FS FIFormal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 30

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationSto hasti TimeFormally, I (α,H) sto hasti S

(e, ξS) ∈ TEvol(S) =⇒ γ(ξS , Sampling(H,Φ)(e)) > α


MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationSto hasti TimeOnly di�eren e with previous tests: Random variables areassigned to passing states.Test exe utions are ompared with the random variableatta hed to the pass states.Test exe utions are used by di�erent tests sharing the sameobserved sequen e.


MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationSto hasti TimePassing a TestI passes the test T with probability α if after applying the testseveral times:The test always �nishes in a passing state.Time values observed might have been generated by therandom variables of the test.Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 33

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationOne step forward...We might like to quantify non-deterministi hoi es of systems.Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 34

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationAdding probabilities to FSMsGoal: quantify non-determinism.x/y x/y ′ x/y ,p x/y ′,1-p


MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationAdding probabilities to FSMsGoal: quantify non-determinism.No probabilisti relation among di�erent inputs.x1/y1x1/y2

x2/y3x2/y4x2/y5Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 35

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationAdding probabilities to FSMsGoal: quantify non-determinism.No probabilisti relation among di�erent inputs.For ea h input x and state s, the probabilities asso iated withs and x add up to 1 x1/y1, 13x1/y2, 23

x2/y3, 12x2/y4, 14x2/y5, 14Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 35

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationProbabilisti Sto hasti Finite State Ma hineA non-deterministi �nite state ma hine in whi h every transitionalso has an asso iated probability and a random variable.M = (S , s0, Li , Lo,PT ,PV)S is a �nite set of statess0 is the initial stateLi is the set of input a tionsLo is the set of output a tionsPT : S × Li × Lo × S → [0, 1] is theprobability-transition fun tionPV : S × Li × Lo × S → V is the time fun tion.For all s ∈ S and a ∈ Li , ∑p∈PT (s,a,x ,s′) p = 1.Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 36

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationProbabilisti Sto hasti Finite State Ma hineFor all (s, a, x , s ′) ∈ S × Li × Lo × S if PT (s, a, x , s ′) = p > 0and PV(s, a, x , s ′) = ξ then (s, a, x , p, ξ, s ′) is a transition ofM.Transition: (s, a, x , p, ξ, s ′)If the ma hine is in state s and re eives the input a thenwith probability pit produ es the output x , andit moves into the state s ′.before time t with probability Fξ(t)Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 37

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationExe ution Time of Input Sequen esLet M = (S , s0, Li , Lo,PT ,PV) be a PSFSM,a/y be an input/output sequen e, ands ∈ S a state.Time spent to rea h the state s ′ from s performing with a/yP∗

V (s, ε, x , s′) = θP∗

V (s, aa, xx , s′) =

P∗

V(s, a, x , s′′) + PV (s′′, a, x , s′) if ∃ s′′ ∈ S :PT (s′′, a, x , s′) > 0

∧PT (s, a, x, s′′) > 0θ otherwiseFormal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 38

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationExample s1s2 s3x1/y2/1, ξ1x2/y3/1, ξ2x1/y3/12 , ξ3x1/y1/34 , ξ1 x2/y1/34 , ξ3

x2/y1/1, ξ1x3/y1/1, ξ2x2/y2/14 , ξ2x1/y1/14 , ξ1

x1/y1/12 , ξ3x3/y2/1, ξ1p∗T(s1, x1x2, y1y1) = 14 ∗

34p∗V(s1, x1x2, y1y1) = ξ1 + ξ3Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 39

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationConforman e RelationsCorre tnessRequire that tra es of the spe i� ation that an be performed by theimplementation have the same asso iated probability and delay


MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationConforman e RelationsCorre tnessRequire that tra es of the spe i� ation that an be performed by theimplementation have the same asso iated probability and delayThe same problemNo a ess to probabilities and random variables of the implementa-tionFormal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 40

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationConforman e RelationsCorre tnessRequire that tra es of the spe i� ation that an be performed by theimplementation have the same asso iated probability and delayThe same problemNo a ess to probabilities and random variables of the implementa-tionProposal based on a �nite set of observationsChe k that the observed outputs and exe ution times in the imple-mentation �t the probabilities and random variables of the spe i�- ation.Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 40

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationNotion of �ttingProbabilisti onforman e: Interval estimationDo not request that the probabilities of the implementation be equalto the ones orresponding to the spe i� ation but that this fa thappens up to a ertain probability .Temporal onforman e: Hypothesis ontrastDe ide wether the sample ould be generated by the orrespondingrandom variable with a ertain on�den e.Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 41

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationLet S and I be PSFSM,H be a multiset of timed exe utions of IΦ = {σ | ∃ t : (σ, t) ∈ H}, and0 ≤ α ≤ 1.

(α,H)-probabilisti ally onforms to SFor all σ = a/x su h that P∗T (s0, a, x) > 0 we haveP∗T (s0, a, x) ∈ CIα(SeqSampling(H,Φ)(σ))

(α,H)-sto hasti ally onforman e to SFor all σ = a/x we haveγ(P∗

V(s0, a, x), Sampling(H,Φ)(σ)

)

> αFormal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 42

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationTesting Probabilisti and Sto hasti SystemsChe king behavior1 Tests in lude verdi ts to determine whether the outputobserved belongs to the set of expe ted ones or not.2 Tests in lude probabilities.3 Tests in lude random variables.Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 43

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationTest Case Tapass, 12 , ξ1 x ybfail y xbpass, 112 , ξ2 y failx failz pass, 16 , ξ5z pass, 14 , ξ3z


MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationApplying test asess1 s2s4 s3

a/x , γ3, 1− p1a/y , γ3, p1a/x , γ1, 1b/z , γ2, p2a/y , γ3, 1b/y , γ2, 1b/x , γ1, p3

b/z , γ2, 1− p3 a/z , γ1, 1b/z , γ1, 1

b/y , γ3, 1− p2Ta

pass, ξ1, 14 x ybpass, ξ3, 38x faily pass, ξ1, 18z

failz


MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationApplying test ases - Initial states1 s2s4 s3


b/z , γ2, 1− p3 a/z , γ1, 1b/z , γ1, 1

b/y , γ3, 1− p2Ta


failz


MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationApplying test ases - We apply the input as1 s2s4 s3


b/z , γ2, 1− p3 a/z , γ1, 1b/z , γ1, 1

b/y , γ3, 1− p2Ta


failz


MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationApplying test ases - We apply the input as1 s2s4 s3


b/z , γ2, 1− p3 a/z , γ1, 1b/z , γ1, 1

b/y , γ3, 1− p2Ta


failz


MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationApplying test ases - The output y is emitteds1 s2s4 s3

a/x , γ3, 1− p1a/y, γ3, p1a/x , γ1, 1b/z , γ2, p2a/y , γ3, 1b/y , γ2, 1b/x , γ1, p3

b/z , γ2, 1− p3 a/z , γ1, 1b/z , γ1, 1

b/y , γ3, 1− p2Timed test exe utions {(a/y/3)}

Tapass, ξ1, 14 x yb

pass, ξ3, 38x faily pass, ξ1, 18zfailz


MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationApplying test ases - State s4 is rea heds1 s2s4 s3


b/z , γ2, 1− p3 a/z , γ1, 1b/z , γ1, 1





MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationApplying test ases - We apply the input bs1 s2s4 s3


b/z , γ2, 1− p3 a/z , γ1, 1b/z , γ1, 1





MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationApplying test ases - The output x is emitteds1 s2s4 s3

a/x , γ3, 1− p1a/y , γ3, p1a/x , γ1, 1b/z , γ2, p2a/y , γ3, 1b/y , γ2, 1b/x, γ1, p3

b/z , γ2, 1− p3 a/z , γ1, 1b/z , γ1, 1

b/y , γ3, 1− p2Timed test exe utions {(ab/yx/7)}




MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationChe king probabilities and delays onditionsWe need several appli ations of the test to the implementation.To evaluate if a set of test exe utions mat h the distributionfun tion asso iated to the random variable indi ated by the orresponding state of the test.To evaluate if a set of test exe utions �t the probabilitiesasso iated to the orresponding states of the test.Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 53

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationChe king delays and probabilitiesSample of timed test exe utions(ab/yx/7),(ab/yz/5),(ab/yx/6.9),(ab/yx/7.1),. . .38 ∈ CIα({|ab/yx , ab/yz , ab/yx , ab/yx , . . . |})

γ({|7 6.9 7.1 . . . |}, ξ3)Ta


failz


MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationSome referen esM.G. Merayo, M. Núñez, and I. Rodríguez.Formal testing from timed �nite state ma hines.Computer Networks, 52(2):432�460, 2008.M.G. Merayo, I. Hwang, M. Núñez and Ana Cavalli.A statisti al approa h to test sto hasti and probabilisti systems.11th Int. Conf. on Formal Engineering Methods, ICFEM'09,LNCS 5885:186�205,2009Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 55

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationOther workM.G. Merayo, M. Núñez, and I. Rodríguez.Extending EFSMs to spe ify and test timed systems witha tion durations and timeouts.IEEE Transa tions on Computers, 57(6):835�848, 2008M. G. Merayo and M. Núñez and I. RodríguezA formal framework to test soft and hard deadlines in timedsystems.Software Testing, Veri� ation and Reliability, doi:10.1002/stvr.448, 2012Formal testing of timed and probabilisti systems 23rd Int. Conf. on Testing of Software and Systems 56

MotivationThe FormalismConforman e RelationsAppli ation and Derivation of Test CasesOther Time Domain: Sto hasti timeTimed Systems with Probabilisti InformationThanks for you attention!


Documents

Formal testing of timed and probabilistic systemsantares.sip.ucm.es/manolo/papers/InvTalk_ICTSS11.pdfMotivation The F o rmalism Confo rmance Relations Application and Derivation of