Testing for Educational Gaming and Educational Gaming for Testing

  • Published on
    09-May-2015

  • View
    312

  • Download
    1

Embed Size (px)

DESCRIPTION

A talk given at Google Test Automation Conference (GTAC 2013) https://developers.google.com/google-test-automation-conference/

Transcript

<ul><li>1.Collaborators: Microsoft Research RiSE (Nikolai Tillmann, Jonathan de Halleux)Microsoft Research Connections (Judith Bishop)Testing forEducational Gaming andEducational Gamingfor TestingTao XieNorth Carolina State Universitycontact: taoxie@gmail.com</li></ul> <p>2. Testing Educational GamingDynamic Symbolic Execution (Pex)Parameterized Unit TestingPex for Fun (Pex4Fun)TrainingSupporthttp://research.microsoft.com/pex/ http://pex4fun.com 3. AgendaBackgroundPex, Dynamic Symbolic Execution,Parameterized Unit TestingWriting Code in a BrowserCoding DuelsEducational PlatformConclusion 4. PEX, DYNAMIC SYMBOLIC EXECUTION,PARAMETERIZED UNIT TESTINGBackground 5. http://research.microsoft.com/pex/ 6. void CoverMe(int[] a){if (a == null) return;if (a.Length &gt; 0)if (a[0] == 1234567890)throw new Exception("bug");}Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork 7. void CoverMe(int[] a){if (a == null) return;if (a.Length &gt; 0)if (a[0] == 1234567890)throw new Exception("bug");}InputnullDynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork 8. void CoverMe(int[] a){if (a == null) return;if (a.Length &gt; 0)if (a[0] == 1234567890)throw new Exception("bug");}Ta==nullInputnullExecute&amp;MonitorObserved constraintsa==nullDynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork 9. void CoverMe(int[] a){if (a == null) return;if (a.Length &gt; 0)if (a[0] == 1234567890)throw new Exception("bug");}Ta==nullConstraints to solvea!=nullInputnullExecute&amp;MonitorChoose next pathObserved constraintsa==nullDynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork 10. void CoverMe(int[] a){if (a == null) return;if (a.Length &gt; 0)if (a[0] == 1234567890)throw new Exception("bug");}Ta==nullConstraints to solvea!=nullInputnull{}Execute&amp;MonitorSolveChoose next pathObserved constraintsa==nullDynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork 11. void CoverMe(int[] a){if (a == null) return;if (a.Length &gt; 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length&gt;0FTFa==nullConstraints to solvea!=nullInputnull{}Execute&amp;MonitorSolveChoose next pathObserved constraintsa==nulla!=null &amp;&amp;!(a.Length&gt;0)Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork 12. void CoverMe(int[] a){if (a == null) return;if (a.Length &gt; 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length&gt;0FTFa==nullConstraints to solvea!=nulla!=null &amp;&amp;a.Length&gt;0Inputnull{}Execute&amp;MonitorSolveChoose next pathObserved constraintsa==nulla!=null &amp;&amp;!(a.Length&gt;0)Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork 13. void CoverMe(int[] a){if (a == null) return;if (a.Length &gt; 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length&gt;0a[0]==123TFTFFa==nullConstraints to solvea!=nulla!=null &amp;&amp;a.Length&gt;0Inputnull{}{0}Execute&amp;MonitorSolveChoose next pathObserved constraintsa==nulla!=null &amp;&amp;!(a.Length&gt;0)Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork 14. void CoverMe(int[] a){if (a == null) return;if (a.Length &gt; 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length&gt;0a[0]==123TFTFFa==nullConstraints to solvea!=nulla!=null &amp;&amp;a.Length&gt;0Inputnull{}{0}Execute&amp;MonitorSolveChoose next pathObserved constraintsa==nulla!=null &amp;&amp;!(a.Length&gt;0)a==null &amp;&amp;a.Length&gt;0 &amp;&amp;a[0]!=1234567890Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork 15. void CoverMe(int[] a){if (a == null) return;if (a.Length &gt; 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length&gt;0a[0]==123TFTFFa==nullConstraints to solvea!=nulla!=null &amp;&amp;a.Length&gt;0a!=null &amp;&amp;a.Length&gt;0 &amp;&amp;a[0]==123456890Inputnull{}{0}Execute&amp;MonitorSolveChoose next pathObserved constraintsa==nulla!=null &amp;&amp;!(a.Length&gt;0)a==null &amp;&amp;a.Length&gt;0 &amp;&amp;a[0]!=1234567890Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork 16. void CoverMe(int[] a){if (a == null) return;if (a.Length &gt; 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length&gt;0a[0]==123TFTFFa==nullConstraints to solvea!=nulla!=null &amp;&amp;a.Length&gt;0a!=null &amp;&amp;a.Length&gt;0 &amp;&amp;a[0]==123456890Inputnull{}{0}{123}Execute&amp;MonitorSolveChoose next pathObserved constraintsa==nulla!=null &amp;&amp;!(a.Length&gt;0)a==null &amp;&amp;a.Length&gt;0 &amp;&amp;a[0]!=1234567890Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork 17. void CoverMe(int[] a){if (a == null) return;if (a.Length &gt; 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length&gt;0a[0]==123TFTFFa==nullTConstraints to solvea!=nulla!=null &amp;&amp;a.Length&gt;0a!=null &amp;&amp;a.Length&gt;0 &amp;&amp;a[0]==123456890Inputnull{}{0}{123}Execute&amp;MonitorSolveChoose next pathObserved constraintsa==nulla!=null &amp;&amp;!(a.Length&gt;0)a==null &amp;&amp;a.Length&gt;0 &amp;&amp;a[0]!=1234567890a==null &amp;&amp;a.Length&gt;0 &amp;&amp;a[0]==1234567890Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork 18. void CoverMe(int[] a){if (a == null) return;if (a.Length &gt; 0)if (a[0] == 1234567890)throw new Exception("bug");}a.Length&gt;0a[0]==123TFTFFa==nullTConstraints to solvea!=nulla!=null &amp;&amp;a.Length&gt;0a!=null &amp;&amp;a.Length&gt;0 &amp;&amp;a[0]==123456890Inputnull{}{0}{123}Execute&amp;MonitorSolveChoose next pathObserved constraintsa==nulla!=null &amp;&amp;!(a.Length&gt;0)a==null &amp;&amp;a.Length&gt;0 &amp;&amp;a[0]!=1234567890a==null &amp;&amp;a.Length&gt;0 &amp;&amp;a[0]==1234567890Done: There is no path left.Dynamic Symbolic Execution in Pexhttp://pex4fun.com/HowDoesPexWork 19. Vision: Parameterized Unit Testingvoid AddAndCountTest(List list, int item) {var count = list.Count;list.Add(item);Assert.AreEqual(count + 1, list.Count);} 20. Vision: Parameterized Unit Testingvoid AddAndCountTest(List list, int item) {var count = list.Count;list.Add(item);Assert.AreEqual(count + 1, list.Count);}for any list,for any item, 21. Vision: Parameterized Unit Testingvoid AddAndCountTest(List list, int item) {var count = list.Count;list.Add(item);Assert.AreEqual(count + 1, list.Count);} adding 1 itemincreases Count by 1 22. Vision: Parameterized Unit Testingvoid AddAndCountTest(List list, int item) {Assume.IsTrue(list != null);var count = list.Count;list.Add(item);Assert.AreEqual(count + 1, list.Count);} 23. Parameterized Unit Test =Unit Test with ParametersVision: Parameterized Unit Testingvoid AddAndCountTest(List list, int item) {Assume.IsTrue(list != null);var count = list.Count;list.Add(item);Assert.AreEqual(count + 1, list.Count);} 24. Parameterized Unit Test =Unit Test with ParametersSeparation of concernsSpecification of behaviorData to achieve coverageVision: Parameterized Unit Testingvoid AddAndCountTest(List list, int item) {Assume.IsTrue(list != null);var count = list.Count;list.Add(item);Assert.AreEqual(count + 1, list.Count);} 25. Parameterized Unit Testing in PracticeGetting PopularParameterized Unit Tests (PUTs) commonlysupported by various test frameworks .NET: Supported by .NET test frameworks Java: Supported by JUnit 4.XGenerating test inputs for PUTs supported by tools .NET: Supported by Microsoft Research Pex Java: Supported by Agitar AgitarOne(http://www.agitar.com/) 26. http://pex4fun.com/CoverMe 27. Limitations16K characters of code (single file / editor window)*Single-threaded onlyNo environment interactions*No non-determinism*Try to avoid floating-point computationsMax. 20s total exploration time** Limitation only applies to pex4fun.com, but not standalone Pex tool 28. Dynamic Symbolic ExecutionSummaryAsk Pex sends code to cloudCode is compiled and analyzed in cloudDynamic Symbolic Execution automatically findsrelevant interesting test inputs that achieve highcode coverageResults are shown in browser 29. WRITING CODEIN A BROWSERCode Auto-Completion 30. Auto-Completion 31. Writing Code in BrowserSummaryCode is compiled in cloudCode is executed in cloudAuto-completion via cloud/Javascriptin browser 32. CODING DUELSFun and Engaging Serious Game Win by Writing Code 33. Coding Duels1,168,577 clicked Ask Pex! 34. Coding DuelsPex computes semantic diff in cloudcode written in browser vs.secret reference implementationYou win when Pex finds no differences 35. Behind the Scene of Pex for FunSecret Implementationclass Secret {public static int Puzzle(int x) {if (x</p>