Embed Size (px)
Citation preview
Automated Unit Test GenerationVassil Popovski, VMware
The Cost of Software Bugs?
$59.9 B per year in USA (NIST, 2002)*
(*) http://www.nist.gov/director/planning/upload/report02-3.pdf
$312 B per year worldwide (Cambridge University, 2012)**
(**) http://www.roguewave.com/DesktopModules/Bring2mind/DMX/Download.aspx?entryid=1606&command=core_download&PortalId=0&TabId=607
$312,000,000,000
The Cost of Software Bugs?
The Cost of Software Bugs?
Total market capitalization for all 3 companies(*): !
$177.36 B
(*) Google finance - as of Nov 2nd, 2013
The Cost of World Hunger?
$40B per year(*)
(*) According to United Nations
The Cost of Fixing Bugs?
http://www.amazon.com/Assessments-Benchmarks-Addison-Wesley-Information-Technology/dp/0201485427?ie=UTF8&s=books&qid=1209056706&sr=1-1
Software Testing Pyramid
http://blogs.agilefaqs.com/2011/02/01/inverting-the-testing-pyramid/
Unit Testing: 80-90%
Integration testing: 4-15%
E2E/GUI testing: 1-5%
Techniques/Tools
1. Random data generation 2. Search based 3. Symbolic/Concolic based
The goal: High (close to 100% code coverage*)
Anatomy of a Unit Test
Arrange Act Assert
Anatomy of a Unit Test
Arrange Data Arrange Mocks/Stubs Act Assert
Anatomy of a Unit Test
Evaluation of Tools
Arrange/Generate Data
Arrange/Generate Mocks/Stubs
Act/Invoke methods
Assert/Verify the results
Techniques/Tools
x != 100 x == 100
x>=z x<z
if
if
Techniques/Tools
x>=z x<z
if
ifRandom data generation: !Feed x,y with random data
x != 100 x == 100
Demo
Evaluation of Tools
Arrange/Generate Data
Arrange/Generate Mocks/Stubs
Act/Invoke methods
Assert/Verify the results
junit-quickcheck feed4JUnit
yes - some
no
no
no
Techniques/Tools
1. Random Data Generation: • junit-quickcheck (Java) https://github.com/pholser/junit-quickcheck/
• Feed4JUnit & Feed4TestNG + Benerator (java) http://databene.org/feed4-tools.html http://databene.org/databene-benerator.html
• Randoop (Java + .NET) https://code.google.com/p/randoop/
• T2 (Java) https://code.google.com/p/t2framework/
Techniques/Tools
x != 100 x == 100
x>=z x<z
if
ifSearch based approach: !1. Start with random/initial data
(x=1, y=1)
Techniques/Tools
x != 100 x == 100
x>=z x<z
if
ifSearch based approach: !1. Start with random/initial data
(x=1, y=1)
x=1, y=1
Techniques/Tools
x != 100
x>=z x<z
if
ifSearch based approach: !1. Start with random/initial data
(x=1, y=1)
x=1, y=1
x == 100
Techniques/Tools
x == 100
x>=z x<z
if
ifSearch based approach: !1. Start with random/initial data
(x=1, y=1) 2. Based on coverage information
- select new value(s) (x==100) !
x != 100
x=1, y=1
Techniques/Tools
x == 100
x>=z x<z
if
ifSearch based approach: !1. Start with random/initial data
(x=1, y=1) 2. Based on coverage information
- select new value(s) (x==100)
x=100, y=1
x != 100
Demo
Evaluation of Tools
Arrange/Generate Data
Arrange/Generate Mocks/Stubs
Act/Invoke methods
Assert/Verify the results
yes - some
no
no
no
EvoSuite
yes
no
yes
yes
junit-quickcheck feed4JUnit
Techniques/Tools
2. Search Based: • EvoSuite (Java)
http://www.evosuite.org/ • Testful (Java)
https://code.google.com/p/testful/
Techniques/Tools
x != 100 x == 100
x>=z x<z
if
ifSymbolic based approach: !1. Evaluate path conditions (PC)
on every branch PC: X!=100, Y PC: X == 100, Y
x=X, y=Y, z=2*Y
PC: X == 100, X>=2*Y
PC: X == 100, X<2*Y
1. X!=100, Y => (x=1, y=1) 2. X=100, X>=2*Y =>(x=100, y=1) 3. X=100, X<2*Y =>(x=100, y=51)
2. For each leaf - calculate X & Y using constraints solver:
Techniques/Tools
x != 100 x == 100
x>=z x<z
if
ifConcolic based approach: !Optimization of symbolic execution - but use real values for some of the branches
Demo
Evaluation of Tools
Arrange/Generate Data
Arrange/Generate Mocks/Stubs
Act/Invoke methods
Assert/Verify the results
yes - some
no
no
no
EvoSuite
yes
no
yes
yes
PEX (+ Moles)
yes
yes - some
yes
yes
junit-quickcheck feed4JUnit
Techniques/Tools
3. Symbolic/Concolic based: • Microsoft Pex (.NET) http://research.microsoft.com/en-us/projects/pex/
• jCUTE (Java) http://osl.cs.uiuc.edu/software/jcute/
• jalangi (Java Script) https://github.com/SRA-SiliconValley/jalangi
• AgitarOne (Java, commercial) http://www.agitar.com/solutions/products/agitarone.html
• Jtest (Java, commercial) http://www.parasoft.com/jsp/products/jtest.jsp
!
Best usage of Auto Generated Unit Tests
1.Regression testing of legacy code !2.To support your unit testing / TDD
Automatic Generation of GUI/API tests?
Q&A
