Simplify Testing using Fault Injection

Simplify Testing using Fault Injection

Rob Grzywinskirgrzywinski@yahoo.com

Motivation

• Writing and testing an IO-intensive application• Robust, asynchronous event (message) collector /

router• NIO (early 1.4 days)

• How to best simulate and test various failure modes?• NIO buggy as all get-out• Networks are inherently flaky and problematic

• Didn’t want to have to obfuscate or drastically change code in order to test it

Agenda

• Quick Introduction to Fault Injection• Why Use Fault Injection• Fault Injection Examples• Beyond Fault Injection• Quick Introduction to AOP• Fault Injection in Java

• AspectJ• Javassist

• Wrap Up

Fault Injection 101

• A fault is an abnormal condition or defect which may lead to a failure [10]

• Fault injection involves the deliberate insertion of faults or errors into a computer system in order to determine its response [9]

• The goal isn’t to recreate the conditions that produced the fault

Fault Injection 101 (cont)

There are two primary steps to Fault Injection:

• To identify the set of faults that can occur within an application, module, class, method, etc.• e.g. if the application does not use the network then

there’s no point in injecting network faults• In practice this isn’t as easy as it sounds

e.g. 3rd party libs

• To exercise those faults to evaluate how the application responds• Does the application detect the fault, is it isolated

and does the application recover from it? [8]

Example

byte[] readFile() throws IOException { ... final InputStream is = new FileInputStream(…); ... while((offset < bytes.length) && (numRead = is.read(bytes, offset, (bytes.length - offset))) >= 0) offset += numRead; ... is.close(); return bytes;}(from http://javaalmanac.com/egs/java.io/File2ByteArray.html)

What could go wrong with this code?

• new FileInputStream() can throw FileNotFoundException• InputStream.read() can throw IOException and

IndexOutOfBoundsException and can return -1 for end of file

• is.close() can throw IOException

How do we test how the application responds to one of these situations?

Specifically, how do we inject these faults so that we can test how the application responds?

Possible Techniques

• Force the situations at the OS level• Quite hairy to reproduce reliably when they’re

needed

• Refactor the snot out of it• Replace the call to InputStream.read() to some

local instrumented method• Create our own instrumented InputStream subclass

possibly using mock objects• Inject the subclass via IoC (requires some framework

such as PicoContainer or Spring)

• Just comment out the code and replace with throw new IOException()• Egad!

We need a way to inject a fault without changing the code!

Why Use Fault Injection

• Fault occurred in a production application and it is determined that that is a valid fault (i.e. it is expected to occur)

• Inject the fault in a testing environment without having to reproduce the conditions that actually caused the fault

• Any fault that is difficult or inconvenient to reproduce in testing:

• Out of memory• Disk full• Database crash• 3rd party library / Java bug• Deadlock• Resource contention• Dumb users• Invalid / stale sessions• Corner cases

Why Use Fault Injection (cont)

• You publish an API and you want to be tolerant of user input or response

• It is inconvenient or undesirable to refactor code to facilitate testing• e.g. 3rd party library

• You believe that testing code coverage is a good metric to measure for code quality [11]

• Injecting faults forces exception paths to be followed

More to come …

Reminder

You don’t have to recreate the conditions that caused the fault …

… you only have to recreate the fault

e.g. if an OhNoException crashed your application because some little old lady drove her 1966 Buick Skylark into your datacenter -- you don’t need the Skylark or the little old lady to fix the application

Quick Aside

Traditionally, Fault Injection is centers around faults derived from hardware:

• Disk faults such as file missing and corrupt file• Network faults such as host not found, host

inaccessible, and high latency• Memory faults such as corrupt memory• Operating system faults such as unexpected hardware

interrupts

Traditionally used in mission-critical fault tolerant environments

Quick Aside (cont)

Fault Injection comes in two favors:

• Hardware-based• Typically requires specialized hardware

• Software-based• Traditionally attempts to mimic hardware-based

fault injection and typically involves direct interaction with the operating system

• More recently expanded to include any fault that can occur in software such as:

• Mutated bytecode• Exceptions• Invalid or mutated input / output values• Deadlock / resource contention• Application Security• etc.

Example

Creating a directory and temporary file

final File directory = new File(…);final boolean success = directory.mkdirs();if(success) { final File file = File.createTempFile(…, …, directory); ...}

Faults:

• Making the parent directory(s) fails (returns false)

• Temp file creation failure (throws IOException)

• Permissions• Parent directory(s) deleted

• Injection of a specific temp file

Example (cont)

Great uses of Fault Injection that I can’t easily show in examples:

• Deeply nested exceptions• Effects of unchecked exceptions• Consequences of using catch(Exception e)

i.e. NullPointerExceptions that go unnoticed

• Complex / real-world cases

Intelligent Fault Injection

Fault injection doesn’t have to be all on or all off

Logic can be coded around injected faults

e.g. InputStream.read()• Throw IOException after n bytes are read• Return -1 (EOF) one byte before the actual EOF• Sporadically mutate the read bytes

Beyond Fault Injection

The techniques that I’m going to demonstrate aren’t specific to injecting faults (i.e. conditions that may lead to a failure)

• “What if” analysis• Test difficult corner cases

e.g. some optimization routine runs when certain conditions are met

• Initialize application state to some desired value• etc.

The techniques are similar to mock objects or IoC but much more granular

Summary

Intercept operations and inject code to:

• Perform a custom function• Return a custom value• Modify a input parameter• Throw an exception• Set the state of an object to a specific value

Checkpoint

Done:• Quick Introduction to Fault Injection• Why Use Fault Injection• Fault Injection Examples• Beyond Fault Injection

Next:• Quick Introduction to AOP• Fault Injection in Java

• AspectJ• Javassist

• Wrap Up

Checkpoint (cont)

Questions?

Aspect Oriented Programming 101

Join Point

a point in the flow of a running program [14]

It is the “where” in AOP

• Not all AO environments support all join pointse.g. Spring AOP only supports method-based join

points

Aspect Oriented Programming 101 (cont)

Join point examples

• reading or writing a field• calling or executing a method or constructor• catching or throwing an exception

A “join point shadow” is the location of a join point in the source code or bytecode of the program [5]

We will use “join point” and “join point shadow” interchangeably

Aspect Oriented Programming 101 (cont)

Pointcut

a set of join points [6]

Effectively a query where the join points are the data that is being queried – the “which”

Aspect Oriented Programming 101 (cont)

Advice

code that executes at each join point picked out by a pointcut [7]

Advice is the code that you’re injecting -- the “what”

Aspect Oriented Programming 101 (cont)

Aspect

a logical collection of advice

An aspect is to AO what an object is to OO

Aspect Oriented Programming 101 (cont)

Introduction (aka Mixin or inter-type declaration)

advice that provides additional functionality to a class

e.g. methods or member variables can be added via an Introduction to the “interface” of a class

Aspect Oriented Programming 101 (cont)

Weaving

the process by which specific join points are identified by pointcuts and advice is injected into bytecode

How AOP Works

• Compile-timea specialized compiler is used to weave the source code and advice together to produce bytecode

• Post-compile (aka binary weaving)

weaves advice into existing class files modifying them as necessary

• Load-timea classloader performs binary weaving as the class files are loaded into the JVM or a Java agent is used via JVMTI (JVM Tool Interface)

• Run-timeclasses that are already loaded by the JVM are woven with aspects via JPDA (Java Platform Debugger Architecture) to replace method bodies. Dynamic proxies can also be used.

Load-Time Weaving

We’re going to focus on load-time weaving (LTW)

• Advantange (over compile-time or post-compile): does not modify classes on disk

• Disadvantage: increases application startup time

Java AOP• AspectJ (http://www.eclipse.org/aspectj/): Excellent all-around solution.• dynaop (https://dynaop.dev.java.net/) Bob Lee’s stripped down AOP:

Requires proxy creation in code (i.e. not 100% transparent).• CGLIB (http://cglib.sourceforge.net/): a roll-your-own solution using

proxies. Requires proxy creation in code (i.e. not 100% transparent). For load-time weaving a custom class loader would be needed.

• BCEL (http://jakarta.apache.org/bcel/): roll-your-own solution requiring knowledge of Java byte code to write advice. For load-time weaving a custom class loader would be needed.

• Javassist (http://www.csg.is.titech.ac.jp/~chiba/javassist/): A large step above CGLIB / BCEL in that it includes a stripped down compiler and modifies the advised class on the fly. Load-time weaving is supported.

• Spring AOP (http://www.springframework.org/docs/reference/aop.html): Limited to method-based joinpoints. If using the @AspectJ annotations then it effectively becomes AspectJ otherwise standard Spring-style XML configuration. JDK dynamic proxies or CGLIB for implementation. If you’re already using Spring and only need to advise coarse-grained objects (i.e. the objects that you’re IoC’ing) and method-based jointpoints are suitable then this is a good solution.

• JMangler (http://roots.iai.uni-bonn.de/research/jmangler/): A step above BCEL / CGLIB but a step below Javassist. Requires knowledge of Java byte code (it uses the BCEL API for advice). Load-time weaving is supported.

• JBoss-AOP (http://labs.jboss.com/portal/jbossaop/): Effectively on par with AspectJ. If you’re using JBoss then this would be the best solution. Even has support in their IDE.

Questions?

AspectJ

• AspectJhttp://www.eclipse.org/aspectj/

• Documentationhttp://www.eclipse.org/aspectj/doc/released/

• AspectJ FAQ http://www.eclipse.org/aspectj/doc/next/faq.html

• AspectJ Development Environment Guide http://

www.eclipse.org/aspectj/doc/next/devguide/index.html

• AspectJ Programming Guidehttp://www.eclipse.org/aspectj/doc/released/progguide/

index.html

• AspectJ Quick Referencehttp://www.eclipse.org/aspectj/doc/released/quick5.pdf

Load-Time Weaving AspectJ

Load-time Weaving can be accomplished via [13]:

• -javaagent:pathto/aspectjweaver.jar• aj.bat• Custom classloader (WeavingURLClassLoader)

Both require a META-INF/aop.xml to be located in the classpath

AspectJ Examples

There are three primary classes in the example:

• Main• Creates and calls the other classes

• ReadFile• Accepts a File on construction• displayExistence() displays if the file exists• byte[] getBytesFromFile() returns the contents

of the file as a byte array• TemporaryFile

• readWriteTemporaryFile() • Creates a temp file• Writes data into it• Reads contents line-by-line• Reads contents byte-by-byte• Deletes the temp file

AspectJ Pros

• Quite mature

• Integration with IDEs (e.g. AJDT)

• Responsive forums

• Does 95% of what you need

AspectJ Cons

• Expect to burn a lot of time figuring out the difference between call and execute, within and withincode, etc.

• Can add quite a bit of time to incremental compiles (Also see bugs #146071 and #145438)

• It’s very easy to confuse the IDE• No access to local variables• You have to be careful about other errors in your

application. A compile error in some completely unrelated file may cause errors in aspects.

• Brittle with respect to refactoring especially in the case of load-time weaving. If using AJDT with compile-time weaving then refactorings apply across aspects.

Javassist

• Javassisthttp://www.csg.is.titech.ac.jp/~chiba/javassist/

• Javadochttp://www.csg.is.titech.ac.jp/~chiba/javassist/html/

index.html

• Tutorialhttp://www.csg.is.titech.ac.jp/~chiba/javassist/tutorial/

tutorial.html

Javassist Example

Javassist Pros

• Provides a relatively simple interface to do fault injection at the level of the bytecode

• Provides a limited compiler to take away the need to work directly in bytecode (e.g. BCEL)

Javassist Cons

• Limited compiler

• There’s much that you need to roll in order to use it

• Confusing code fragment language

Future Work

• Simplify fault injection + JUnite.g. run a set of test cases with different faults

• Library of faults for java.* and javax.*

• Profiler for java.* and javax.* faults

• Encourage fault libraries and profilers for common 3rd party libs

Questions?

References[1] Harbulot, B. & Gurd, J. (2005). A join point for loops in AspectJ.

http://www.cs.iastate.edu/~leavens/FOAL/papers-2005/harbulot-gurd.pdf

[2] http://www.eclipse.org/aspectj/doc/released/faq.html#q:joinpoints[3] http://www.aspectprogrammer.org/blogs/adrian/2004/06/go_from_stateme.html[4] Ossher, H. & Tarr, P. Operation-Level Composition: A Case in (Join)

Point http://www.ccs.neu.edu/research/demeter/SOP/Tarr.pdf

[5] Hilsdale, E. & Hugunin, J. Advice Weaving in AspectJ http://hugunin.net/papers/aosd-2004-cameraReady.pdf

[6] http://en.wikipedia.org/wiki/Pointcut[7] http://www.eclipse.org/aspectj/doc/released/faq.html#q:advice[8] Hsueh, M, Tasi, T, & Iyer, R. Fault InjectionTechniques and Tools.

IEEE CS Press, 1997, pp. 75-82[9] Clark, J. & Pradhan, D. Fault Injection. IEEE CS Press, 1995, pp. 47-56[10] http://en.wikipedia.org/wiki/Fault_%28technology%29[11] http://blogs.msdn.com/bwill/archive/2003/09/23/51169.aspx[12] Espak, M. Improving Efficiency by Weaving at Run-time

http://www.old.netobjectdays.org/pdf/03/papers/ws-yrw/415.pdf[13] http://www.eclipse.org/aspectj/doc/released/devguide/ltw-

configuration.html[14] http://en.wikipedia.org/wiki/Join_point

Thank You

Rob Grzywinskihttp://www.realityinteractive.com/rgrzywinski/rgrzywinski@yahoo.com