1

Steve Chenoweth

Tuesday, 10/04/11

Week 5, Day 2

Right – Typical tool for reading out error codes logged by your car’s computer, to help analyze its problems. From http://niceparts.blogspot.com/2007_12_01_archive.html.

CSSE 377 – Intro to Testability

2

Today

• How to do Project 4…– We’ll get your “implementers” a bit more

involved this time

• Software testability – this – Bass’s Ch 4, pp 88-90 (Testability

scenarios) & Ch 5, pp. 118-121 (Testability tactics)

– Lots of discussion also in CSSE 376!

Q 1

3

Coming up…

• Thursday – (Steve’s at Purdue)– Guest speakers – Matt Ellis, Justin Hutchings of

Microsoft– Remaining time – if any – Work on first turnin for

Testability

• Friday – More on this & time to work on Project 4 – including…– Getting your “implementers” involved – first talk with

them about “what should be more testable” on your system, then have them try to run tests themselves.

• Monday: Analyzing archs & the ATAM / CBAM

4

We next pick testability from Bass’s QA list…

• Bass’s list of six, from the inside back cover of his book:– Availability– Modifiability– Performance– Security– Testability– Usability

5

And here’s the project about it:

Overall - On the same project,• Use an arch tactic to make it more testable:

– Determine what is not very testable in your current project system.

– Implement a tactic to improve this by a predicted amount!– Have your implementers observe the “before” and “after” testing,

and confirm that there’s an improvement• We’re also going to continue our study of how to

document the arch, so:– An added feature of this project – add one more section to your

draft of that, for your sys.

6

And a first step to take:

• Try to think of a problem area in testing your current system.• Run through this area of tests, of your system as it exists

now. Time how long this takes, and find a way to measure how “effective” (thorough) it is.

• See if you can get your “implementers” to repeat the same tests.

• Make a “prediction” about how fast or effective this testing will be with the testing change you plan to make.

• Turn in your ideas, those “existing times,” and other measures, with your prediction, in your “team journal” by Friday at 11:55 PM.

Next step will then be – Make the changes to testing that will speed this up or make it more effective.

7

What’s Bass say about this QA?

• Problem – Testing needs to be more efficient & effective– It consumes a high percentage of software development time &

effort– The heuristic is “half”

• Goal – Allow easier testing when an increment of software development is completed.

• Motivation – The overall arch of the system, and of its testing methods, can help make testing easier

• Scenarios – What’s in “The Notes” at the end of the supplementary spec template?

• What is Testability “about” – Ch 4?• What are some good tactics – Ch 5?

Q 2

8

Bass’s testability scenarios

• Source: Unit developer– Increment integrator– System verifier– Client acceptance tester– System user

• Stimulus: Analysis, architecture, design, class, subsystem integration completed; system delivered

• Artifact: Piece of design, piece of code, complete application• Environment: At design time, at development time, at compile time, at

deployment time• Response: Provides access to state values; provides computed values;

prepares test environment• Response Measures: (examples)

– Percent executable statements executed– Probability of failure if fault exists– Time to perform tests– Length of longest dependency chain in a test– Length of time to prepare test environment

Q 3

9

Example scenario

• Source: Unit tester• Stimulus: Performs unit test• Artifact: Component of the system• Environment: At the completion of the component• Response: Component has interface for controlling

behavior, and output of the component is observable• Response Measure: Path coverage of 85% is achieved

within 3 hours

10

Testability situations

It’s a development-time attribute: • Probability of fault discovery• Need to control

components• Need to observe

component failure

Ch 4

Right: Ren & Stimpy tell why people ignore system testability. From web site sctest.cse.ucsc.edu/ .

11

Tactics to achieve testability

• Manage Input/Output– Record/playback– Separate interface from implementation– Specialize access

• Internal monitoring– Built-in monitors

• Generally, the goal is to automate as much testing as you can – why?

Ch 5Q 4

12

Examples - 1

• Test harness – like the stimulator you might have done for performance– Automates some aspect of testing– But, for finding errors, the test cases need to

be more complete (unless performance / reliability are what you’re testing!)

– Can be for internal classes, etc., as well as external – See next slide

– Give this itself an “architecture” with options, different modes, etc.

Q 5

13

Examples – 1, cntd

• Typical test harness:• In MVC, replace the “View” with a stimulator that

fires test cases at the rest of the system:

Replace withstimulator, fortesting

14

Examples – 1, cntd

• How about stimulating the GUI?• A slightly harder trick, replacing the human with

the mouse and keyboard.

How do we testthe GUI itself?(Or the wholeSystem!?)

See list of open source tools at http://en.wikipedia.org/wiki/List_of_GUI_testing_tools

15

Examples - 2

• Built-in monitors – the system itself is “instrumented” to provide test results. – Usually, this can be turned on/off so it doesn’t

interfere with normal operations.– May have multiple special modes.– The “heart” of white-box testing!

16

Industrial strength tools

• Good example – IBM’s Rational PurifyPlus– Shows code coverage– Analyzes memory problems (e.g., buffer

overflow, leaks)– Identifies performance issues like bottlenecks– Runs on Linux, Unix, Windows

See http://www-01.ibm.com/software/awdtools/purify/?cn=agus_rtnlrp-20100514&cm=k&csr=google&cr=rational_purify&ct=AGRAK605&ck=rational_purify&mkwid=s28L9ro6h_5281300654_432dt32930&cmp=109HG

17

Project 4 – Testability (overall)

Overall theme (of the testability part of the project) – • Pick an area where you want to improve testing.• Pick a strategy to improve it, using one of the above tactics, and

verify (Friday) with your implementers. The improvement can be in either of these two areas:– Effectiveness (thoroughness) of the testing, or– Speed of the testing, with the same effectiveness.

• Try it “as-is” to verify how long it takes and the coverage.– Includes letting novice testers, your “implementers,” try it.

• Guess how much you can improve that with your change.• Make the change to improve testability.• Run the testing again, with the change. Measure how fast and how

effective the new testing is. Let your implementers verify the differences.

• Report on the results.