Ideas for Rapid Test Management

Ideas for Rapid Test Management

Jon Bach

QA Manager

2/19/09

SASQAG

What is Rapid Testing?

“The skill of testing any software, any time,

under any conditions, such that your work

stands up to scrutiny. The closest thing in the

business to a martial art of software testing.”

Skills

Some Exploration Skills and Tactics

Modeling

Resourcing

Questioning

Recording

Reporting

Chartering

Observing

Manipulating

Pairing

Generating/Elaborating

Refocusing

Alternating

Branching/Backtracking

Conjecturing

“MR.Q COMP GRABC R&R?”

• More time than ALL of your time

• Everything is pri 1

• Multi-tasking

• No specs

• No time for training

• Cubicles / team doesn’t share info with each other

• Heavy tools

• Interruptions

Conditions

Scrutiny

• Programmers

• Business Analysts

• Test Managers

• Directors

• Customers

• Other Testers

• You

Backstory

• Start Date

• Projects

• Staff

• Mission

• Values

Problems

• Who’s on your team?

• How do you find time to learn?

• How does exploration fit?

• What’s your test approach?

• What are you doing right now?

• Where are we? How we doin’?

• How did you do that?

• Can we do this more smartly?

Ideas

• Surveys

• Open-Book Testing

• Dawn Patrol – SBTM

• Color-aided Test Design

• Testing Pyramid

• Hybrid

• Start / End daily status

• Drive-by learning

• Lightning talks

• Tools: WME, SWP, PK

Ideas

•Surveys• Open-Book Testing

• Dawn Patrol – SBTM

• Color-aided Test Design

• Testing Pyramid

• Hybrid

• Start / End daily status

• Drive-by learning

• Lightning talks

• Tools: WME, SWP, PK

Surveys…

•What time works for you for a one-on-one with me this week?

•Do you have any ideas of how to dogfood Concordance?

•What are your working hours?

•What’s you biggest annoyance when working?

•What’s the name of your spouse / children?

•What are 3 aspects of your ideal manager?

•What’s the best business / testing book you’ve read?

•What’s the best overall book you’ve read?

•What one project problem do you wished was solved NOW?

•What aspect of working here is most important to you?

Survey Says…

Team meeting consisting of revealing the answers together -- Family Feud style

Speed round…

1) Name a LexisNexis customer that tends to be mentioned a lot in meetings.

2) Jon has one full row of books on the bookshelf in his cube (standard size). How many books are on it?

3) Name the genre of book that most of the staff says they like.

4) Name the most interesting project codename we’ve used (any project).

5) Name a popular first name for staff on all four project teams.

6) Name a test management tool we use.

7) Name a neighboring state to Ohio.

8) What would you say is the average amount of emails each person on Jon’s staff gets per day?

9) Name another state (other than Washington) where LexisNexis Home-Based Employees (LAW, Concordance, TL, or CaseMap) are based?

10) Name a resolution category for a bug that would be assigned back to you.

Ideas

• Surveys

•Open-Book Testing• Dawn Patrol – SBTM

• Color-aided Test Design

• Testing Pyramid

• Hybrid

• Start / End daily status

• Drive-by learning

• Lightning talks

• Tools: WME, SWP, PK

What is Open-Book Testing?

The act of assigning testers and students open-ended questions such that…

Testers Instructors and managers

…are immersed in the product right away, building a model or mind map.

…create test “charters”, or missions that seek to reveal many bugs instead of one (akin to using scenarios and personas).

…learn how they are provoked into critical thinking by being exposed to many types of questions (test ideas).

…learn how their students are provoked to know if they can perform critical thinking.

… quickly find bugs and raise issues in answering the questions they are given

…evaluate tester skill and test coverage.

Excerpts from Flight Sim Ground School exam:

• Which airplanes are equipped with slotted flaps?

• Describe the steps needed to start the engine of the DC3.

• If the vacuum system failed in the 172, what instruments would no longer function?

• Joe, an aviation history fanatic, buys Combat FS and complains to tech support that the P-51 flight model is “all wrong” because in a dogfight, “the airplane stalls at 200 MPH.” Joe flies with the Realism Settings set to Hard and he knows that the P-51 should stall at about 100 MPH. Explain to Joe how it is possible for an airplane to stall at twice its published stalling speed.

Closed-book exams test knowledge.

Open-book exams test resourcefulness.

“Imagination is more important than knowledge” -- Einstein

The answer may matter less than how it was derived.

For example,

Which airplanes are equipped with slotted flaps?

…can lead to the following strategies to find the answer:

• Select aircraft / spot plane view should show me the type

• Aircraft details: should tell me the type of flaps

• Flap function (shortcut key, panel icon, keyboard) can be tested to verify the flap animation is consistent with model

• Flight model affect (whether the flaps behave as slotted) can be tested in flight

OBT question The spirit of the test

Which airplanes are equipped with slotted flaps?

Confirm or refute a rumor that the texture maps for slotted flap animations weren’t ready, but were dropped into the build anyway

Describe the steps needed to start the engine of the DC3.

Investigate a claim from the Developer, a DC3 pilot, that omissions in the “pilot checklist” feature are now fixed

If the vacuum system failed in the 172, what instruments would no longer function?

When enabled, does the “system failures” feature show that the altimeter and airspeed indicators are incorrect?

Can a 737 stall at a speed that’s twice as fast as its published stalling speed?

Any plane should stall at any speed if the attitude of the angle of attack is too steep.

Ideas

• Surveys

• Open-Book Testing

•Dawn Patrol – SBTM• Color-aided Test Design

• Testing Pyramid

• Hybrid

• Start / End daily status

• Drive-by learning

• Lightning talks

• Tools: WME, SWP, PK

Dawn Patrol

A SWAT team of hardcore testers who do exploratory testing when no one is looking.

Jon Bach / Quardev Laboratories, Seattle, WA 98119

Some Exploration Skills and Tactics

Modeling

Resourcing

Questioning

Recording

Reporting

Exploratory testing is a mindset using this skillset.

Chartering

Observing

Manipulating

Pairing

Generating/Elaborating

Refocusing

Alternating

Branching/Backtracking

Conjecturing

“MR.Q COMP GRABC R&R?”

Jon Bach / Quardev Laboratories, Seattle, WA 98119

Scrutiny

What’s being asked What they may be thinking

What was your mission? Remind me what I told you to do.

Why did you do that?

How did it go? Were you careful or reckless?

What should I be worried about?

How far did you get? Are we closer to shipping?

Can you help me know our status?

Need anything? Can I speed this along?

Do you need more of my time?

When will you be done? Will I get my bonus?

I have a new task for you…

Jon Bach / Quardev Laboratories, Seattle, WA 98119

Session-Based Test Management:

1) Time Box

2) Reviewable Result

3) Debriefing

The “Session”

Jon Bach / Quardev Laboratories, Seattle, WA 98119

The deliverable: a session report

• Charter– #AREAS

CHARTER-----------------------------------------------Analyze MapMaker’s View menu functionality and report on areas of potential risk. #AREASOS | Windows 2000Menu | et-jsb-010417-c.sesViewStrategy | Function TestingStrategy | Functional Analysis SampleSTART-----------------------------------------------5/30/00 03:20 pm TESTER-----------------------------------------------Jonathan Bach TASK BREAKDOWN----------------------------------------------- #DURATIONshort #TEST DESIGN AND EXECUTION65 #BUG INVESTIGATION AND REPORTING25 #SESSION SETUP20 

Metrics– #DURATION– #TEST DESIGN AND EXECUTION– #SESSION SETUP– #BUG INVESTIGATION AND REPORTING– #CHARTER / OPPORTUNITY

Issues– #ISSUE

Bugs– #BUG

Notes

Jon Bach / Quardev Laboratories, Seattle, WA 98119

Are you accountable?

Was this mission reasonable?

What did you find?

How did you spend your time?

Did you need some help / tools?

Do you think there’s more to do here?

Agenda: “PROOF”Past Results Obstacles Outlook Feelings

Jon Bach / Quardev Laboratories, Seattle, WA 98119

Sample project

HTML status report

Ideas

• Surveys

• Open-Book Testing

• Dawn Patrol – SBTM

•Color-aided Test Design• Testing Pyramid

• Hybrid

• Start / End daily status

• Drive-by learning

• Lightning talks

• Tools: WME, SWP, PK

Jon Bach / Quardev Laboratories, Seattle, WA 98119

Color-aided Test Design

•Color-aided test design is a method of test creation which aims at giving value by shaping, finding relevance and organizing tests based on over arching and broad testing concepts.

•By categorizing tests at a high level, test concepts designated by color, we can quickly and easily focus our aim and maximize reliable test coverage in a collaborative method to expose risks in the test plan.

Jon Bach / Quardev Laboratories, Seattle, WA 98119

Jon Bach / Quardev Laboratories, Seattle, WA 98119

Color-aided Test Design

Ideas

• Surveys

• Open-Book Testing

• Dawn Patrol – SBTM

• Color-aided Test Design

•Testing Pyramid

• Hybrid

• Start / End daily status

• Drive-by learning

• Lightning talks

• Tools: WME, SWP, PK

Jon Bach / Quardev Laboratories, Seattle, WA 98119

Testing Pyramid

Ideas

• Surveys

• Open-Book Testing

• Dawn Patrol – SBTM

• Color-aided Test Design

• Testing Pyramid

•Hybrid• Start / End daily status

• Drive-by learning

• Lightning talks

• Tools: WME, SWP, PK

Jon Bach / Quardev Laboratories, Seattle, WA 98119

Hybrid

A test document that combines exploratory and matrix-driven verification testing.

Sample

Ideas

• Surveys

• Open-Book Testing

• Dawn Patrol – SBTM

• Color-aided Test Design

• Testing Pyramid

• Hybrid

•Start / End daily status• Drive-by learning

• Lightning talks

• Tools: WME, SWP, PK

Ideas

• Surveys

• Open-Book Testing

• Dawn Patrol – SBTM

• Color-aided Test Design

• Testing Pyramid

• Hybrid

• Start / End daily status

•Drive-by learning– Lightning talks

– Combo lock

• Tools: WME, SWP, PK

Tools

• ScrumWorksPro

• Windows Media Encoder

• Perfect Keyboard

• SourceForge wiki

Ideas

• ScrumWorksPro

• Windows Media Encoder

• Perfect Keyboard

• SourceForge wiki

Ideas

• ScrumWorksPro

• Windows Media Encoder

• Perfect Keyboard

•SourceForge wiki

Wiki

Recap

“The skill of testing any software, any time,

under any conditions, such that your work

stands up to scrutiny. The closest thing in the

business to a martial art of software testing.”

Problems

• Who’s on your team?

• How do you find time to learn?

• How does exploration fit?

• What’s your test approach?

• What are you doing right now?

• Where are we? How we doin’?

• How did you do that?

• Can we do this more smartly?

Bake your half-baked ideas

• Surveys

• Open-Book Testing

• Dawn Patrol – SBTM

• Color-aided Test Design

• Testing Pyramid

• Hybrid

• Start / End daily status

• Drive-by learning

• Lightning talks

• Tools: WME, SWP, PK