Understanding Quality Understanding Quality through the Consumer's through the Consumer's

EyesEyes

Reality Unveiled During PAS UATReality Unveiled During PAS UAT

Michelle DunnMichelle Dunn

Calgary, AlbertaCalgary, Albertamandjdunn@shaw.ca

IntroductionIntroduction

Software development teams Software development teams typically never use the typically never use the application they are responsible application they are responsible for creating. for creating.

Join me in the journey of Join me in the journey of discovering what happens after discovering what happens after the software vendor has released the software vendor has released their product.their product.

Ready … Set … Ship!Ready … Set … Ship!

Your company is now ready to Your company is now ready to deliver a new software product to deliver a new software product to the customer. the customer.

You have met the schedule, the You have met the schedule, the scope and the quality scope and the quality requirements of the contract.requirements of the contract.

… … More Testing???More Testing???

You have completed various forms of You have completed various forms of testing throughout the SDLCtesting throughout the SDLC

Your daily automated suite containsYour daily automated suite contains– 20,000+ JUNIT tests 20,000+ JUNIT tests – 1,000 GUI (front end tests) 1,000 GUI (front end tests) – 350 System tests 350 System tests – 100 Report tests 100 Report tests

So why does your customer still feel So why does your customer still feel the need to do rigorous User the need to do rigorous User Acceptance Testing? What is UAT?Acceptance Testing? What is UAT?

Why the Secrecy Why the Secrecy Surrounding UAT?Surrounding UAT?

Although most large companies perform Although most large companies perform UAT prior to deploying an application, UAT prior to deploying an application, there is limited articles/books published there is limited articles/books published on the subject by leading Test Experts.on the subject by leading Test Experts.– I could not find one reference to UAT in my I could not find one reference to UAT in my

favorite book “Lessons Learned in Software favorite book “Lessons Learned in Software Testing” by James Bach.Testing” by James Bach.

– The bestselling test book of all time “Testing The bestselling test book of all time “Testing Computer Software” by Cem Kaner has only Computer Software” by Cem Kaner has only one paragraph on the subject.one paragraph on the subject.

If it’s not talked about, then it cannot be If it’s not talked about, then it cannot be important … right?important … right?

UAT RedefinedUAT Redefined

A Common Definition (coined by A Common Definition (coined by me!)me!)– ““UAT is performed to determine if UAT is performed to determine if

the application will support the the application will support the needs of the user in the real world.”needs of the user in the real world.”

What does this mean?What does this mean? Let’s probe further and dissect Let’s probe further and dissect

the term UATthe term UAT

UUAT: Who is the AT: Who is the ““UserUser””

Who will be affected by the Who will be affected by the project/application?project/application?– Project TeamProject Team– Support and Operation TeamSupport and Operation Team– End UserEnd User– End User’s customerEnd User’s customer– Legal or regulatory authorityLegal or regulatory authority– Monetary authorityMonetary authority– Anyone else?Anyone else?

UUAAT: What is T: What is ““AcceptanceAcceptance”?”?

““Acceptance Testing is any testing Acceptance Testing is any testing done by one party for the purpose done by one party for the purpose of accepting another party’s work”of accepting another party’s work”– Michael Bolton, 2001 Michael Bolton, 2001

What testing needs to be performed What testing needs to be performed to determine if the software to determine if the software application is acceptable?application is acceptable?

UAUATT: What is “: What is “TestingTesting””

– FunctionalityFunctionality– RegressionRegression– ScenarioScenario– ReliabilityReliability– End to EndEnd to End

– UsabilityUsability– SecuritySecurity– ScalabilityScalability– PerformancePerformance– And more “ility” TestsAnd more “ility” Tests

““Testing is questioning a product in order Testing is questioning a product in order to evaluate it” to evaluate it” – James Bach, “Rapid Software Testing”, 2007James Bach, “Rapid Software Testing”, 2007

Test Techniques:Test Techniques:

What is Your What is Your Company’s UAT Goal?Company’s UAT Goal?

UAT Goal: Beta UAT Goal: Beta ReleaseRelease

User base tests the applicationUser base tests the application– Example: Book4GolfExample: Book4Golf– Rigorously tested by the software Rigorously tested by the software

development teamdevelopment team– Mass software release where users were Mass software release where users were

asked to testasked to test– Beta users had no motivation to test, thus Beta users had no motivation to test, thus

only usability bugs were communicated to only usability bugs were communicated to teamteam

– UAT: UAT: UUser: Public Tested (no test experience) ser: Public Tested (no test experience) AAcceptable: Acceptable to the Shareholderscceptable: Acceptable to the Shareholders TTesting: No Testing Performedesting: No Testing Performed

UAT Goal: Rubber UAT Goal: Rubber StampStamp

Demonstration of product to usersDemonstration of product to users– Example: NetellerExample: Neteller– Rigorously tested by software Rigorously tested by software

development teamdevelopment team– UAT Performed the day before releaseUAT Performed the day before release

30 Minute Scripted Test Performed by 30 Minute Scripted Test Performed by Customer SupportCustomer Support

Demo to the Product Managers Demo to the Product Managers – Bugs found would not stop the releaseBugs found would not stop the release– UAT: UAT:

UUser: Customer Support, Product Managers ser: Customer Support, Product Managers AAcceptable: Yes, Acceptable to the cceptable: Yes, Acceptable to the

ShareholdersShareholders TTesting: Scripted and Demo onlyesting: Scripted and Demo only

UAT Goal : Thorough UAT Goal : Thorough Form of TestingForm of Testing

Test the application End to End Test the application End to End – Example: PASExample: PAS– Rigorously tested by Development TeamRigorously tested by Development Team– UAT performed 8+ months before Go LiveUAT performed 8+ months before Go Live

Performed by SME and trained TestersPerformed by SME and trained Testers Test to validate functionality and to find bugsTest to validate functionality and to find bugs Bugs found could stop the releaseBugs found could stop the release

– UAT: UAT: UUser: All users that would be touched by ser: All users that would be touched by

application application AAcceptable: Yes, Acceptable to the Advisory cceptable: Yes, Acceptable to the Advisory

BoardBoard TTesting: All forms of testingesting: All forms of testing

Why Are UAT Goals so Why Are UAT Goals so Different?Different?

Mandate to use application? Mandate to use application? Mission Critical application?Mission Critical application? Size and Complexity of Size and Complexity of

application?application? Risk to Organization?Risk to Organization? Risk to Life?Risk to Life? Monetary Risk?Monetary Risk?

Concerns a Customer Concerns a Customer HasHas

1.1. Will the client deliver the application on Will the client deliver the application on time, in scope and with quality?time, in scope and with quality?

2.2. Will the application work with my Will the application work with my infrastructure, with my data load and with infrastructure, with my data load and with my user load?my user load?

3.3. Will the application integrate with interfaces Will the application integrate with interfaces and secondary applications easily? and secondary applications easily?

4.4. Will I meet my business timeframes and Will I meet my business timeframes and regulatory requirements?regulatory requirements?

5.5. Will the Service Level Agreements be met?Will the Service Level Agreements be met?6.6. Will the Application Upgrade without data Will the Application Upgrade without data

corruption? Will functionality regress?corruption? Will functionality regress?

Application UsageApplication Usage

Naming Standards

Business Processes

People

Training

The Application is One The Application is One Piece of the PuzzlePiece of the Puzzle

Application

Infrastructure

Data Interfaces

What is UAT Test What is UAT Test Coverage?Coverage?

““Test Coverage is the proportion of Test Coverage is the proportion of the product that has been tested”the product that has been tested”

Additionally Test Coverage for UAT Additionally Test Coverage for UAT not only includes testing the not only includes testing the application but also:application but also:– Testing using real environments, with Testing using real environments, with

real data, with real business scenarios , real data, with real business scenarios , with real userswith real users

– End to End testing from “cradle to End to End testing from “cradle to grave” grave”

Determining your UAT Test Coverage

ASK: What is the complete picture?

1. Application2. Interfaces3. Environment4. Data5. Performance6. Security

Determining your UAT Test Coverage Cont’d

Application– Business Functionality

Business Processes Application Functionality (80% rule) Complex Functionality

Interfaces – Push to or Pull from (Applications,

Databases, Secondary Applications, Reports, Etc.)

Environment – Platforms /Configuration / Infrastructure

Determining your UAT Test Coverage Cont’d

Data – Data Characteristics – Naming Standards

Performance – Data Load – User Load – Timeframes

Security

What is UAT Risk What is UAT Risk Based Testing?Based Testing?

Testing the real and potential risks that threaten the quality of the application

Defining Test RiskDefining Test Risk

Using the Test Coverage Matrix Using the Test Coverage Matrix created, ask the Businesscreated, ask the Business– What Must Work?What Must Work?– What Do You Value?What Do You Value?– What Threatens the Value of the What Threatens the Value of the

Product?Product?– Brainstorm functional, technical, Brainstorm functional, technical,

performance, security risks that the performance, security risks that the test coverage matrix may have missedtest coverage matrix may have missed

Categorize the Test Categorize the Test RiskRisk

High, Medium, LowHigh, Medium, Low How bad would it be if this How bad would it be if this

problem occurred? problem occurred? What are the chances that this What are the chances that this

problem can occur?problem can occur? Has it been tested by the Has it been tested by the

Development Team?Development Team? New or Legacy Application?

Create Tests to Exploit Risk

What tests must be run to exploit What tests must be run to exploit these risks?these risks?

What business processes, data, What business processes, data, infrastructure, and interfaces infrastructure, and interfaces must be used in the test?must be used in the test?

Continually Ask the Test Team Continually Ask the Test Team ““Are we Testing the Right Stuff the Are we Testing the Right Stuff the

Right Way?”Right Way?”

Finding UAT Defects

All High Severity bugs found will not be fixed

Bugs that are a nuisance in testing become issues in PROD where:– Workarounds need to be developed– Training needs to be incorporated– Users must live with the bug until the

bug is fixed and that release is in PROD

UAT Regression UAT Regression TestingTesting

Test Risk evolves once the application Test Risk evolves once the application is Live and in Productionis Live and in Production

Review Test Risks from before and Review Test Risks from before and question the test’s current valuequestion the test’s current value– Why repeat the test? Why repeat the test? – What is it’s return on investment (ROI)?What is it’s return on investment (ROI)?– Can the test be run the same way?Can the test be run the same way?

Add new tests to exploit new risksAdd new tests to exploit new risks– Data Migration and Data IntegrityData Migration and Data Integrity– New Functionality and Bug FixesNew Functionality and Bug Fixes

Questions

?

AcknowledgementsAcknowledgements

Sabina FabbianSabina Fabbian