© Copyright 2015 Vivit Worldwide© Copyright 2015 Vivit Worldwide
“Developing a Comprehensive Test Automation QA Strategy”
August 11, 2015
© Copyright 2015 Vivit Worldwide
Brought to you by
© Copyright 2015 Vivit Worldwide
Hosted by
Robert Linton
Testing, Quality and ALM SIG Leader
VP ALM at CorTechs, Inc.
© Copyright 2015 Vivit Worldwide
Today’s Speaker
Becky Karch
Director of Customer Success
TurnKey Solutions
© Copyright 2015 Vivit Worldwide
Housekeeping
• This “LIVE” session is being recorded
Recordings are available to all Vivit members
• Session Q&A:
Please type questions in the Questions Pane
© Copyright 2015 Vivit Worldwide
Webinar Control Panel
Questions
Toggle View Window between
Full screen/window mode.
Developing a Comprehensive Test Automation QA Strategy
TurnKey Solutions
August 2015
What We Will Cover
- Assessing your current state of test automation readiness
- Optimizing test automation, people, and processes
- Benefits of data-driven, scriptless test technology
- Maintaining regression tests
- Leveraging a mix of tools and technologies in your
testing
- Quantifying value from your test automation
Polling Questions
What is the biggest challenge you face with your software delivery cycle?
Testing process is slowing us down Knowing when to release Difficulty getting the teams organized Finding the right tools to streamline the process
Delivery
Feedback
Development
The Release Cycle
Faster release cycles shorten the gap between what the customer wants and how quickly you are able to deliver
Testing is typically the #1 time sink in the delivery process –this requires getting smart about what you test and how you execute your tests
Setting Quality GoalsApplication
#Tests Planned #Executed
%Executed of Tests Planned
#Passed
%Passed of Tests Executed
%Failed of Tests Executed
%Blocked of Tests Executed
Admin Rx HH Total 97 97 100.00% 97 100.00% 0.00% 0.00%
CAF Total 3 3 100.00% 3 100.00% 0.00% 0.00%
Carelink Total 223 122 54.71% 120 98.36% 1.64% 0.00%
CTM Total 22 10 45.45% 9 90.00% 0.00% 10.00%
EDTB Total 6 6 100.00% 6 100.00% 0.00% 0.00%
eMAR Total 565 565 100.00% 523 92.57% 1.59% 5.84%
Enterprise Prescription Writer Total 522 453 86.78% 443 97.79% 1.99% 0.22%
HCR Total 6 2 33.33% 2 100.00% 0.00% 0.00%
HEC Total 425 407 95.76% 403 99.02% 0.98% 0.00%
HED Total 26 25 96.15% 25 100.00% 0.00% 0.00%
HEN Total 746 694 93.03% 649 93.52% 1.73% 4.76%
HEO Total 3955 2916 73.73% 2786 95.54% 2.40% 2.06%
HEP Total 7 7 100.00% 7 100.00% 0.00% 0.00%
HHS Total 3130 1224 39.11% 1044 85.29% 13.89% 0.82%
HMR Total 120 118 98.33% 109 92.37% 7.63% 0.00%
HMS Admin Tool Total 4 3 75.00% 3 100.00% 0.00% 0.00%
HOM Total 783 755 96.42% 755 100.00% 0.00% 0.00%
Horizon Care Alerts Total 19 18 94.74% 17 94.44% 5.56% 0.00%
HPC Total 42 33 78.57% 33 100.00% 0.00% 0.00%
Open Connect Total 34 14 41.18% 13 92.86% 7.14% 0.00%
OTK Total 1244 925 74.36% 912 98.59% 1.41% 0.00%
PRCS Total 244 173 70.90% 90 52.02% 0.00% 47.98%
Reports Total 186 56 30.11% 55 98.21% 0.00% 1.79%
Reports - Oracle Total 36 23 63.89% 23 100.00% 0.00% 0.00%
Grand Total 12456 8660 69.52% 8138 93.97% 3.46% 2.56%
Test metrics must represent overall product quality and be directly related to business goals
What Should You Test?
Regarding the application(s) under test, answer these questions:
What would cause the company to lose money? What would cause the company to lose business? What is the environment in which I need to run my tests? What types of issues are “escaping” the testing process? What are the high-risk end-to-end business processes? What are the processes that your business runs every day?
Your tests should reflect “Meaningful Use” of your System
Assess Current TestsWhat do your current tests do?How are they written and do they cover what they should cover?How stable is your application?
Develop Your Plan - Assess What You Have
Staffing The Necessary Testing ResourcesWhat skills does your team have?Can you hire more resources?Can you share/integrate roles & responsibilities?
Assess Test ToolsHow and where do you organize your tests?Do you have an automated test tool? Do you use it? How often?
Getting FeedbackWhat Support issues come in?Do you have enhancement request tracking?What services do your customers ask for?
Develop the Plan – Narrow the Focus What tests do you need to meet your quality goal?
Scope
Approach
What needs to be automated vs. run manually?
Setting up the environment
Test Data
Roles, Responsibilities & Credentials
Documentation
Segment the application into components as UI screens become available
BPT supports this well, but changes happen rapidly
Organizing for Success
Test Manager/Director
XYZ Division
Test Lead
App X
Test Automation Engineer (TAE)
App X
Test Architect
App X
Business Analyst
App X
. . .
Developer
App X
Everyone on the team must be familiar with the release in a continuous delivery model
“Day in the Life”
Bu
sin
ess
An
alys
tTe
stLe
adTA
ETe
stA
rch
itec
t
DocumentsTest
Strategy & Standards
Verifies Test Artifacts /
Compliance to Standards
Test Planning Test Design
Supports DataSheet
Entry
Creates & Unit Tests
Components
Participates in Requirements
Review
Participates in DataSheet
Entry
Assembles Test Scripts
Determines Component
Needs
Test Execution
Monitors Test
Execution
Reports Test Progress /
Defects
Document Functional
Requirements
Participates in Test Design Review
Enters Data into
DataSheets
IDs Test Risks and Issues
Coordinates Test
Estimates
Assists with Test
Estimates
Provides Tech Test
Direction to TAE
Executes Test Scripts
Opens Defects &
Retests
Maintains Components
& Tests
Ensures that ALM Test
Environment is Up-to-Date
Reviews Test Results
IDs Business Risks and
Issues
Organization-wide
• SCRUM teams form for each application or functional area
Scrum Team
Application Z
Test Lead
App Z
Test Automation Engineer (TAE)
App Z
Test Architect
App Z
Business Analyst
App Z
Scrum Team
Application Y
Test Lead
App Y
Test Automation Engineer (TAE)
App Y
Test Architect
App Y
Business Analyst
App Y
Scrum Team
Application X
Test Lead
App X
Test Automation Engineer (TAE)
App X
Test Architect
App X
Business Analyst
App X
• META SCRUM meets to ensure “meaningful use” end-to-end flows
Building the Regression Suites
Release CRelease BRelease A
Automation
A
Regression
A
Regression
A
Regression
B
Automation
B
Final Release to Production
Regression
A
Regression
B
Regression CAutomation
C
Continuous Testing
How Scriptless Automation Can Help You Use the people in your organization who know the end-to-end business
flows (“meaningful use”) the most
Break up your application into components (or screens) that can be
assembled into tests and reused as needed
Allows you to run end-to-end tests
within a single application
or spanning multiple applications
Keeps your data organized and attached to the relevant tests
Allows you to easily maintain tests when things change
Maximizes your testing time to get to faster release
Continuous Test Development
Up
Dev
.Te
st/Q
AB
iz A
nal
yst
Sprint 1 Sprint 8Sprint 2 Sprint 3 . . .
• Analyze requirements• Develop first screen• Create first component
• Develop additional screens
• Develop additional components
• Begin to assemble test
• Add underlying structure
• Change interface for one screen
• Update components and data
• Execute tests • Debug tests• Fix product bugs
Requirements1 – display this2 – enter xyz in field13 – Navigation links on left4 – graphic must be centered
Test PlanTest 1: do thisTest 2: then do thatTest 3: try to break ITest 4: do it again X5
Maintaining Your Tests
Tests must be easy to maintain without re-coding and debugging
Application-aware test tools make this easier Update components and not entire test scripts Side-by-side comparison of new, deleted, or changed UI
fields Automatically identify changes in your data sheets
Building End-to-End “Meaningful Use” Tests
Requisition Requisition Summary
Individual data scenarios within your data sheet
Value from AutomationTestsPlanned
TestsExecuted
PercentTests Executed
Tests Passed
Percent of Executed TestsPassed
Percent of Executed Tests Failed
Percent of Planned Tests Passed
TOTAL 5514 5306 96.23% 5222 98.42% 1.58% 94.70%
Automating “Meaningful Use” gives your organization confidence in product quality
Regression test sets build over each sprint and release With test automation you can measure the time to
release savings Measure efficiency of scrum teams by their velocity and
defect backlog
Continuous Delivery Process
Application Development
Requirements Development
System TestingAutomated
Testing
Requirements Tracking
Automated Build
Production Release
Leverage a Mix of Tools
Agile user story tracking
Version Control System
Automated Build System
Defect tracking
Test management tool
Test automation tool
Data management tool
Use Customer Feedback in the SDLC
Measure “escapes” from each release Support Tickets User Forums
Customer Satisfaction Survey information Customer Advisory Boards
© Copyright 2015 Vivit Worldwide
Thank you
• Complete the short survey and opt-in for more information from
TurnKey Solutions.
www.turnkeysolutions.com
www.vivit-worldwide.org