Agile Testing Analytics
STARWEST 2016
Presenter
Jonathan Alexander CTO of QASymphony
SDLC
QASymphony - A Robust Testing Platform
Monitor PPM
Require-ments
Build
Test
Defects
Deploy
Automation
Our Solutions
For Test Case ManagementqTest is a powerful,
elegant, scalable test management solution that works with small
teams and allows large enterprises to coordinate
and track hundreds of projects across many
locations.
For Unscripted Testing
qTest eXplorer is a ground-breaking test
documentation tool that supports exploratory
and unscripted testing AND saves time when performing traditional
manual testing.
For Enterprise Reporting
Insights 2.0 gives the testing team a self-
service business intelligence tool to
consolidate, manage and analyze all your
testing data.
Our Customers
In This Session
Analytics - What and Why
Quality Analytics
Coverage and Risk Analytics
Velocity Analytics
Test Case Optimization
How to Get Started
Analytics - Data Sources
Many potential data sources
Need to create links: dev tickets -> code check-ins, test cases, defects, support cases
The Goal: Leverage Data for Improvement
● Use analytics to improve:
○ Test coverage
○ Forecasting completion dates
○ Efficiency and most effective use of resources
○ Test case quality
○ Productivity
● Think of analytics as an objective input to the planning process
Quality Analytics
● Core:○ Test result %s by project/release
■ separate out latest runs○ Defect priority and status %s
● Extra:○ Test results by day or week○ Defect status/priority crosstab○ Defects per test run○ Defect leakage (found by users)
● Tips:○ Use color-coding to identify potential issues○ Put manual and automated results side-by-side
Coverage Analytics● Core:
○ Test cases by requirement■ Latest run results■ Breakdown by type
○ Defects by requirement
● Extra:○ Test case complexity○ Test time per requirement○ Last date of test run(s)
● Tip:○ Use data visualization to spot risks
Velocity Analytics● Core:
○ Requirements inflow rate○ Test case creation rate○ Test run rate (cases & steps)○ % tests complete and blocked○ Defects opened and closed
● Extra:○ Avg. and total testing time spent○ Mean time to test(s) created, run, passed○ Forecast time and defects remaining
● Tip:○ Breakdown analytics by tester
Test Case Optimization● Start to think of test cases like source code
● Track manual and automated test cases, exploratory scripts
● Track analytics that will help optimize your test case library○ Days since last run
■ Archive test cases that are not used anymore○ Flapping (# of times consecutive runs have different results)
■ Examine these tests and code/functional areas, might indicate need to refactor one or the other
○ Percentile complexity (steps and time spent per test)■ Refactor highly complex tests for greater efficiency and
more pinpoint understanding of results○ Cumulative execution time
■ Automate the manual tests that are taking the most time■ Refactor automated tests that are running the longest
How To Do This Yourself● Setup a Test Analytics Reporting Server
○ Use an open source or 3rd party BI tool (such as qTest Insights)○ Keep it simple
● Identify team members that will Work on Test Analytics○ Depending on tool(s) may need technical and “analyst”○ Commit to a certain # of hours per week or per month
● Start with Requirements, Test Results, and Defects data○ For most companies data size will be very manageable○ Don’t tackle big data problems (partition data if necessary)
● Start with Quality analytics, then add Coverage, then Velocity○ Focus on weekly project reports
● Add More Detail and More Data Sources Over Time
Thank You for Listening!Questions?