These are the slides from the webinar HP ALM Less Test Cases, More Coverage which took place on the 28/05/2014. The slides discuss: • Agile Designer™'s ability to reduce the number of your test cases and improve your functional coverage • How test cases can be created in a fraction of the time • How you can quickly find which test cases need to change when a requirement changes and automatically repair them • Agile Designer™'s ability to quickly find and reserve test data as well as build sophisticated pools of data to drive test cases • How Agile Designer™ can eliminate error prone and manual test case design, reduce testing time and improve quality.
Text of Agile Designer™: HP ALM Less Test Cases, More Coverage Webinar
1. The science behind software testing Grid Tools Ltd 2014
2. HP ALM: Less Test Cases more Coverage Huw Price MD Grid Tools Ltd 3. OR Huw Price MD Grid Tools Ltd How Many Testers does it take to screw in a light bulb? 4. The Answer is 3! 5. What is Agile Designer? Requirements Clear Logical Specs Storyboards Expected Results Outsource Work Packages Test Cases Data-2 DRAW STORY BOARD TEST CASES TEST DATA CLEAR SPECIFICATIONS Data and Virtualization Agile Outsourcing Data Flow Logic 6. Agenda Different Coverage Techniques Building the perfect set of test cases from Requirements Updating Test Cases when Requirements Change Optimizing Existing Test Cases 7. Common Challenges 56% of software defects can be traced back to ambiguities in requirements1 Detecting and resolving production defects is over 40x more expensive than in requirements2 Industry standard for manual test case design coverage is between 10-20% Over-testing of some functions by factors of 40 is not uncommon Change requests to existing software requirements can take up to as much as 60% of the SDLC (Stats: 1 and 2 - Bender RBT, Others: Grid-Tools) 8. What is Coverage? It is not: Code Covered Number of Test Covered Tests Run Percentage of use cases All Paired Combinations It is: Designing Sufficient Tests To VERIFY That The Design And Code Correctly Implement The Requirements Did you get the right answer for the right reason - Two or more defects may sometimes cancel each other out - Observability 9. Copyright 2008 Bender RBT Inc. 9 What is Coverage? If the customer is a business client or a preferred personal client and they have a checking account, $100,000 or more in deposits, no overdraft protection and fewer than 5 overdrafts in the last 12 months, set up free overdraft protection otherwise do not give overdraft protection. This function has sixty-four possible combinations of input from which to select test cases: So which ones to choose? 10. Different Coverage Techniques Combinatorial All Pairs Constrained All Pairs Does not support Expected Results - You have to work it out for each combination which is very time consuming Combinatorial is something that accidentally increases functional coverage (and only to a point) Combinatorial does not give you any solid metrics on how good your testing actually is you can only infer that x out of y combinations have been covered, but it has no bearing in actual functional logic. You end up with lots of false errors that have to be checked by hand 11. Copyright Bender RBT Inc. 2009 11RBT24261 Cause-Effect Graphing Observable Events and Path Sensitizing Assume C and F are not observable events. Assume A is stuck at FALSE. Enter as a test case A(T), B(T), D(T), E(T). Results should be C(T), F(T) and G(T). T T T T T T T Different Coverage Techniques 12. Different Coverage Techniques Optimized Flow Chart Modelling Advanced Graph Optimization Most projects already have a flow chart Similar results to Cause and effect Supports Looping Supports Constraints Not applicable for non sequential based logic 13. Agile Designer Test Case Techniques Coverage Feature Agile Designer Combinatorial Yes Flow Chart Yes Cause and Effect Yes Constraints Yes Observability Yes Looping Yes Complexity Metrics Yes 14. Typical Examples of Test Cases built Manually from Requirements Baseline Agile Designer Test Cases Created 14 17 Time Taken 5 hours 2 hours Functional Coverage - % 16% 100% Baseline Agile Designer Test Cases Created 150 19 Over Testing x18 x2 Time Taken 5 hours 40 mins Functional Coverage - % 80% ~100% Baseline Agile Designer Test Cases Created 3 12 Time Taken 4 hours 30 mins Functional Coverage - % 5% 100% 15. Leg Before Wicket Rule Demo 1 Visio Version Node Types Existing Coverage HP ALM Data Packs 16. Screw in a Light Bulb Demo 2 Different Optimization Looping Path Hints Testing Importance, risk based 17. Business Analyst Programmer Tester User Initial Project Test Cases Program Requirements Business Analyst Programmer Tester User Change Request Test Cases Program Change request Business Analyst Programmer Tester User New Process Requirements Tester Test Cases Program RED=HARD WORK Evolving to Greater Efficiency with Agile Designer 18. A day at the Races Demo 3 Changes to Requirements Fix Broken Test Cases Return to a 100% 19. The HP Test Case Factory 20. Baseline Optimized Reduction in Testing Existing Test Cases 8 9 Test Coverage - % 20% 100% Functional Nodes 78 31 60% Edges 76 44 Baseline Optimized Reduction in Testing Existing Test Cases 15 7 Test Coverage - % 30% 100% Functional Nodes 113 39 65% Edges 117 50 N.B. Node = design step in test case / Edge = Journey from one design step to the next Typical Examples of Optimized Test Cases Built with No Requirements 21. Leg Before Wicket Rule Demo 4 Reverse Engineer 22. Test cases designed manually by hand usually have quite a static flow. As you can see by these examples there are a few flows which cover the same functionality. A->B->C->D A->B->C->E R->A->B->C->D Agile Designer Test Cases 23. Agile Designer Test Cases Each has its individual starting point and set of steps. G->F->A->B->C Q->P->R->G But often cross paths in terms of Testing the same features several times. We want to avoid this. 24. When we have imported these into Agile Designer we can see we have 5 paths but as we mentioned before some cover similar paths. A occurs 4 times B occurs 4 times C occurs 4 times R occurs 2 times G occurs 2 times This gives us the perfect opportunity to optimize and improve the test cases Agile Designer Test Cases 25. From this example we have 5 test cases but have we covered each functional step in the most optimal of fashions? 5 Test Cases 24 Nodes (Design Steps) 22 In/Out Edges (Journeys) Agile Designer Test Cases 26. Agile Designer Test Cases So now we optimized the flow and get something like this. Optimize: - Removed duplicate design steps and revaluated how the test cases should link together. During this evaluation process we are able to navigate through the diagram from Start->End without duplicates causing us problems. 27. Agile Designer Test Cases Post Optimize you can see that the number of Nodes/Edges we are testing has reduced significantly. - 12 Nodes (Design Steps) - 20 Edges (Decisions) This shows us a 50% reduction in steps we need to take whilst giving us 100% in functional coverage. Weve also reduced the number of decisions by 2. This highlights that although a smaller reduction our decision process is much more intelligent and with just 2 fewer we are covering more functions. 28. Agile Designer makes Continuous Integration = Idea to Production Optimized Tests Existing Test Cases Test Data Automation Continuous Development 29. Agile Designer - Summary Think of a Requirement like a circuit board The Test Cases will reflect the requirements Static Testing You can Accurately Measure your Coverage You can Automatically create the Minimum Test Cases With Maximum Coverage You can reduce and optimize your existing test cases You can predict how long software will take to develop 30. Agile Designer Next Steps There is a free limited download Contact sales for a full copy trial Linked to HP ALM and Datamaker We offer half day training courses (Free T shirt!) http://info.grid-tools.com/agile-designer-training-workshops We offer an Agile Designer Certification We are looking for partners to roll out the test optimization factory We are looking for local partners 31. Huw.firstname.lastname@example.org email@example.com Huw Price MD Grid Tools Ltd