Continuous Automated Regression Testing to the Rescue

BT4 Concurrent Session 11/14/2013 10:15 AM

"Continuous Automated Regression Testing to the

Rescue"

Presented by:

Brenda Kise ProtoLabs, Inc.

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073 888‐268‐8770 ∙ 904‐278‐0524 ∙ [email protected] ∙ www.sqe.com

Brenda Kise Proto Labs, Inc.

Brenda Kise has fifteen years of experience in software, including development, test, test automation, management, and operations. Brenda has led teams that have automated testing from the ground up with a new code base and for Symantec’s NetBackup division, leading a team automating a legacy code base. In the first team, close to 100 percent automation was achieved using the JUnit framework in a tightly integrated Scrum/agile environment and went from a formal test cycle of six weeks to just eleven hours! At Symantec, with an 8 million line code base, the automation team focused on automating regression tests against the completed code base - going from 0% coverage to 14% coverage in one year. Brenda is currently managing a team of .NET developers working on internal applications and the Enterprise Data team doing business intelligence work for Proto Labs Inc.

9/9/13

1

B R E N D A K I S E D E V E L O P M E N T M A N A G E R , P R O T O L A B S

B R E N D A . K I S E @ P R O T O L A B S . C O M

Brenda Kise - Proto Labs Inc.

Continuous Automated Regression Testing to the Rescue!

Who am I?


� Currently a development manager for Proto Labs Inc in Maple Plain MN

� Manage a team of C# developers � Manage a team of Business Intelligence / Data

Warehouse developers � Proto Labs is a software company that does a little

manufacturing on the side � Hiring – growing by leaps and bounds

9/9/13

2

Background


�  16 years software experience �  Java / Database developer � Managed operations, development, hardware

buildouts, quality, test and automation teams � Test team used Junit to automate 99% of tests

¡  Went from a 3-6 week test cycle to 8 hours

� Automation team gained code coverage ground against 200 developers

Earlier is better


� The earlier after code check-in a defect is found, the easier it is to fix it

�  Few people affected the earlier the defect is found �  Less paperwork involved when found earlier � Often several layers of testing – if found early, less

re-testing is needed �  If a defect makes it to customers, the repercussions

can affect more than just the software

9/9/13

3

Proof Earlier is Better


Automated Testing


� Different levels for automation ¡  Component / Unit testing

÷ Generally done by developers

¡  Integration

¡  System level

¡  GUI

9/9/13

4

How is the test automated?


� Run with each build ¡  Applications such as Hudson will run the unit tests and

fail the build if the tests do not pass ¡  Immediate feedback, helps keep the build clean ¡  Can tell if someone is following process or not

÷ Broken build indicates coder did not run the build prior to checking into the main branch



� Run on a schedule ¡  Generally need a test frame work to run the tests ¡  Developers can run the tests prior to check-in ¡  Process is dis-associated with the build

÷ Build is completed ÷ Regression tests kicked off through a chron job ÷ Need a check to ensure build does not fail prior to test

kick-off ÷ Generally uses a separate system

9/9/13

5



�  Tester kicks off the test ¡  While this type of test is automated, it is not considered

automated regression ¡  No schedule or automated kick-off, requires human

intervention ¡  Still valuable as a time saving tool – allows the tester to

work on something else while test is running ¡  Especially valuable for long scenarios

How?


� Have a plan ¡  Are you working with a new code base? ¡  Are you focusing on unit / component testing? ¡  Is the code base stable?

� What level of testing? ¡  System ¡  Unit ¡  GUI

9/9/13

6

How?


� Choose your tools ¡  Custom framework? ¡  Junit or Cunit? ¡  OTS software?

÷ Watir ÷ Selenium ÷ QuickTest Pro ÷ WinRunner

How?


� Choose your approach ¡  Testers automate tests as they have time

÷ Like that ever happens ÷ Always seen as a secondary task

¡  Separate team of automaters ÷ Take tests that the testers use and automate ÷ Do not need to be experts in the areas of code ÷ Relies on well documented test scripts / scenarios ÷ Main focus of job

9/9/13

7

How?


� What type of code are you testing? ¡  New code?

÷ Start with Unit tests – fastest and cheapest way to get a lot of code coverage

¡  Legacy Code? ÷ Do you have test hooks or command lines? ÷ Very difficult to add unit tests at this point

¡  Web site or program? ÷ Different approaches for each

Case 1


�  FDA regulated website � Written in Java, so used Junit � Very specific requirements to test against � Testing the values displayed on the website � Did not include testing the webpage itself � Tested against 6 languages � Used drivers from Chrome, Firefox and IE

9/9/13

8

Case 1


� Previously used manual testing for majority of tests � Took 6 weeks to complete a full regression run

with no issues � Testers were asked to automate tests as they had

time � Using custom written testing software – no off the

shelf software

Case 1


� New code base started, done completely in Agile � No previous code or tests were re-used � Testers given training to use Junit – most had

never used it prior � Testers were not doing unit tests – considered

component or integration tests � Testers automated the tests as they created them

9/9/13

9

Case 1


� All integration, unit and component tests were written in the same sprint as the code

�  Scrum teams were comprised of Product Owners, Business Analysts, Developer and Testers

� All tests had to run against all languages on Production Equivalent boxes

�  4 levels of testing – subsystem, DAT, system and simulated use testing

Case 1


� Tests were created to specifically test each requirement (FDA requirement)

� Data was created and seeded into the database to test against ¡  Payload Data was actual production data that had been

de-identified ¡  Very important to use as close to production data as

possible

� Expected outcomes decided prior to tests being written

9/9/13

10

Case 1


�  Junit tests / setups were created in a test project � One piece of custom software used for testing was

the ability to create patients / clinics / physicians with needed relationships ¡  Test frame work used the actual code to create the same

as a human user would

�  Setup file was run, then the test. �  Single failure would log, but test run would

continue

Case 1


� Code was exercised in test using the interface ¡  Code was a client server base – published interfaces for

all components

� XML being delivered to the web browser was intercepted and searched for the values expected

�  Logged whether value found or not

9/9/13

11

Case 1


� Things to remember 1.  Does not test the page layout, only the content 2.  Works especially well if calculations are performed

prior to display 3.  Can be used with graphics, although this is consuming

to get done and every time the data changes the test has to be recaptured

Case 1


�  Since the test code was created in it’s own project, it was compiled and then run on each localized environment

� All information was logged, including pass, fail, start and end time

�  Logs were scraped and funneled to a status website used by management ¡  Thus relieving the testers of the inevitable question – are

you done yet? �  Logs were archived as testing evidence for the

release

9/9/13

12

Case 1


� Questions or comments about the first style of test automation?

Case 2


�  Large software package � Had a web interface, but not tested with

automation � Code was written in C++ / Java / XML � Vague requirements to test against � Testing to ensure functionality � Tested against all supported versions of Linux,

Windows, Solaris, HP

9/9/13

13

Case 2


�  Still using manual testing for majority of tests � Testers were asked to automate tests as they had

time � Manual tests were run through the different GUIs

available to the customer (Java, MVC and Web) � Test steps left to the individual tester, rarely

written in detail

Case 2


�  Single legacy code base, with modules added � Very little component style coding � Code had an extensive command line availability � Very little code coverage from unit tests

¡  Since it was mostly one big pile of code, hard to test small areas

� Testers and automaters were separate skills

9/9/13

14

Case 2


� Test automation happened on it’s own schedule, without regard to releases

� Automation team was divided into three areas ¡  Test engine and tools ¡  Automation Leads ¡  Automaters

� All tests had to run against all OS flavors supported �  3 levels of testing – manual testing, automated

regression and system level testing

Case 2


�  Large test harness created to run the tests ¡  3 full time people working on it ¡  1 million + lines of Perl code ¡  Included log scrapers, setup code and failure processing

� Automation script would call setup routines from the test harness to do standard things ¡  Create storage unit, create back files etc

� Expected outcomes decided prior to automation scripts being written

9/9/13

15

Case 2


� Automaters are not testers – they are developers � Automaters do not create the tests, test steps or

decide what to test � Are not experts on the code or the functionality

since they works across a large area of code � Allowed automation to continue no matter what

the shape of the release � Need a stable code base – attempt this type of

automation on a moving target is immensely painful.

Case 2


� Code was exercised using the published command lines

�  Looking for code coverage, not edge cases � Covered the “vanilla” cases – allowed manual

testers to concentrate on more complex cases not able to be tested in automation

9/9/13

16

Case 2


� Things to remember 1.  Need a stable code base 2.  Needs either command line or interface to call 3.  Often will need help from both development and test

1.  If the test case says something like “test X” with no steps, only tester knows what they mean.

2.  Amazing how often the command line information and error code details were incorrect

Case 2


� All automation was written in Perl � All information was logged, including pass, fail,

start and end time ¡  Timing was important in this case – when tests started

running long it was usually an early issue indicator

�  Logs were scraped and funneled to a storage area to be used if needed for debugging of issues found

� Pass / Failures were automatically uploaded to a webpage

9/9/13

17

Case 2


� Developers were expected to run specific tests against their code prior to check in.

� Unfortunately, any break in a regression test was automatically assumed to be a bad test ¡  Had an entire team in China that would analyze test

results and create issue reports � Regression score was used by management to show

health of the release ¡  If < 80% of test passed, score was 0 ¡  Only a singe instance of a perfect regression score in the 6

years it has been running

Case 2


� While you can get massive amounts of regression testing, very expensive approach

� Team of 10 full time employees and 15 part time contractors, plus a team of 8 doing analysis

� Massive hardware usage – several hundred VMs on a nightly basis

� Had so many tests running took a full 24 hours to complete a full run against a single build

9/9/13

18

Case 2


� Advantages of using Perl to test C code: ¡  You can start small – you don’t need a full test harness,

although you will like find yourself creating one to do basic setup tasks ÷ Watch out, this takes on a life of it’s own

¡  Can use a Chron job or Hudson to kick off the test scripts ¡  Logging is easy and straightforward in Perl

Case 2


� Questions or comments about the second style of test automation?

9/9/13

19

Take Aways


� Have a plan! � Know what type of testing you are trying to

accomplish � Know what type of results you are looking for

¡  Code coverage? ¡  Regression for found bugs?

� Know what type of team you are working with ¡  Automaters or testers?

Technology

Continuous Automated Regression Testing to the Rescue