Get Full Value from Your Automated Testsxunitpatterns.com/~gerard/StarCanada2015Tutorial.pdf · 2015-06-26 · ROI of (Automated) Tests ROI = Value / Cost ROI: Return on Investment

1

1 Copyright 2015 Gerard Meszaroshttp://StarTutorial2015.testAutomationPatterns.com

Get Full Value from YourAutomated Tests

Gerard MeszarosIndependent Consultant

CTO of [email protected]

These Slides:http://StarTutorial2015.testAutomationPatterns.com


This is final version of my tutorialslides and handouts.

See you in Vancouver June 23rd!

Gerard

2


Product & I.T.

I.T.

EmbeddedTelecom

My Background

Gerard [email protected]

•Software developer

•Development manager

•Project Manager

•Software architect

•OOA/OOD Mentor

•Requirements (Use Case) Mentor

•XP/TDD Mentor

•Agile PM Mentor

•Test Automation Consultant & Trainer

•Lean/Agile Coach/Consultant


Product & I.T.

I.T.

EmbeddedTelecom

My Background

Gerard [email protected]

•Software developer

•Development manager

•Project Manager

•Software architect

•OOA/OOD Mentor

•Requirements (Use Case) Mentor

•XP/TDD Mentor

•Agile PM Mentor

•Test Automation Consultant & Trainer

•Lean/Agile Coach/Consultant

3


Agenda• Motivation?

– Why we need to rethink how we test and automate• Overall Strategy

– Tests as Specification, made executable• Tactics and Practices

– Scope vs. Detail, Refactoring, Collaboration• Specification Hands-On

– A chance to try this out• Automation and Tools

– How do we automate these specifications?


Functional Test Automation?

4


Unit Test Automation


Effectiveness of Automation

5


Happiness With Test Automation?


So Why Are You Unhappy?

6


The Agile Test Problem

RequirementsDevelopment

Testing



• As development increments reduce induration, testing needs to be reducedaccordingly

TestingDevelopment

Requirements

TestingDevelopment

Requirements

7



... and traditional approaches to testing nolonger work

TestingDevelopment

Requirements

TestingDevelopment

Requirements

TestingDevelopment

Requirements

TestingDevelopment

Requirements


ROI of (Automated) Tests

ROI = Value / Cost

ROI: Return on Investment

8


Exercise: Value of Tests1. What value can automated tests provide?

2. How can this value be increased?


Exercise: Cost of Tests1. What are the costs associated with

automated tests?

2. How Can Cost of Tests be Reduced?

9


Value of Tests• Validate correct functionality was built

– Built the right product

• Verify implementation is correct– Built the right product right


Value of Tests• Validate correct functionality was built

– Built the right product– After the fact is much too late to discover this!– Should be discovered before development starts.

• Verify implementation is correct– Built the right product right– Tests as Defect Detection– But, Cannot test quality into the product

10


Business

Test “After”

• Different bosses & definitions of success

Development TestTeam


Test&Fix Cycle Battleship™

DevelopmentIndependent

Test Team

Requirements

B6! Missed!

Requirements

11


Result of Test “After”• Hard to predict when product will be “done”

– Since unknown number of test&fix cycles required• Lots of effort wasted building the wrong

product and then fixing it– Why not build the right product the first time?

• Product is not designed for testability– Because there is no “testability spec”– Therefore, automated testing is difficult


Conway’s Law• “The Architecture of System will resemble the

Organization that Built it.”

DevelopmentTeam

TestTeam

BusinessTeam

12


Resulting Architecture

Specifi-cations

ProductCode

TestCode


Resulting Tests Are:• Forced to find sub-optimal ways to interact

with the Product Code– Via interfaces not designed for tests– Can only test the whole system, not the parts that

contain the logic we want to test• Forced to find ways to control the state of the

system– Can’t set state directly; must use convoluted paths to

set the state

13


Where Does This Leave Us?• Tests are more complex than they should be• Test coverage is poor so important bugs are

missed• Tests are fragile with frequent “false positive”

test failures• Tests require frequent, expensive

maintenance as code changes• Tests take longer to run


Agenda– Overall Strategy• Motivation?

• Overall Strategy– Tests as Specification, made executable

• Tactics and Practices– Scope vs. Detail, Refactoring, Collaboration

• Specification Hands-On– A chance to try this out

• Automation and Tools– How do we automate these specifications?

14


Example-Driven Development• Business & Testers come up with examples;

Developers help them formalize it.• Developers write interpreters for examples;

design product code to support them• Developers write product code guided by

execution of the examples

This is Like TDD but at a Very Large ScaleSometimes called Behaviour-Driven Dev’t (BDD)


ExampleInterpreter

Code

D,TB,T,D

ProductCode

D

Example-Driven Development

This is Like TDD but at a Very Large ScaleSometimes called Behaviour-Driven Dev’t (BDD)

PotentiallyExecutableExamples

15


Example-Driven Development• Runtime View:

ProductCode

ExampleInterpreter

Code

B,T,D D,TDExample

ExecutionResults

PotentiallyExecutableExamples


Example-Driven Development• Feedback Loops:

PotentiallyExecutable

Specifications ProductCode

ExampleInterpreter

CodeHow they say

ExampleExecutionResults

What they say

16


Example-Driven Development• Feedback Loops:

PotentiallyExecutable

Specifications ProductCode

ExampleInterpreter

Code

ExampleExecutionResults

What they say

How they say

Isn’t This “Just”Test AutomationBy Another Name?


Life Cycle of an Example

UserGoal

Feature

StoryTitle

StoryNarrative

StoryScenarios

StoryExamples

ExecutableExamples

SatisfiedExamples

17


Activities on an Example

UserGoal

Feature

StoryTitle

StoryNarrative

StoryScenarios

StoryExamples

ExecutableExamples

SatisfiedExamples

FormalizationAutomation

ProductDevelopment

Make ConcreteAdd Data

DefineAcceptance

Criteria

WorkBreakdown

ProductDesign &Validation

Elaborate


The Goal:• All functionality is clearly specified and

covered by automated checking– Of product against the examples/specification

• All traditional functional testing is now“regression testing”– Since developers had it working before they declared it

“done”• Human testers can focus on exploratory

testing– Instead of running the same functional tests every

iteration

18


Exercise: What’s in a Name• How does calling it an “example” or “spec”

change your expectations and approach?


Agenda– Tactics & Practices• Motivation?

• Overall Strategy• Tactics and Practices

– Getting the Terminology Right;» Managing Scope vs. Detail» Use the terminology of the domain

– Refactoring Examples (Simplifying, Clarifying)– Collaborative Process

• Specification Hands-On– A chance to try this out


19


How to Make Examples Understandable• Traditional automated tests are code!

– Provide all details that product code requires– Way more detail than reader can easily understand

• We need to focus on reader’s needs– Not the computer’s or products needs

• But “The Devil is in the Details”– Yes, we need to specify the details somewhere– But not everywhere!– Can be hidden in test harness for most examples– And specified in just a few very detailed examples


Test Automation Pyramid• Large numbers of very

small unit tests– Ensures integrity of code

• Smaller number offunctional tests for majorcomponents– Verify integration of units

• Even fewer tests for theentire application &workflow– Ensure application(s)

support users’requirements

• Tools to supporteffective exploratorytesting

Unit Tests

ComponentTests

SystemTests

Pyramid originally proposed by Mike Cohn

Exploratory Tests

20


Behavior Specification at Right LevelNeed to specify at the right detail and scope.

Broad Narrow

Detail

High

L

ow

Scope

Unit Tests

ComponentTests

SystemTests


Behavior Specification at Right LevelNeed to specify at the right detail and scope.

Broad Narrow

Detail

High

L

ow

Scope

ComponentTests

SystemTests

21


ComponentTests

SystemTests

Behavior Specification at Right Level

Rules &Algorithms

Multi-Use Case

WorkflowsTransactions(Use Cases)

IncompleteSpec

Toomuch detailUnmaintainable

Detail

High

Low

Broad NarrowScope

• Specify broad scope at minimum detail• Specify most detailed req’ts at narrowest scope


Use Cases & User Stories

ConfigureNotification

Rules

SuspendNotification

ResumeNotification

AccountHolder

ProcessTransaction

TransactionSettlement

Not

ify

base

d on

cha

rge

Type

Not

ify

base

d on

cha

rge

Cont

inen

tN

otif

y ba

sed

on c

harg

e Co

untr

yN

otif

y ba

sed

on c

harg

e Ci

ty &

Sta

te

Susp

end

Not

ific

atio

n

22


Testing Notifications• Open MegaBank app• Log in as “BobMa” with password ******• Click on “Manage Notifications” tab• Click on “Add New Rule” button• Select account “10035692877”• Type “Default Rule” into field “rule name”• Type “1000” into field “threshold amount”• Click on “all transaction types” radio button• Click on “all locations” radio button• Click on “save changes” button

And this is just the first step in the overall workflow!


Testing Notifications – 1/2Given:

User and Accounts

Anotherattempt:

When:Notification

Rule is Configured

Then:Notification RuleConfigured and

Active

Then:Notification Rule

is Active

23


Testing Notifications – 2/2

Then: ExpectedNotifications

When: TheTransactions tobe processed

Use Case:Process

Transactions

Anotherattempt:


Issues With This Test• Difficult to understand which TX’s should notify

– because cause (rules) and effect (notification) are far apart• Only verifies one simple combination of rules

– We will require many more tests to test all the othercombinations

– Lots of repetition of workflow & data across test cases• Simplest workflow;

– More complex workflows will be even longer and harder tounderstand

• Tests will take a long time to run– Due to need to configure first, then run transaction

processing

24


How Can We Improve Tests?• Make tests shorter

– Fewer steps• Make tests less detailed

– Omit unnecessary information• Focus tests on specific aspects of behaviour

– Test/Spec Algorithms & Rules directly– Test less functionality, but more thoroughly

Fine advice, but too vague

Difficult without Test-Friendly Architecture


How Can Example Mindset Help• Describe overall workflow

– with much less detail• Provide more details

– For individual transactions• Provide even more details

– For business rules.• But not embedded in the workflow

specifications

25


Changing Level of Abstraction/Detail• Need to Reduce Detail or Reduce Scope

Rules &Algorithms

Multi-Use Case


Toomuch detailUnmaintainable

IncompleteSpec

Broad Narrow

Detail

High

L

ow

Scope


Omitting Unnecessary Information• Ask not what you can include in an example…

• Ask instead:

• What can I leave out of this example?

“If it isn’t essential to conveying the essence ofthe behavior, it is essential to not include it.”

26


Refactoring Workflow ExampleGiven:

User and Accounts

Example:

When:Notification

Rule is Configured

Then:Notification RuleConfigured and

Active

Then:Notification Rule

is Active


Example:Refactoring Workflow Example

27


Then: ExpectedNotifications

Example:

When: TheTransactions tobe processed

Use Case:Process

TransactionsRefactoring Workflow Example



28




Example: Given:User &

Thresholds

When:Transactions

Are Processed

Then:We Expect

Notifications

Refactoring Workflow Example

29


Example: Given:User &

Thresholds

When:Transactions

Are Processed

Then:We Expect

Notifications

credit

credit

9991113333



9991113333

Example:

Broad Scope (Multi-Actor);Minimum Detail (per Actor/Transaction);

Given:User &

Thresholds

When:Transactions

Are Processed

Then:We Expect

Notifications

credit

credit


30


Specifying Transaction Details• So, where should omitted details go?

Rules &Algorithms

Multi-Use Case


IncompleteSpec

Toomuch detailUnmaintainableBroad Narrow

Detail

High

L

ow

Scope


Single Use Case ExampleUse Case:

ManageNotifications

Medium Scope (Single Actor)Medium Detail (Transaction, not UI);

Example:

Given: User & Accounts

When: Customer sets up threshold

Then: We ExpectLogging,

Updated Statuses

Then:We Expect Logging& Updated Statuses

31


Specifying Business Rules• Need the more detail; need to reduce scope

Rules &Algorithms

Multi-Use Case


IncompleteSpec

Toomuch detailUnmaintainableBroad Narrow

Detail

High

L

ow

Scope


Kinds of Business Rules & Algorithms• Generation of Choices

– Pick List Contents» Payment methods allowed,

• Validation of inputs– user names, passwords, customer names, addresses

• Calculation of derived values– Cost of order, taxes

• Basis of decisions– Should you do something

» Should you notify?– How should you do something?

» How should you notify? E-mail? Text? Phone Call?

32


Business Rule Example

Process TransactionThreshold per Charge Type

Example:

Configuration




Narrow Scope (Single Rule)High Detail (Everything that matters)

Example:

Configuration Given theserules

Then: Theanswer should be

When we askNotificationRequired?with this transaction:

33


Agenda– Specification Hands-On• Motivation?

• Overall Strategy• Tactics and Practices• Specification Hands-On

– Workflow Specification– Transaction Specification– Business Rules Specification



Exercise – Write Some Examples• Write a Workflow Example

– Identify a few scenarios– Identify what would cause different scenarios to be executed– Write a specification for one of them– Identify any business rules or calculations that would be involved; add

them to your list of algorithms and rules

• Write Examples for a Business Rule or Two– Describe the behaviour of an algorithm or business rule identified in

your workflow examples using tables.– Focus on what the inputs/outputs are, not where they came from

34


Exercise Logistics• Pair up with someone

– Avoids getting stuck; learn from each other• Pick an Application At Least One of You Knows

– Could be your own application– Or a public domain one like Amazon

• Pick a couple of related workflows– Choose variations of same workflow so there is some

common steps– But there should be some different steps too.


Exercise Logistics (2)• You Can Choose to Work:

– Outside In: Workflow example first, then Business Ruleexample

– All at Once: Put details into Business Rule examplewhile sketching out Workflow in higher level example

• Don’t Worry About How to Automate– We’ll discuss this later

35


List of Workflow Scenarios


List of Business Rules & Algorithms

36


Workflow Example Template


Workflow Example Template

37


Business Rule Examples Template


Business Rule Examples Template

38


Example: List of Scenarios• Basic Notification Workflow

– Notify user of a transaction above threshold– Don’t notify user of a transaction below threshold– Changing threshold for one user doesn’t affect threshold

for other users• Suspend & Resume Notifications

– No notification is sent while notifications are suspended– Notifications are sent for new transactions when

notifications are resumed– Suspending one user doesn’t affect other users

notification state


Example: List of Business Rules & Algorithms

• Content Generation– What is the list of accounts associated with this user’s

login credentials?– What types of transactions should the user be able to

pick?• Notification Decision

– Should we notify the user about this transaction?– By what means should we notify the user?

39


Example: Notification Workflow (initially)Given BobMa has set notification threshold to $10,000

for all transactions on his accountWhen the bank processes debit for 15,000 to his accountAnd the bank processes debit for 9,999 to his accountAnd the bank process debit for 10,000 to his accountThen BobMa receives notification for debit 15,000And BobMa receives notification for debit 10,000When the bank processes debit for 10,000 to Fred’s

accountThen FredS receives no notification


Example: Notification Workflow (simplified)Given BobMa has set notification threshold to $10,000 for all

transactions on his accountWhen the bank processes debit for 10,000 to his accountThen BobMa receives notification for debit 10,000

Given BobMa has set notification threshold to $10,000 for alltransactions on his account

When the bank processes debit for 9,999.99 to his accountThen BobMa receives no notification

Given BobMa has set notification threshold to $10,000 for alltransactions on his account

And FredS has not set notification threshold on his accountWhen the bank processes debit for 10,000 to his accountThen FredS receives no notification

Positive Case

negative Case:< Threshold

negative Case:Another user

40


Example: Suspension WorkflowGiven BobMa has set notification threshold to $10,000

for all transactions on his accountWhen the bank processes debit for 15,000 to his accountThen BobMa receives notification for debit 15,000

When BobMa suspends notification on his accountAnd the bank processes debit for 15,000 to his accountThen BobMa receives no notification

When BobMa resumes notification on his accountAnd the bank processes debit for 15,000 to his accountThen BobMa receives notification for debit 15,000


Example: Suspension WorkflowGiven BobMa has set notification threshold to $10,000

for all transactions on his accountWhen the bank processes debit for 15,000 to his accountThen BobMa receives notification for debit 15,000

When BobMa suspends notification on his accountAnd the bank processes debit for 15,000 to his accountThen BobMa receives no notification

When BobMa resumes notification on his accountAnd the bank processes debit for 15,000 to his accountThen BobMa receives notification for debit 15,000

Over threshold

Over threshold

Over threshold

41


Example: Suspension Workflow (simplified)Given BobMa has set up notification for all transactions

on his accountWhen the bank processes an over-threshold-txThen BobMa receives notification for that tx

When BobMa suspends notification on his accountAnd the bank processes an over-threshold-txThen BobMa receives no notification for that tx

When BobMa resumes notification on his accountAnd the bank processes an over-threshold-txThen BobMa receives notification for that tx


Example: Business Rule Spec


Configuration

42


Example: Single Use-Case Test Use Case:Manage

Notifications

Medium Scope (Single Actor)Medium Detail (Transaction, not UI);

Given: User & Accounts

When: Customer sets up threshold

Then: We ExpectLogging,

Updated Statuses

Then:We Expect Logging& Updated Statuses


Exercise – Simplify Your Examples• Simplify Your Workflow Examples

– Review your list of workflow scenarios– Write a specification for another workflow example– Refactor both examples, simplify & clarify– Ruthlessly remove anything that isn’t needed

• Move Complexity into Transaction Examples– If Workflow examples are too complex (too much detail, more than

one steps per user/actor), try moving details/steps into examples forsingle transactions

• Simplify Your Business Rule Examples– Review your rule/algorithm examples– Add additional examples as needed– Ruthlessly remove anything that isn’t needed

43


Agenda– Automation and Tools• Motivation?

• Overall Strategy• Tactics and Practices• Specification Hands-On• Automation and Tools

– How do we automate these specifications?– What tools can we use to automate them?


Automating Workflow Example – via User InterfaceGiven BobMa has set notification threshold to $10,000

for all transactions on his accountLogin as BobMaGo to notification configuration pageSelect chequing accountSelect “edit notification rules”Choose action “add new rule”Set tx type to “All” and threshold to “10,000”…

When the bank processes debit for 10,000 to BobMa’saccount

Add transaction record for “debit” “10,000” “bobma’s account” totransaction queue

Run the batch job “processTransactionQ”

Then BobMa receives notification for debit 10,000Run DB query “select * in notifications where date=today and user=“BobMa”Ensure count is 1 and record.amount=10,000

44


Test-After Architecture• Must test through User Interface

• Must load lots of data

System Under Test

Should weNotify?Process

Transaction

ConfigureNotificationThreshold

NotificationRules

TransactionInterface

ConfigurationUser

Interface

NotificationLog

WorkflowTest

DoNotification.


Example-Driven Architecture• Need to provide API’s to invoke functionality

• Bypassing the UI• Supports workflow and

use case / transaction example automation

System Under Test

Should weNotify?

DoNotification.

ProcessTransaction


NotificationRules

NotificationLog


ConfigurationUser

InterfaceWorkflowExample

WorkflowKeyword

Interpreter

45


Automating Workflow Specification – via API

Given BobMa has set notification threshold to $10,000for all transactions on his account

sut.setNotificationThresholdForUserAccount( “bobma”, “chequing”, 10,000);

When the bank processes debit for 10,000 to his accountsut.processTransactions( [ {“bobma”,”chequing”,”debit”, 10,000} ] );

Then BobMa receives notification for debit 10,000allTx = sut.getTransactionNotificationsForAccount( “BobMa”, “chequing”);assertAllTx(alltx, {“BobMa”, “chequing”, 10,000});

Before Eachsut = new MegaBankNotificationsAPI();bobma = sut.createUser(“bobma”, ….);sut.addUserAccount(“bobma”, “checking”, …..);

sut =


Automating Workflow Specification – via API(made parameterized)

Given {{user}} has set notification threshold to {{amount}} forall transactions on his account

sut.setNotificationThresholdForUserAccount( {{user}},“chequing”, {{amount}});

When the bank processes debit for {{amount}} to his accountsut.processTransactions( [ { {{user}}, ”chequing” , ”debit”, {{amount}} } ] );

Then {{user}} receives notification for debit {{amount}}allTx = sut.getTransactionNotificationsForAccount( {{user}} , “chequing”);assertAllTx(alltx, { {{user}} , “chequing”, {{amount}} });

Before Eachsut = new MegaBankNotificationsAPI();bobma = sut.createUser(“bobma”, ….);sut.addUserAccount(“bobma”, “checking”, …..);

sut =

46


EDA – Single Tx Examples• Same API supports Transaction (Use Case)

examples: System Under Test

Should weNotify?

DoNotification.

ProcessTransaction


NotificationRules

NotificationLog


ConfigurationUser

Interface

Config’nTX

Example

Config’nKeyword

Interpreter




Example:

Configuration

47


ConfigureNotificationThreshold System Under Test

EDA – Business Rules• Encapsulation of Rules in “Rules Component”

• which can be accessed directly via the Test

DoNotification.

ProcessTransaction

NotificationRules

NotificationLog


ConfigurationUser

Interface

NotificationMethod Example

NotificationRules Fake

NotificationRule Example

Should weNotify?


EDA – Business Rules• Encapsulation of Rules in “Rules Component”

• which can be accessed directly via the Test

• Component Accepts Rules via “Data Injection”– Simplify Automation by avoiding need to configure

• Ensures Design-for-testability– But Only When Automation is a Dev’t Responsibility

Should weNotify?

NotificationRules Fake

NotificationRule Example

48


public class NotificationRequired ColumnFixture {

public String account;

public String ChargeType;

public Currency amount;

public notify( ) {NotificationDecider nd =

new NotificationDecider ();return nd.shouldNotify(account, chargeType, amount);

}}

Sample Fit Column Fixture


Exercise: Automate Your Examples• For each of your workflow steps,

– implement a keyword in psuedo-code calling the APIyou would like the system to expose

– Pass all the information your test provided, plus– Pass default values for any info the system would

require• For each of your business rule tables,

– implement a keyword or fixture in psuedo-code callingthe business rule component API you would like thesystem to expose

– Pass input values provided by example– Ensure output values specified by example are returned

49


Workflow Keyword ImplementationKeyword:

Keyword:

Keyword:

Keyword:


Workflow Keyword ImplementationKeyword:

Keyword:

Keyword:

Keyword:

50


Business Rule Table ImplementationInput Columns:

• What would you do with them?

Output Columns:

• Where would you get them from?


Business Rule Table ImplementationInput Columns:

• What would you do with them?

Output Columns:

• Where would you get them from?

51


Agenda– Automation and Tools• Motivation?

• Overall Strategy• Tactics and Practices• Specification Hands-On• Automation and Tools

– How do we automate these specifications?– What tools can we use to automate them?


Tools Selection• Tools must support the strategy

– Support business/testers to write specs/tests inbusiness language» More or less “natural” language

– Let them write specs before the specs are automated orthe system is built» This disqualifies Capture & Replay tools

– Can be automated by technical people, later» Separation of business skills from technical skills» e.g. Table or Keyword-based tools

– Can be run by anyone, especially developers» This disqualifies expensive tester-specific tools

52


Business Rules Tools - A Sampling• Fit

– Column Fixture• SLIM (Fitnesse)

– Decision Table• Robot Framework

– Parameterized Testcase


Workflow Spec Tools (my fav’s)• Gherkin (Given-When-Then terminology)

– Cucumber (various tech-specific implementations)– JBehave (Java/JVM specific)– SpecFlow (.Net)

• Other GWT (Given-When-Then terminology)– Robot Framework GWT

» https://blog.codecentric.de/en/2009/11/givenwhenthen-and-example-tables-using-the-robot-framework/

• Robot Framework– Keyword-based Testcases

Resource: Agile Alliance Functional Testing Tools program

53


More Complete Survey of Tools

• Agile Alliance Functional Testing Toolsprogram Yahoo! group– https://groups.yahoo.com/neo/groups/aa-ftt/info

• Agile Alliance Functional Testing Toolsspreadsheet– https://docs.google.com/spreadsheet/ccc?key=0Apag_J

97l3CTdHR0WS1sZGFVaFA0dEpXYWRqLXBxV3c#gid=1


Exercise: Automation Tool AssessmentAssess each of the “test automation” tools your

organization currently uses (or is consideringfor future use) for suitability to this approach.

Use the worksheet on the next page.

54


Automation Tool AssessmentTool Name Supports

naturallanguage?

Can be specifiedbefore system isbuilt?

Automation canbe done byDevelopers?

Can be run byDevelopersToo?


Summary of Approach• Business & Testers come up with examples;

Developers help them formalize it.– Ensures everyone agrees on what needs to be built

• Manage Scope vs. Detail in examples– Different kinds of examples for each of:

» Workflow sequences» Transaction interactions» Business rules and algorithms

– Avoids long, hard-to-understand examples

55


Summary of Approach• Developers write interpreters for examples;

design product code to support them– Ensures the architecture supports automation

• Developers write product code guided byexecution of the examples– They know what needs to be built– They can run the examples frequently to know how they

are doing


Summary of Benefits• Examples encourage collaboration between all

project roles.• Built-in automation helps avoid Fragile Tests• Automated Examples help us prevent defects

being introduced through misunderstandingof requirements

• Automated Examples ensure complete “test”coverage of functionality

• Frees up testers to do more ExploratoryTesting

•

56


Closing Thoughts• Are you automating to find defects or prevent

them?• Are your automated tests good examples?

– Why not? What would you need to change?• Are your tests low maintenance?

– Why not? What causes them to break?– What could you change…– … to make them break less often?– .... to reduce the impact of breakage?


Thank You!Gerard Meszaros

[email protected]://www.xunitpatterns.com

Slides:http://StarTutorial2015.testAutomationPatterns.com

Call me when you:• Want to transition to Agile or Lean• Want to do Agile or Lean better• Want to teach developers how to test• Need help with test automation strategy• Want to improve your test automation

Jolt Productivity Awardwinner - Technical Books

Available on MSDN:

57


References & Resources - Tools• AAFTT

– https://groups.yahoo.com/neo/groups/aa-ftt/info

• Cucumber– https://cucumber.io/– http://www.specflow.org/

• Fitnesse– http://www.fitnesse.org/

• Fit– http://fit.c2.com/– http://en.wikipedia.org/wiki/Framework_for_integrated_test– https://code.google.com/p/refit/

• Robot Framework– http://robotframework.org/

• Robot Framework GWT– https://blog.codecentric.de/en/2009/11/givenwhenthen-and-example-

tables-using-the-robot-framework/


References & Resources - Reading• Acceptance Test Engineering

– Grigori Melnik, Gerard Meszaros• Agile Testing + More Agile Testing

– Lisa Crisping, Janet Gregory• The Cucumber Book: Behaviour-Driven Development for

Testers and Developers– Matt Wynne and Aslak Hellesøy

• Fit for Developing Software: Framework for IntegratedTests– Ward Cunningham, Rick Mugridge

• Lean-Agile Acceptance Test Driven Development– By Ken Pugh

• Test Driven .NET Development with FitNesse• Gojko Adzic

58


Robot Framework GWT example*** Settings ***Resource ${RESOURCES}/BDD.txt

*** Keyword ***Example

[Arguments] ${periodClosed} ${periodOpenAndModified} ${importDay} ${oldManagerValidUntil}${newManagerValidFrom}

Given initialized criteria for bonus commercialAnd a branch B with branch manager M_OLD and employee E1And evaluation for E1 for period ${periodClosed} which is closedAnd evaluation for E1 for period ${periodOpenAndModified} which is open and modified

When M_NEW becomes new manager of branch BAnd import service is called on ${importDay}

Then the new branch manager of branch B is M_NEW valid from ${newManagerValidFrom}And branch manager M_OLD manages employee E until ${oldManagerValidUntil}And branch manager M_NEW manages employee E from ${newManagerValidFrom}And Evaluations for E1 still have the same content

| *Test Case* | | *Closed Period* | *Open Period* | *Run Import On* | *Old Manager Stops* | *New Manager Starts* || 1 | Example | 1.11.2009 - 30.11.2009 | 1.12.2009 - 31.12.2009 | 11.11.2009 | 30.11.2009 | 1.12.2009| 2 | Example | 1.11.2009 - 30.11.2009 | 1.12.2009 - 31.12.2009 | 1.11.2009 | 31.10.2009 | 1.11.2009| 3 | Example | 1.11.2009 - 30.11.2009 | 1.12.2009 - 31.12.2009 | 1.12.2009 | 30.11.2009 | 1.12.2009


Summary of Exercises• Value of Automated Tests• Cost of Automated Tests• What’s in a Name?• Write a Workflow Example• Write a Business Rule Example• Refactor a Workflow Example• Automate Execution of Your Examples• Automation Tool Assessment

Documents

Get Full Value from Your Automated Testsxunitpatterns.com/~gerard/StarCanada2015Tutorial.pdf · 2015-06-26 · ROI of (Automated) Tests ROI = Value / Cost ROI: Return on Investment