Meszaros Itake Keynote v5 - XUnitPatterns.comxunitpatterns.com/~gerard/itake2017keynote.pdf · &rs\uljkw *hudug 0hv]durv kwws lwdnh whvw$xwrpdwlrq3dwwhuqv frp,w $lq¶w7hvwdeoh 8qwlo

$Page 1: Meszaros Itake Keynote v5 - XUnitPatterns.comxunitpatterns.com/~gerard/itake2017keynote.pdf · &rs\uljkw *hudug 0hv]durv kwws lwdnh whvw$xwrpdwlrq3dwwhuqv frp,w $lq¶w7hvwdeoh 8qwlo$
1

1 Copyright 2017 Gerard Meszaroshttp://itake2017.testAutomationPatterns.com

It Ain’t Testable Until it’s Tested

Gerard MeszarosIndependent Consultant

CTO of FeedXL.Com

[email protected]

These slides: http://itake2017.testAutomationPatterns.com


Product & I.T.

I.T.

EmbeddedTelecom

My Background

Gerard [email protected]

•Software developer

•Development manager

•Project Manager

•Software architect

•OOA/OOD Mentor

•Requirements (Use Case) Mentor

•XP/TDD Mentor

•Agile PM Mentor

•Test Automation Consultant & Trainer

•Lean/Agile Coach/Consultant

2


Product & I.T.

I.T.

EmbeddedTelecom

My Background

•Software developer

•Development manager

•Project Manager

•Software architect

•OOA/OOD Mentor

•Requirements (Use Case) Mentor

•XP/TDD Mentor

•Agile PM Mentor

•Test Automation Consultant & Trainer

•Lean/Agile Coach/Consultant Gerard [email protected]


Key Messages

• Systems are hard to test in the whole

• Better to ensure parts work properly before integrating

• Full TDD is great but hard to achieve

• Component Tests provide more direct value at lower cost

• Especially when dealing with legacy code

3


Outline

• Why is (Automated) Testing Necessary?

• Why is Traditional Test Automation Hard?– Systems diagram without preamble/legend

• Why is TDD Hard? (Optional)– Hard to convince and a process (hard to enforce)


Audience Survey

https://lmsa.site-ym.com/news/221726/Sex-and-Gender-Medical-Education-National-Student-Survey.htm

4


Questions

• Who likes using software?

• Who likes using BUGGY software?

• Why is it buggy?

• How hard is *your* code to test?

• What makes code hard to test?

• Should code be easy to test?

• How can we make our code easier to test?

• Whose job is “quality”?

• What kinds of tests do we need?

• Difference between EDD and BDD?


Johnny’s Report Card

http://fabulesslyfrugal.com/report-card-freebies-for-good-grades/

5


Our Report Card


Functionality

Performance

Ease of Change

Reliability


Our Report Card


Functionality

Performance

Ease of Change

Reliability

Report Card

6


QA as Quality Assessment


Dev

Test & Fix “Battleship”

DevelopmentQuality

Assessment

Requirements

B6! Missed!

Requirements

Test

7


Insanity:

• Doing the same thing over and over again and expecting different results


The Toyota Way

8


5 Why’s

www.123rf.com


5 Why’s of Test Automation Being Hard

Tests need to interact via

the UITest

Automation is Hard!

Product isn’t designed for testability

Why? Why?

Why?

Tests are very complex

9



Dev doesn’t know how (or care)


the UITest

Automation is Hard!

It’s QA’s problem

Tests are automated after

Dev Complete

QA isn’t involved in product

definition


Why? Why?

Why?

Why?

Why?

Why?


Why?

Why?





the UITest

Automation is Hard!



Dev Complete


definition

QA is too busy testing last release


Why? Why?

Why?

Why?

Why?

Why?

Why?

Why?


Why?

Why?

QA is supposed to be

“independent”

Why?

10


Systems Thinking

A B+

C-

Notation:+ Increasing- Reducing


Systems Thinking

A B+

C-

+ +


ReinforcingCycle

BalancingCycle

11


Systems Thinking

A B+

+


Vicious orVirtuous

Cycle

(Depending on whether A & B

are good or bad)





the UITest

Automation is Hard!



Dev Complete


definition



Why? Why?

Why?

Why?

Why?

Why?

Why?

Why?


Why?

Why?


“independent”

Why?

12


Test Automation – Systems Thinking Diagram



the UITest

Automation is Hard!



Dev Complete


definition



+ +

+

+

+

+

+

+


+

+


“independent”

+

A Vicious Cycle


Root, Root Cause: Conway’s LawThe Architecture of System will resemble the

Organization that Built it.

Development Team

TestTeam

BusinessTeam

Req’ts Code

13


Resulting Architecture

Specifi-cations

Product Code

Test Code

Bus Test

Dev


Automated Tests are Just Code

That Tests Other Code

Who Tests This Code?Who Should Write This Code?

14


TDD Is One Solution

But is hard to implement because:

• Requires everyone to do it

• all the time

• And is a large cultural change

And it doesn’t solve the whole problem:

• Ensures “code quality” but not “product quality”

• Can build great software that doesn’t work as a whole


Example-Driven DevelopmentDevelopment flow

Potentially Executable

Specifications

ExampleInterpreter

Code ProductCode

DevTest

DevDevTestBus Test

Expertise or Roles , not Teams!

15



Example-Driven DevelopmentRuntime View:

ProductCode

ExampleInterpreter

Code

ReportCard


Specifications

DevTest

DevDevTestBus Test



Example-Driven DevelopmentFeedback Loops:


Specifications ProductCode

ExampleInterpreter

Code

ReportCard

What they say

How they say it

Dev

DevTest

DevTestBus Test

16



Example-Driven DevelopmentFeedback Loops:


Specifications ProductCode

ExampleInterpreter

Code

ReportCard

What they say

How they say it

DevTest

DevTestBus TestDev

Isn’t This “Just” Test Automation By Another Name?


Tests vs. Checks vs. Examples?ref: http://www.satisfice.com/blog/archives/856

17


Preparing Good Examples


Example: Notifying of Bank Transactions

Configure Notification

Rules

AccountHolder

Process Transaction

TransactionSettlement

Not

ify

base

d on

cha

rge

Type

18




Rules

AccountHolder

Process Transaction


Not

ify

base

d on

cha

rge

Type

Not

ify

base

d on

cha

rge

Cont

inen

tN

otif

y ba

sed

on c

harg

e Co

untr

yN

otif

y ba

sed

on c

harg

e Ci

ty &

Sta

te




Rules

AccountHolder

Suspend Notification

Resume Notification

Process Transaction


Not

ify

base

d on

cha

rge

Type

Not

ify

base

d on

cha

rge

Cont

inen

tN

otif

y ba

sed

on c

harg

e Co

untr

yN

otif

y ba

sed

on c

harg

e Ci

ty &

Sta

te

Sus

pend

Not

ific

atio

n

19


Checking Notifications• Open MegaBank app

• Log in as “BobMa” with password ******

• Click on “Manage Notifications” tab

• Click on “Add New Rule” button

• Select account “10035692877”

• Type “Default Rule” into field “rule name”

• Type “1000” into field “threshold amount”

• Click on “all transaction types” radio button

• Click on “all locations” radio button

• Click on “save changes” button

• …

• …


Checking Notifications – 1/2Given:

User and Accounts

Another attempt:

When: Notification

Rule is Configured

Then:Notification Rule Configured and

Active

Then:Notification Rule

is Active

20


Checking Notifications – 2/2

Then: ExpectedNotifications

When: The Transactions to be processed

Another attempt:


Test Automation Pyramid

Pyramid originally proposed by Mike Cohn

Exploratory Tests

Unit Tests

ComponentTests (API)

SystemTests (GUI)

21


Behavior Specification at Right Level


SystemTests (GUI)

Broad Narrow

Det

ail

High

Low

Scope

Unit Tests TDD/BDD

ATDD/EDD




SystemTests (GUI)

Broad Narrow

Det

ail

High

Low

Scope

ATDD/EDD

22




SystemTests (GUI)

Unit Tests TDD/BDD

ATDD/EDD




SystemTests (GUI)

Broad Narrow

Det

ail

High

Low

Scope

Unit Tests TDD/BDD

ATDD/EDD

23



Broad Narrow

Det

ail

High

Low

Scope


SystemTests (GUI)


Test Automation Pyramid - Redrawn


Broad Narrow

Det

ail

High

Low

Scope

Unit TestsTDD/ BDD

ATDD/EDDSystemTests

24



Broad Narrow

Det

ail

High

Low

Scope


SystemTests ATDD/EDD



Broad Narrow

Det

ail

High

Low

Scope


SystemTests

25



SystemTests


Rules &Algorithms

Multi-Use Case

Workflows

IncompleteSpec

Toomuch detailUnmaintainable

Det

ail

High

Low

Broad NarrowScope

Transactions(Use Cases)


Toomuch detailUnmaintainable

Changing Level of Abstraction/Detail

Rules &Algorithms

Multi-Use Case

WorkflowsTransactions(Use Cases)

IncompleteSpec

Broad Narrow

Det

ail

High

L

ow

Scope

26


Refactoring Workflow ExampleGiven:

User and Accounts

When: Notification

Rule is Configured

Then:Notification Rule Configured and

Active

Then:Notification Rule

is Active


Refactoring Workflow Example

“If it isn’t essential to conveying the essence of the behavior, it is essential to

not include it.”

27


Then: ExpectedNotifications

When: The Transactions to be processed




28




Refactoring Workflow ExampleGiven: User &

Thresholds

When: Transactions

Are Processed

Then: We Expect

Notifications

29


9991113333

Given: User &

Thresholds

When: Transactions

Are Processed

Then: We Expect

Notifications

credit

credit



9991113333


Broad Scope (Multi-Actor);

Minimum Detail (per Actor/Transaction);

Given: User &

Thresholds

When: Transactions

Are Processed

Then: We Expect

Notifications

credit

credit

30


Filling in the Details

Rules &Algorithms

Multi-Use Case

WorkflowsTransactions(Use Cases)

IncompleteSpec

Too much detailUnmaintainableBroad Narrow

Det

ail

High

Low

Scope


Business Rule Example

Process Transaction

Threshold per Charge Type

Configuration

31


Process Transaction

Threshold per Charge Type

Narrow Scope (Single Rule)

High Detail (Everything that matters)

Example:

Configuration Given these rules

Then: The answer should be

Business Rule ExampleWhen we ask

NotificationRequired? with this transaction:


Life Cycle of an Example

User Goal

Feature

StoryTitle

Story Narrative

Story Scenarios

Story Examples

Executable Examples

Satisfied Examples

FormalizeAutomate

DevelopProduct

Make ConcreteAdd Data

DefineAcceptance

Criteria

WorkBreakdown

Product Design & Validation

Elaborate

Each

32


Who Does Activities on an Example

User Goal

Feature

StoryTitle

Story Narrative

Story Scenarios

Story Examples

Executable Examples

Satisfied Examples

FormalizeAutomate

DevelopProduct

Make ConcreteAdd Data

DefineAcceptance

Criteria

WorkBreakdown

Product Design & Validation

Elaborate

DevTestBus

DevTest Bus

DevTest

Bus

Each


Test-After Architecture

System Under Test

ProcessTransaction

ConfigureNotificationThreshold

TransactionInterface

ConfigurationUser

Interface Should weNotify

andDo

Notification?

NotificationRules

NotificationLog

Bus

Workflow Test

Test Dev

! ! ! !

33



ConfigurationUser

Interface

System Under TestConfigure

NotificationThreshold

Example-Driven ArchitectureWorkflow Examples Drive API

Workflow Example

Dev

Workflow Keyword

Interpreter

Bus

Test

ProcessTransaction

Process Transaction (including:

Do We Notify and How to Notify)

Dev



ConfigurationUser

Interface

System Under TestConfigure

NotificationThreshold

Example-Driven Architecture... but not Component Structure

Workflow Example

Dev

Workflow Keyword

Interpreter

Bus

Test Dev

NotificationMethod Example

NotificationRule Example Process

Transaction

System Under Test

34


Example-Driven ArchitectureRules Examples Drive Component Dev’t

Bus

Test


NotificationRule Example

System Under Test


System Under Test

Example-Driven ArchitectureRules Examples Drive Component Dev’t

DoNotification.

NotificationLog



Should weNotify?

ExampleInterpreter

ExampleInterpreter

DevBus

TestDev

NotificationRules

35


NotificationRules

Example-Driven ArchitectureRules Examples Ensure Testability

Should weNotify?

NotificationRules Fake


ExampleInterpreter

DevBus

TestDev


System Under Test

Example-Driven ArchitectureRules Examples Ensure Testability

DoNotification.

NotificationRules

NotificationLog


Bus

TestDev

NotificationRules


NotificationLog Fake

NM ExampleInterpreter

Dev

36


System Under Test

Example-Driven ArchitectureUnit Tests Fill in Gaps in Coverage

DoNotification.

NotificationRules

NotificationLog


Bus

TestDev

NotificationRules


NotificationLog Fake

NM ExampleInterpreter

Dev

UnitTestsUnit

TestsUnitTestsUnit

Tests


System Under Test

Putting it All Together

DoNotification.

ProcessTransaction

NotificationRules

NotificationLog




Should weNotify?

ExampleInterpreter

ExampleInterpreter

DevBus

Test Dev

UnitTestsUnit

TestsUnitTestsUnit

Tests


ConfigurationUser

Interface

ConfigureNotificationThreshold

Workflow Example

Workflow Keyword

Interpreter

37


tools

Tools

As adapted by Gerard Meszaros


Tools (My Prefs Only)

38


In Conclusion



BusinessRule Examples

Broad Narrow

Det

ail

High

Low

Scope

Unit Testsor Specs (BDD)

TDD/ BDD

ATDD/EDDWorkflowExamples

39


After State – Back to Our Systems Diagram



the UITest

Automation is Hard!



Dev Complete


definition



+ +

+

+

+

+

+

+


+

+


“independent”

+

X

XX

X X

XXX


After State – Back to Our Systems Diagram

Dev knowshow (and cares)

Examples don’t need to interact via

the UI

Automated checking is

Easy!

Automation is Dev’sproblem

Examples are automated before Dev

starts

QA isinvolved in product

definition

QA has time for Examples and Exploratory

testing

Product isdesigned for testability

+ +

+

+

+

+

+

+

Examples are simple

+

+

A Virtuous Cycle

40


Benefits• Defining Examples first helps us understand

– What the system should do

– How it should be structured to ease testability

• Using business language for examples– Reduces opportunities for “lost in translation”

• Developers responsible for automation– Ensures the design is testable

– Provides immediate feedback on function & testability.

– Reduces duplication with unit tests

• Applicable to new code *and* legacy code– Start with Workflow or Component tests, not unit tests


Thank You!Gerard Meszaros

[email protected] http://www.xunitpatterns.com

Slides: http://itake2017.testAutomationpatterns.com

Call me when you:• Want to transition to Agile or Lean• Want to do Agile or Lean better• Want to teach developers how to test• Need help with test automation strategy• Want to improve your test automation

Jolt Productivity Award winner - Technical Books

Available on MSDN:

41


Tools Selection

• Tools must support the strategy– Support business/testers to write specs/tests in

business language» More or less “natural” language

– Let them write specs before the specs are automated or the system is built» This disqualifies Capture & Replay tools

– Can be automated by technical people, later» Separation of business skills from technical skills

» e.g. Table or Keyword-based tools

– Can be run by anyone, especially developers» This disqualifies expensive tester-specific tools


Business Rules Tools - A Sampling

• Fit– Column Fixture

• SLIM (Fitnesse) – Decision Table

• Robot Framework – Parameterized Testcase

42


Workflow Spec Tools (my fav’s)

• Gherkin (Given-When-Then terminology)– Cucumber (various tech-specific implementations)

– JBehave (Java/JVM specific)

– SpecFlow (.Net)

• Other GWT (Given-When-Then terminology)– Robot Framework GWT

» https://blog.codecentric.de/en/2009/11/givenwhenthen-and-example-tables-using-the-robot-framework/

• Robot Framework– Keyword-based Testcases

Resource: Agile Alliance Functional Testing Tools program


More Complete Survey of Tools

• Agile Alliance Functional Testing Tools program Yahoo! group– https://groups.yahoo.com/neo/groups/aa-ftt/info

• Agile Alliance Functional Testing Tools spreadsheet– https://docs.google.com/spreadsheet/ccc?key=0Apag_J

97l3CTdHR0WS1sZGFVaFA0dEpXYWRqLXBxV3c#gid=1

Documents

Meszaros Itake Keynote v5 - XUnitPatterns.comxunitpatterns.com/~gerard/itake2017keynote.pdf · &rs\uljkw *hudug 0hv]durv kwws lwdnh whvw$xwrpdwlrq3dwwhuqv frp,w $lq¶w7hvwdeoh 8qwlo