Upload
martingijsen
View
515
Download
2
Tags:
Embed Size (px)
DESCRIPTION
Free Test 2009 conference slides
Citation preview
Exploring no man's landwith keyword driven testing
Martin Gijsen, [email protected]
Free Test ConferenceMarch 23, 2009
Presentation overview● Automated testing● When (not) to use Record & Playback● Automated testing as no man's land● Keyword driven testing● The process● Where to get the automator
Automated (functional) testing● A computer performs the test● Fast and consistent● Potential benefits:
● Time (to market) / cost / quality● More effective use of (test) resources● More challenged (= happier) testers● Accurate and up2date test status
● Continuity● Business goals are drivers
The Record & Playback approach● Record test actions, replay as needed● Checks are added to scripts● Fundamental problem is maintenance
● Software evolves until it is decommissioned● Maintainability requires structure and abstractions● Recorded test cases are long, unstructured and low
level programs● Maintenance sensitivity is high● Maintainability is low
So why is it still used?● Automated testing is often the job of a tester● Most testers are:
● willing to record, but● unwilling or unable to create good automation
solutions● Vendor sales pitch is not always clear on this
Result and conclusion
It is often initially unclear whateffective automated testing requires
2 out of every 3 TA projectsfails sooner or later
– Brian LaSuer
Use R&P only in case oflittle manual maintenance
What about the developers?● Test automation is software engineering● Developers have the skills to create a good
automation solution● Many developers are unwilling or unable to
create good test sets● … or automate test cases that testers wrote
Welcome to no man's land● Most testers cannot or will not automate well● Most developers cannot or will not test well● Automated testing is like a no man's land in
between testing and development
● Is automated testing doomed?
Exploring no man's land
Testing
Automated testing
Development
Testing
Automated testing= analysis (what to test)
+ automation (how to test)
Development
Exploring no man's land
What is in between?
Testing
?
Development
Test analysis
Test automation
No more no man's land?
Testing
Development
Test analysis
Test automation
Keywords – The DS(T)L● Domain Specific (Test) Language
● Used to express test cases● Defined by (or with) the testers● Specific to the application under test● No programming● High level
Bad example
select (“a box”);type (“some text”);click (“OK”);...
Better example
create account;1234567890;John;Doecreate account;2345678901;Jane;Doe
deposit;1234567890;12345,67
transfer;1234,56;1234567890;2345678901
check balance;1234567890;11111,11check balance;2345678901;1234,56
Good exampleAccount First name Last name
create account 1234567890 John Doe
create account 2345678901 Jane Doe
Account Amount
deposit John 12345,67
Amount From To
transfer 1234,56 John Jane
Account Amount
check balance John 11111,11
check balance Jane 1234,56
Keywords / Instructions● Often have arguments
● No (irrelevant) test execution details● No (irrelevant) interfacing details● No tooling details
● Only the essence of a test case remains● Easy to read, write and maintain test cases● Low maintenance sensitivity
Sample instruction documentationName and aliases 'begin test case' and 'begin testcase'
Description Indicates the beginning of a (new) test case.
Parameters Two parameters, both optional
1 The identification of the test case, printable characters, preferably unique within the test. Optional.
2 The test case description, printable characters. Optional.
Validity Anywhere except inside a procedure definition.
Pre-condition None.
Post-condition If the previous input line was part of a test case, that test case is closed. A new test case is opened and assigned the next sequence number for the report(s), starting at one.
Error condition If in a procedure definition.
Automating (basic)
AutomationTest Testreport
Application under test
Automating (more detailed)
EngineTest Testreport
Application under test
Instructions
Interfacing
The ETA Framework 'engine'● Can read from and report to a spreadsheet● Supports test cases, variables and procedures● Implement keywords in Java● Use any Java library (Abbot, WebDriver, MQ,
generated web service client, ...)● Documented, actively used and developed● Freeware
The process (waterfall)
Requirements
Keyword defs
Product development
Test cases
Automate Test
The process (agile)
Requirements
Keyword defs
Product development
Test cases
Automate TestRequirements
Keyword defs
Product development
Test cases
Automate Test
Requirements
Keyword defs
Product development
Test cases
Automate TestRequirements
Keyword defs
Product development
Test cases
Automate Test
Requirements
Keyword defs
Product development
Test cases
Automate TestRequirements
Keyword defs
Product development
Test cases
Automate Test
Where to get the automator● Main criteria (for continuity):
● Skills● Availability● Cost
● Educate someone from test team● Borrow from development team● Hire or reserve developer for test team● Hire external consultant
Summary● Automated testing can help achieve your goals● Avoid Record & Playback● Good keywords help:
● The tester can focus on the testing● The developer can focus on developing● Easy to write test cases and little maintenance
● A good test engine also helps● Use skilled automators● The automators must be available when needed
Questions
?
Automation design
EngineTest Testreport
Application under test
Instructions
Interfacing
Design for a telephony switch
Engine
Telephony switch
Instructions (C++)
Existing library (iTCL)
Design for a voice portalEngine
Voice portal
Instructions (C++)
Tool
Tool library
Design for web services (1)
Engine
Web services
Generic web serviceinstructions (Java)
Generatedweb service client (Java)
Design for web services (2)
Engine
Web services
Specific web serviceinstructions (Java)
Generatedweb service client (Java)
Design for a payments system
Engine
SEPA banking system
Iso20022 protocolinstructions Java)
SWIFT MT protocolinstructions (Java)
Web serviceinstructions (Java)
Web instructions(Java)
Messaging library (Java) Generated web serviceclient (Java)
Web driverlibrary (Java)