How To Grade Your Selenium Tests
Special Thanks to Dave Haeffner!
http://elementalselenium.com
What We’ll Cover• Good/bad Tests, Testing Rubric
• Good/bad Page Objects, PO Rubric
• Good/Bad Locators, Locator Rubric
• Report Card
• A Concept For Automating This
Who, Me? Okay• Marcus Merrell (@mmerrell)
• Manager, Web Platform Engineering Team at RetailMeNot, inc
• QA Test Automation from 2001-2015
• Selenium/WebDriver since 2007
• The “bass player” in all things
Let’s talk about tests
What Makes a Bad Test?
• Fails For No Good Reason
• Changes in AUT, Need to Update Test(s)
• Slow
• Unreliable
• Hard to Understand and Maintain
What Makes a Good Test?• Written for BDD or xUnit test framework• Test one thing (atomic)• Each test can be run independently (autonomous,
“idempotent”)• Anyone can understand what it is doing (easy to read)• Similar tests are grouped together• Centralized setup/teardown• Uses Page Objects
Some Solid Tests
A Testing Rubric(Each test starts with 100 points, deduct as necessary)
Item Belong in tests? Score
Selenium Commands No -3 per (max of -9)Locators No -2 per (max of -8)Selenium
setup/teardown* No -20
Hard-coded sleeps No -5 per (max of -20)Implicit wait calls No -10Explicit Wait calls No -3
Conditionals No -5 (max of -20)Calls to Page Objects Yes N/A
Assertion(s)* Yes N/A
Let’s talk aboutPage Objects
http://martinfowler.com/bliki/PageObject.html
What Makes a Bad PO?
• Gigantic (i.e., hundreds or thousands of lines of code)
• Over-reaching responsibility
• Contains overly complicated logic
• Return not enough or too much information to the test (leaky abstraction)
• Assertions happening in the PO instead of the test
What Makes a Good PO?
• Contains State (e.g., locators)
• Contains Behavior (e.g., methods to interact with the page)
• Returns some information about the page (e.g., new page object, text from the page, a boolean result for some check, etc. — never a WebElement)
• Verifies page ready state as part of initialization with a found Element
A Page Object Rubric(Each Page Object starts with 100, deduct points as necessary)
Item Belong in POs? Score
> 200 lines of code No -5 per 50 lines over
Assertions No* -5 per (max of -20)
Hard-coded sleeps No -5 per (max of -20)
Implicit wait calls No -10
Explicit waits Yes N/AVerify page ready
state* Yes -8 if not verifying an element
Locators* Yes N/A
http://se.tips/se-waiting-jim-evans
Let’s talk about locators
Locator Strategies• Class• CSS selectors• ID• Link Text• Partial Link Text• Tag Name• XPath
Good locators are:• unique• descriptive• unlikely to change
That rules a few of these out
Locator Strategies• Class• CSS selectors• ID• Link Text• Partial Link Text• Tag Name• XPath
Good locators are:• unique• descriptive• unlikely to change
That rules a few of these out
Locator Strategies• Class
• CSS selectors• ID
• Link Text• Partial Link Text• Tag Name• XPath
Good locators are:• unique• descriptive• unlikely to change
That rules a few of these out
Start with IDs and Classes
Locator Strategies• Class
• CSS selectors
• ID
• Link Text• Partial Link Text• Tag Name• XPath
Good locators are:• unique• descriptive• unlikely to change
That rules a few of these out
Start with IDs and Classes
Use CSS or XPath (with care)
A Locator Rubric(Each locator gets a grade)
Item Letter Grade E.g.,
Dynamic locators D-User account
specific or tied to page render
Tied to page layout D XPath: / / /CSS: > > >, etc.
Text on the page C Link text, page copy
Reasonable traversal B Parent to child w/in
an element nodeUsing semantic
name B+Input labels
(name='username')
Semantic ID AUnique,
descriptive, unlikely to changeNon-unique locator? -2 full letter grades
So…
What’s your testing GPA?
Topic Weight
Tests 20%
Page Objects 30%
Locators 50%
Example ReportSummary
B-Suite Full Suite
Tests 87Page
Objects 62
Locators 90
Example ReportDrill-down
Test Suite Test Name Test PO Locators OverallUser Login Happy Path 87 90 90 89.4User Login Invalid Email 92 78 93 88.3User Login Insecure PW 99 68 91 85.7User Login Mismatched
PW75 56 90 76.8
User Login Missing Req’d 82 34 95 74.1Store Page Offer Sort 91 62 85 79.3Store Page Code-type 83 44 89 74.3Store Page Sale-type 87 62 87 79.5Analytics Outclick 88 N/A N/A 88Analytics Offer Rank 86 N/A N/A 86Analytics Campaign 87 N/A N/A 87
Example ReportTest Level
Test Name Step Locator Score OverallCode-type Login Xpath(‘//root/@home/@login’) D Relative to rootCode-type Search
‘macys.com’Css(‘class: q’) B Semantic CSS
LinkText(‘submit’) C LinkText brittleCode-type Click Macy’s Id(‘macys.com’) A Gold StandardCode-type Filter Codes Id(‘filterCode’) A Gold StandardCode-type Click Top Code Xpath(‘@div[‘code-link’][0]’) C Brittle
Next Steps• Simple to implement, e.g. IntelliJ plug-in
• Event Listener
• Java/JS SDKs in testing
• Reporting needs work (conceptually)
• Service Performance
Thank You!!
Questions?