Selenium and Open Source Advanced Testing

Selenium at Scale: Advanced tips for success in open source testing in 2017 Web: perfectomobile.com Twitter: @perfectomobile

Selenium at ScaleAdvanced tips for success in open source testing in 2017

Web: perfectomobile.com Twitter: @perfectomobile


What We’ll Cover

● From Selenium 2.0 to 3.0

● The Role of Open Source in Continuous Testing

● Quality Across Platforms: Mobile+Web

● General Q & A


Poll: Concerning Selenium updates...

● I have to upgrade more than one (my) machine

● We upgrade as soon as a new stable version is released

● After upgrade, some of our scripts are broken

● We don’t find all incompatibilities right after the upgrade

● We often significantly delay updating to avoid being beta testers


Our Presenters

David DangVP of Automation Solutions

Zenergy

Darrell GraingerQuality Advocate / Automation Expert

ThoughtWorks

Eran KinsbrunerLead Technical Researcher

Perfecto


Darrell GraingerQuality Advocate / Automation ExpertThoughtWorks

From Selenium 2.0 to 3.0

ISelenium at Scale

Advanced tips for success in open source testing in 2017


A Brief History of Selenium

1.0

rich interface for browser operations

tweaks on every browser update

Support EoL a few years ago

2.0

building blocks to your own DSL

execute 1.0 and 2.0 script

Still supported*

2004 2007

Selenium RC (1.0) WebDriver

2016

Selenium 3.0

2012

Selenium WebDriver

(2.0)


From Selenium 2.0 to 3.0● No philosophical changes from 2.53.x to 3.0

● In theory, upgrading from 2.53 to 3.0 shouldn’t hard… in theory

● To open a Firefox browser in 2.53 using the Java bindings:

WebDriver driver = new Firefox();

● To open a Firefox browser in 3.0 using the Java bindings:

System.setProperty("webdriver.gecko.driver", ”~/geckodriver");

DesiredCapabilities dc = DesiredCapabilities.firefox();

WebDriver driver = new FirefoxDriver(dc);


What does Selenium 3.0 have?Gain Loss

Java bindings Compatibility to 3.0 Hard stop for Java 6/7

Ruby bindings Compatibility to3.0 Need to use Ruby 2.x+

Deprecated functions New replacement functions Old functions removed

Safari 10+ Compatibility to 3.0 Does not work on Safari 9<

Firefox 47.0.1+ Compatibility to 3.0 Does not work on 47.0<

Support IE 9+ Compatibility to 3.0 IE8 not supported


Why stick with the devil you know (i.e. 2.0)?

● There is what the documentation says and what ACTUALLY works

● In certain tech stacks and websites, some things no longer work in 3.0

● Example: site which is using react.js and you need WebDriverWait calls

● However there are things which worked on Selenium 2.0 which no longer

work on Selenium 3.0.○ For example, populating the From field on Southwest’s flight booking website.

○ This does not work when using Firefox

(example)


Code Exampledriver.get("https://www.southwest.com/flight/?clk=GSUBNAV-AIR-BOOK");

WebElement fromInput = driver.findElement(By.cssSelector("#originAirport_displayed"));waitFor.until(ExpectedConditions.elementToBeClickable(fromInput));fromInput.click();waitFor.until(ExpectedConditions.elementToBeClickable(fromInput));fromInput.sendKeys("new york");By resultsLocator = By.cssSelector("div.ac_results li");waitFor.until(ExpectedConditions.numberOfElementsToBeMoreThan(resultsLocator, 3));List<WebElement> suggestions = driver.findElements(resultsLocator);for (WebElement suggestion : suggestions) { String s = suggestion.getText(); System.out.println(s); if(s.equals("New York/Newark, NJ - EWR")) { suggestion.click(); // does not work with Firefox break; }}screenshot();


The usual Selenium baggage● Environment setup is also a consideration

● Downloading the latest drivers and bindings

● Setting up nodes in a Selenium Grid

● Updating the development setup with the new drivers and bindings

● Will the developers need to upgrade the programming language?

● Will you have to upgrade the build agents on your CI?

● Do you need to have support and security vet the new technology?

● When working in enterprises, upgrading the tech stack can be a big deal

● You might need to involve different departments


When switching to Selenium 3.0...● Selenium 3.0 basically requires you to beta test the bindings

● Why not wait for other people to work out the bugs?

○ You need to support the latest browser

○ You need to use the latest binding languages

○ You’ll need to switch at some point, no better time than the present


More info on moving from Selenium 2.0 to 3.0● Good place to find API documentation and changelogs:

○ http://www.seleniumhq.org/download/

● The chromedriver:○ http://chromedriver.storage.googleapis.com/index.html

● The geckodriver for Firefox:○ https://github.com/mozilla/geckodriver/releases

● The safari driver:○ This is actually part of macOS

○ https://webkit.org/blog/6900/webdriver-support-in-safari-10/

● Selenium source code:○ https://github.com/SeleniumHQ/selenium


Flash Q&A : DarrellQuality Advocate / Automation Expert

ThoughtWorks


David DangVP of Automation SolutionsZenergy

The Role of Open Source in Continuous Testing

IISelenium at Scale



● As the industry standard tool for browser automation, countless tools are being created that extend or integrate with Selenium WebDriver

● Tools like Appium and Protractor extend WebDriver for use in other technologies

● Tools like Cucumber integrate with WebDriver so it can be used with methodologies such as BDD

Application Under Test

Selenium WebDriver

WebDriver: the foundation for open source testing


● More technical oriented compare purchase tools

● Decision on a test automation framework is not always straight forward

● Integration with existing toolset can be difficult

● Keeping up with the latest a greatest is always a struggle

● Limitation of technologies support (client/server, ERP, CRM, mainframe, etc…)

Challenges with Open Source Testing at Scale


● What comes for free → WebDriver community, bindings, integrations

● What comes with free → costs: environment config, customization, skills

● What doesn’t come for free → People, technologies, process

○ Culture - Dev/Tests together?

○ Enterprise-ready support

○ Tools and external libraries

Key Considerations for Upgrading Selenium


Selenium 3.0: What you need to be successful

● What you need to be successful

○ Dedicated resources

○ Test automation strategy

○ Test automation framework (page object model, keyword, data-driven, etc…)

○ Integration with other tools/frameworks (Maven, TestNG, Apache POI, Log4J, ReportNG, Serenity, Jenkins, etc…)

○ Development and maintainence plan


Flash Q&A : DavidVP of Automation Solutions

Zenergy


Eran KinsbrunerLead Technical ResearcherPerfecto

Quality Across Platforms: Mobile+Web

IIISelenium at Scale



Matching Tools to Requirements and Persona’s

● Different type of apps (Native/RWD/Web)

● Organizational skill set

● Context Based Testing (In-App/External)

● Embedded into workflow/IDE’s?

● Test coverage requirements (Platforms/Capabilities)


A Day in A Life – Covering Real Environments Isn’t Possible With Every Tool

12/3/2016

Automation ought to cover the full user environment not just the Application Under Test!


Which Tool is the best fit for you?

12/3/2016

24© 2016, Perfecto Mobile Ltd. All Rights Reserved.

SELENIUM APPIUM CALABASHESPRESSO XCTEST UI

When looking at today’s open-source mobile test automation landscape, there

are five highly-adopted test frameworks.

We have summarized each approach to help you decide which makes the most sense for you.

Each tool has advantages for your mobile and web testing depending on your needs.

While reading, consider which of the functions you need most for your own test automation purposes.


Making the DecisionA Comparison

Detailed analysis of each test automation tool:

The next slides offer a breakdown of each tool

http://blog.perfectomobile.com/test-automation/selecting-best-open-source-test-automation-tool/?utm_source=PMSlideshare&utm_medium=BLG20160609


Selenium

• The best choice for web test automation teams testing for responsive web design or stand-alone web sites

• Less suitable for developing unit testing, making this framework less appealing for developers

• Its core test reports are not highly informative and lack unique mobile related insights


Appium

• Best suited for QA teams testing the functionality of native, mobile web and mobile hybrid apps across iOS and Android

• Less suitable for developers who wish to develop and perform unit testing

• Appium reports are a bit limited from a debugging and fast feedback loop perspective, and do not include videos, network logs and key vitals information


Calaba.sh

• Designed for organizations that work in behavior-driven development workflows

• Offers an easy path to both develop and test features in parallel, all in an easy user-flow based language

• Appealing for both dev and QA practitioners

• Provides solid insights and reports to both dev and QA teams


Espresso & XCTest UI

12/3/2016 © 2016, Perfecto Mobile Ltd. All Rights Reserved.

ESPRESSO

XCTEST UI

• Both are designed for target users• Espresso is for Android and XCTest is for iOS• Both tools are fully integrated into development IDEs such as Android

Studio/Xcode, and offer very easy-to-develop techniques, including test recorders

• Fully maintained by Google and Apple, which assures they always support the latest OS features so developers can stay ahead of the market and test accordingly

• Support both unit testing types and functional UI testing• Both are app-context only, which limits their abilities to test for user condition

scenarios

ESPRESSO


Things to Consider

• Lab: Having support for variety of frameworks is important for enterprises • Flexibility• Autonomy• complete E2E coverage• unattended testing

• Automation: support for multiple OSS frameworks becomes important• Different “persona’s” need different tools under 1 lab• Cross platform scripting• Robust framework to support high test coverage

• Analysis: Digital reporting is necessary for teams to take action and fix issues faster & earlier

When choosing a test automation tool


LIVE DEMO – RWD Cross Platform Automation


The Digital Solution“One Script, One Lab” for all digital platforms

Digital Test Suite Reporting for Agile Teams

One Script

For All Digital Platforms

One Lab – Web & Mobile

One Secure Cloud

Manual & Automated Testing


Flash Q&A : EranLead Technical Researcher

Perfecto


Thank you!Next webinar: projectquantum.io

bit.ly/perfecto-bdd-quantum