Node.js and Selenium WebDriver

A journey from the Java side

19th Nov SF Selenium Meetup @ Airware

Mek Srunyu Stittri

Volkan Gurel

2

Agenda

● Background● Challenges, overcoming the asynchronous● Page object implementation● Applitools screenshot validation integration● Fitting in http request for REST API tests● After tests: Continuous deployment with

Hashicorp Kubernetes and Docker

Background

4

Background

Back in June start looking at node.js for selenium E2E functional test framework.

● Kept Node.js adoption in mind● More and more company moving to node and

going full stack.● Share code with developers and get help

5

Problem statement

Disconnected engineering stack

QA, Automation engineers

Frontend engineers

Backend engineers

Java

Javascript

Java

Python

Javascript

Java

Ruby

Javascript

Node.js

Company A Company B Company C

Node.js

Javascript

Node.js

6

Typical Frontend Engineer

7

Typical Automation engineer

8

Googling node.js and selenium

Results● NightwatchJS● WebdriverIO ● WD.js● The Intern● webdriver-sync● Cabbie● Selenium-Webdriver● Protractor ● And many more..

ChallengesOvercoming the Asynchronous

10

Javascript 101

Javascript is AsynchronousExample

var source = ['foo', 'bar', 'baz'];

var result = [];

setTimeout(function () {

for (var i = 0 ; i < source.length ; i++) {

console.log('Stepping through : ' + source[i]);

result.push(source[i]);

console.log('Current result: ' + result);

}

}, 1000); // Wait 1000 ms to finish operation

console.log('Result: ' + result);

console.log('Finished!!');

Output:

Result: ←------- Empty array ?!?!

Finished!!

Stepping through : foo

Current result: foo

Stepping through : bar

Current result: foo,bar

Stepping through : baz

Current result: foo,bar,baz

11

First 2 weeks...

I from Javaland

12

Callbacks

Callback Pattern

driver.get("http://www.google.com", function() {

driver.findElement(By.name("q"), function(q) {

q.sendKeys("webdriver", function() {

driver.findElement(By.name("btnG"), function(btnG) {

btnG.click(function() {

driver.getTitle(function(title) {

assertEquals("webdriver - Google Search", title);

});

});

});

});

});

});

Equivalent Java code

driver.get("http://www.google.com");

driver.findElement(By.name("q")).sendKeys("webdriver");

driver.findElement(By.name("btnG")).click();

assertEquals("webdriver - Google Search", driver.

getTitle());

Pyramid of doomhttps://en.wikipedia.org/wiki/Pyramid_of_doom_(programming)

13

Callback example

Marcel Erz’s Southbay Selenium Slides - NodeJs based selenium

Work on your webelement here

14

Promises

Promise Pattern

driver.get("http://www.google.com"). then(function() { return driver.findElement(By.name("q")); }). then(function(q) { return q.sendKeys("webdriver"); }). then(function() { return driver.findElement(By.name("btnG")); }). then(function(btnG) { return btnG.click(); }). then(function() { return driver.getTitle(); }). then(function(title) { assertEquals("webdriver - Google Search", title); });

Equivalent Java code

driver.get("http://www.google.com");

driver.findElement(By.name("q")).sendKeys("webdriver");

driver.findElement(By.name("btnG")).click();

assertEquals("webdriver - Google Search", driver.

getTitle());

15

Promise example

Marcel Erz’s Southbay Selenium Slides - NodeJs based selenium

Work on your webelement here

16

Back to the list

● NightwatchJS● WebdriverIO formerly WebdriverJS● WD.js● The Intern● webdriver-sync● Cabbie● Selenium-Webdriver now WebDriverJs● Protractor ● And many more…

WebDriverNode jwebdriver ot-webdriverjs burnout testium yiewd nwd co-nwd selenium-node-webdriver nemo.js taxi etc...

???

17

Experimenting with Nightwatch

What Nightwatch offers● Convenient chain APIs

○ A workaround for dealing with callbacks● Wraps Selenium JsonWireProtocol● Some form of page object support● Some extendability custom

commands● Saucelabs / Browserstack integration

out of the box● Pretty good documentation

Nightwatch test

module.exports = {

'Demo test Google' : function (browser) {

browser

.url('http://www.google.com')

.waitForElementVisible('body', 1000)

.setValue('input[type=text]', 'nightwatch')

.waitForElementVisible('button[name=btnG]', 1000)

.click('button[name=btnG]')

.pause(1000)

.assert.containsText('#main', 'Night Watch')

.end();

}

};

18

Experimenting with WebdriverIO

WebdriverIO test

client

.init()

.url('https://duckduckgo.com/')

.setValue('#search_form_input_homepage', 'WebdriverIO')

.click('#search_button_homepage')

.getTitle().then(function(title) {

console.log('Title is: ' + title);

// outputs: "Title is: WebdriverIO

})

.end();

What WebdriverIO offers● Convenient chain APIs● A+ Promise support● Wraps Selenium JsonWireProtocol● Saucelabs / Browserstack

integration out of the box● Some form of visual testing capability

○ Based on WebdriverCSS○ Limited support for Applitools

● But.. pageobject ??

19

WebdriverIO & Pageobject

https://github.com/webdriverio/webdriverio/issues/356https://github.com/webdriverio/webdriverio/issues/583

20

Chain based api - Nightwatch

Pageobject

this.clickLogout = function() {

browser

.waitForElement(USERNAME_DROPDOWN_TRIGGER)

.click(USERNAME_DROPDOWN_TRIGGER)

.waitForElement(LOGOUT_BUTTON)

.click(LOGOUT_BUTTON);

return browser;

};

Test code

testLoginProjectOwner: function (browser) {

browser

.page.Login().enterUserInfo(OWNER_USER,

DEFAULT_PASSWORD)

.page.Login().clickSignIn()

.page.Jobs().isJobListPresent()

.page.TopNavBar().verifyUserName("Project Owner")

.page.TopNavBar().clickLogout()

.page.Login().waitForLoginLoad();

}

The nice part

But.. starting to see a pattern forming : a chain within a chain

21

Chain based api - NightwatchThe not so nice..

browser .page.Login().enterUserInfo(OWNER_USER,DEFAULT_PASSWORD) .page.Jobs().getNumberOfJobs(function (result) { var numberOfJobsBefore = result.value.length; browser .page.JobConfig().createJob(jobName) .page.Jobs().getNumberOfJobs(function (result) { var numberOfJobsAfter = result.value.length; Assert.equal(numberOfJobsAfter, numberOfJobsBefore + 1); browser.page.Jobs().getJobs(function (result) { for (var i = 0; i <= result.length; i++) { if (result[i].name === jobName) { jobInfo = result; break; } } }); }).perform(function(client, done){ Assert.equal(jobInfo.name, expectedJobName, 'Job name is correct'); Assert.equal(jobInfo.creator, expectedJobCreator, 'Job creator is correct'); browser.page.TopNav().clickLogout() .end(); }); });}

Chain breaks once you start to do something complex that is not supported in the api

● Datastructure ● Iterating

22

Chain based api - NightwatchThe not so nice..

function getJobRow (index) {

var deferred = new Q.defer();

var jobInfo = {name: '', creator: '', date: ''};

browser.getText(JOB_LIST_ROW + ':nth-child(' + index + ') > td:nth-child(1)', function (result) {

jobInfo.name = result.value;

console.log('Retrieved job name ' + jobInfo.name);

browser.getText(JOB_LIST_ROW + ':nth-child(' + index + ') > td:nth-child(2)', function (result) {

jobInfo.creator = result.value;

console.log('Retrieved job name ' + jobInfo.creator );

browser.getText(JOB_LIST_ROW + ':nth-child(' + index + ') > td:nth-child(3)', function (result) {

jobInfo.date = result.value;

console.log('Retrieved job date ' + jobInfo.date);

deferred.resolve(jobInfo);

});

});

});

return deferred.promise;

}

23

Lessons learned

● Chain based api - a particular bad pattern when async call are involved. As soon as you try to do something complex (dealing with an array of WebElements) you end up having to break the chain.

● Page Object pattern and chained APIs don’t get along well.○ Methods end up containing another chain which does not help

with code composition. Also still prone to pyramid of doom● Most selenium chain based libraries gives you just one main object and

all interaction commands are tied to that object’s chain○ NightwatchJS : browser○ WebdriverIO : client

● Ignore Github Stars when choosing which projects to use...

24

Kinda miss Java synchronous programming languages at this point

1 month later...

25

selenium-webdriver WebDriverJs

Then we took a deep look at selenium-webdriver the current WebDriverJshttps://code.google.com/p/selenium/wiki/WebDriverJs#Writing_Tests

WebDriverJs uses a promise manager● Coordinate the scheduling and execution of all commands. ● Maintains a queue of scheduled tasks, executing each once the one before it in the queue is

finished. The WebDriver API is layered on top of the promise manager.

Provided Mocha Framework Wrapper with a built in promise manager

There is a built in wrapper for mocha methods that automatically handles all the calls into the

promise manager which makes the code very sync like.

http://selenium.googlecode.com/git/docs/api/javascript/module_selenium-webdriver_testing.html

26

Achieving sync-like code

Code written using Webdriver Promise ManagerJavascript selenium tests using promise manager

driver.get("http://www.google.com");

driver.findElement(webdriver.By.name('q')).sendKeys('webdriver');

driver.findElement(webdriver.By.name('btnG')).click();

driver.getTitle().then(function(title) {

console.log(title);

});

Equivalent Java code

driver.get("http://www.google.com");

driver.findElement(By.name("q")).sendKeys("webdriver");

driver.findElement(By.name("btnG")).click();

assertEquals("webdriver - Google Search", driver.getTitle());

Hey we look similar now!

27

Mocha with selenium wrapperAll callbacks can be omitted and it just works which makes the code very “synchronous” like.

Specifically, you don’t have to chain everything and each individual line of code can do only one

ui action and then some assertion if necessary.

var test = require('selenium-webdriver/testing');var webdriver = require('selenium-webdriver');var By = require('selenium-webdriver').By;var Until = require('selenium-webdriver').until;

test.it('Login and make sure the job menu is there', function() { driver.get(url, 5000); driver.findElement(By.css('input#email')).sendKeys('useremail@email.com'); driver.findElement(By.css('input#password')).sendKeys(password); driver.findElement(By.css('button[type="submit"]')).click(); driver.wait(Until.elementLocated(By.css('li.active > a.jobs'))); var job = driver.findElement(By.css('li.active a.jobs')); job.getText().then(function (text) { assert.equal(text, 'Jobs', 'Job link title is correct'); });});

28

Comparison

Library API structure Underlying implementation

NightwatchJs chain Its own JsonWireProtocol 3457 stars on github

WebDriverIO chain/promise Its own JsonWireProtocol 1322 stars on github

The Intern chain leadfoot 3095 stars on github

WebDriver-sync sync JsonWireProtocol ?? 72 stars on github

WD.js chain/promise Its own JsonWireProtocol 890 stars on github

Officialselenium-webdriver

promise & built-in promise manager

Webdriver API with native WebElement & Driver objects

2016 stars on github

Pageobjects

30

Designing frameworks

The magic number 7https://en.wikipedia.org/wiki/The_Magical_Number_Seven,_Plus_or_Minus_Two

The human brain can only focus on 7 ± 2 things at once. ● Handle all UI interactions and nuances in a common location

○ Stale elements, retries and etc.○ Mimicking an actual human in the UI

● Keep tests dry, more business facing methods and logic in page-objects● Easy to add tests

31

Object Oriented Javascript

● The world's most misunderstood prog language● There are no real classes● Inheritance - different from Java

○ prototype-oriented (has a) vs class-oriented inheritance (is a)○ http://www.crockford.com/javascript/javascript.html

● Recommended reading○ The Principles of Object-Oriented Javascript

Nicholas C. Zakas

32

BasePage.js

var driver;

/**

* Base constructor for a pageobject

* Takes in a WebDriver object

* Sets the Webdriver in the base page surfacing this

to child page objects

* @param webdriver

* @constructor

*/

function BasePage(webdriver) {

this.driver = webdriver;

}

...

LoginPage.js

var BasePage = require('./BasePage');

/**

* Constructor for the Login Page

* Hooks up the Webdriver holder in the base page allowing to

call this.driver in page objects

* @param webdriver

* @constructor

*/

function LoginPage (webdriver) {

BasePage.call(this, webdriver);

this.isLoaded();

}

// Hooking up prototypal inheritance to BasePage

LoginPage.prototype = Object.create(BasePage.prototype);

// Declaring constructor

LoginPage.prototype.constructor = LoginPage;

...

Javascript pageobjects

Kinda like calling super(); in Java

33

BasePage.js con’t

BasePage.prototype.waitForLocated = function(locator, timeout) { var MAX_RETRIES = 5; var retry = 0; timeout = timeout || WAIT_TIME_PRESENT; var _this = this; // The actual wait, but we handle the error return _this.driver.wait(Until.elementLocated(locator),timeout).thenCatch(function (err) { if (err.name !== 'StaleElementReferenceError') { throw new Error(err.stack); } // fail after max retry if (retry >= MAX_RETRIES) { Logger.error('Failed maximum retries (' + MAX_RETRIES + '), error : ' + err.stack); throw new Error('Failed after maximum retries (' + MAX_RETRIES + '), error : ' + err.stack); } //retry retry++; Logger.debug('Element not located with error : ' + err.stack + ' retrying... attempt ' + retry); return _this.waitForLocated(locator, timeout, retry); });};

Javascript pageobjectsHandle most of the UI interaction in a common place● Takes care of stale elements exceptions● Retries● WaitForLocated();● WaitForVisible();● WaitForEnabled(); ● ...

34

LoginPage.js con’t

/*** Page load definition* @returns {LoginPage}*/LoginPage.prototype.isLoaded = function() { this.waitForDisplayed(By.css(EMAIL)); this.waitForDisplayed(By.css(PASSWORD)); this.waitForDisplayed(By.css(LOGIN_BUTTON)); return this;};/*** Enter the user information and login* @param username* @param password* @returns {LoginPage}*/LoginPage.prototype.enterUserInfo = function(username, password) { this.waitForEnabled(By.css(EMAIL)); this.driver.findElement(By.css(EMAIL)).sendKeys(username); this.driver.findElement(By.css(PASSWORD)).sendKeys(password); this.waitForEnabled(By.css(LOGIN_BUTTON)); return this;};

Javascript pageobjects

Pageobject methods work seamlessly with mocha promise manager wrapper● Each line is a promise that gets added to the queue● Everything runs top down just like java

a synchronous language

35

var test = require('selenium-webdriver/testing');var assert = require('chai').assert;var LoginPage = require('./../../src/pageobjects/LoginPage');var SideNav = require('./../../src/pageobjects/SideNav');var TopNav = require('./../../src/pageobjects/TopNav');var url = Constants.launch_url;

//Login Tests

test.describe('Login tests', function() { var driver; test.beforeEach(function() { driver = DriverBuilder.build(); });

test.it('Login with an invalid password @smoke', function() { var login = new LoginPage(driver); login.enterUserInfo(Constants.MANAGER_USER, 'foobar'); login.clickLogin(); login.getLoginErrorText().then(function(result){ assert.include(result, 'Your email or password was incorrect. Please try again.'); }); });});

Putting it together

Sample project : https://github.com/mekdev/mocha-selenium-pageobject

Import statements● pageobjects / other libs

TestNG @beforeTest looks familiar ? :)

Look ma no WebElements or Locators

Visual Validation

37

Visual Validation - Applitools

Went with Applitools● Proven track record of prior implementation in Java

http://www.slideshare.net/MekSrunyuStittri/visual-automation-framework-via-screenshot-comparison

● Made Applitools integration a criteria when building the framework

● 3 implementation choices○ WebdriverIO’s WebdriverCSS○ Official selenium-webdriver WebdriverJs (Driver instance)○ Protractor○ Native eyes.images (manage your own uploads and imgs)

38

Trial runs with WebDriverCSSWebdriverCSS

webdrivercss.init(client, {key: 'your key here'});

client.init()

.url(url)

.webdrivercss('Check #1', {

name : 'Login'

}, function(err, res) {

assert.ifError(err)

})

.setValue('input#email', MANAGER_USER)

.setValue('input#password', PASSWORD)

.click('button[type="submit"]')

.webdrivercss('Check #2', {

name : 'Job page'

}, function(err, res) {

assert.ifError(err)

})

Challenges● Still stuck in chain API world● Cannot choose match level

○ Defaults to strict

● One screenshot eqs 1 test not 1 step● Even harder to do pageobjects

○ .webdrivercss() needs to be chained in order to capture the screenshot

39

Applitools test

test.it("test with login page and applitools", function() { var eyes = new Eyes(); var driver= DriverBuilder.build();

eyes.setApiKey("<your key here>"); eyes.setMatchLevel('Content'); eyes.open(driver, "Airware", "Simple Airware main page") .then(function(eyesDriver) { driver = eyesDriver; });

var login = new LoginPage(driver); login.open(url); eyes.checkWindow("Main Page"); login.enterUserInfo(USERNAME, PASSWORD); login.clickLogin(); eyes.checkWindow("Jobs Page"); eyes.close();});

Tests written using the promise manager fits with Applitools and Pageobjects perfectly.● Maintains app context while allowing the insertion of checkpoints

Applitools JS SDK : https://eyes.applitools.com/app/tutorial.html

Visual Checkpoints

40

Sample run

Http REST requests

Looked at REST frameworks

Supertesthttps://github.com/visionmedia/supertest

● Built on mocha● Chain API based● Asserts are built in

Chakramhttp://dareid.github.io/chakram/

● Runs on mocha● Promise based● Asserts are built in● Needs to return chakram.wait()

describe('GET /users', function(){ it('respond with json', function(done){ request(app) .get('/user') .set('Accept', 'application/json') .expect(200) .end(function(err, res){ if (err) return done(err); done(); }); });});

describe("HTTP assertions", function () { it("Should return 200", function () { var response = chakram.get("your.api/get"); expect(response).status(200); expect(response).header("application/json"); expect(response).comprise.of.json({...}); return chakram.wait(); });});

Request libraryRequesthttps://github.com/request/request

● Standard http request library● Callback syntax - request(options, callback)

it("A series of requests", function (done) { var request = require('request'); request({ method: 'POST', uri: '/login', form: { username: 'username', password: 'password' }, }, function (error, response, body) { request({ method: 'GET', ... }, function (error, response, body) { request({ method: 'PUT', ... }, function (error, response, body) { done(); }); } }); });});

Then around the same time..● Share code with UI devs

○ Generators and Coroutines● Node v4.0.0 (Stable)

2015-09-08

○ Official support for ES6!○ Yield statements!

Generator based callsCo-Requesthttps://github.com/denys/co-request

● wraps http request library but yieldable

it("Login", function *() { var request = require('co-request'); var cookieJar = request.jar();

response = yield request({ method: 'GET', url: BASE_URL, jar: cookieJar, followRedirect: false });

response = yield request({ method: 'POST', url: BASE_URL + '/login', jar: cookieJar, form: { username: 'useremail@email.com', password: 'foobar', }, followAllRedirects: true });});

Generator based requests● Became the base for our WebClient● Same principles as a page object

but for REST APIs● Yield blocks until execution is done● The Magic number 7

45

JSON and Javascript

The hidden power of working in javascript○ JSON stands for JavaScript Object Notation

JSON is a subset of the object literal notation of JavaScript. Since JSON is a subset of JavaScript, it can be used in the language with no muss or fuss. dto jackson

Actual response{ "type": "forbidden", "message": "You do not have permissions in this project"}

Codevar webClient = new WebClient();yield webClient.login(VIEWER_USER, VIEWER_PASSWORD);

// Try to view usersvar projectUsers = yield webClient.getProjectUsers(qeProject.id);assert.strictEqual(projectUsers.type, TYPE_FORBIDDEN, 'Return type should be forbidden');assert.strictEqual(projectUsers.message, 'You do not have permissions in this project');

46

Fitting it with selenium framework● Adapting yield calls to work with the promise manager

○ Selenium Control Flows https://code.google.com/p/selenium/wiki/WebDriverJs#Framing

○ Allows execution order framing and supports generators from the manager queue● Functional programing - high order functions are your friends

test.it('Verify data from both frontend and backend', function() { var webClient = new WebClient(); var projectFromBackend; // API Portion of the test var flow = webdriver.promise.controlFlow(); flow.execute(function *(){ yield webClient.login(Constants.FORSETI001_EMAIL, Constants.FORSETI_PASSWORD); var projects = yield webClient.getProjects(); projectFromBackend = projectutil.getProjectByName(projects, Constants.QE_PROJECT); }); // UI Portion of the test var login = new LoginPage(driver); login.enterUserInfo(Constants.FORSETI001_EMAIL, Constants.FORSETI_PASSWORD); var topNav = new TopNav(driver); topNav.getProjects().then(function (projects){ Logger.debug('Projects from backend:', projectsFromBackend); Logger.debug('Projects from frontend:', projects); assert.equal(projectsFromBackend.size, projects.size);});

Utility module that heavily uses underscore

47

What the final stack looks like

Data

Backend : node.js

Browser

Input / update data

Get data

Frontend : javascript

Microservice 1

Microservice 2 .. n

Rest APIs

Input / update data

Get data

UI Framework : node.js● selenium-webdriver● mocha + wrapper● Applitools● co-wrap for webclient● chai (asserts)

Rest API Framework : node.js● co-requests● mocha● co-mocha● chai (asserts)● json, jayschema

WebClient

Pageobjects

Webclient adaptor

48

Introduction to deployments

Cloud Deployment Culture● Weekly deploys to production● Gated deploys to preprod● Automatic deploys to staging and tests

staging preprodprod

Clouds envs

After selenium testsContinuous deployment with

Kubernetes and Docker

Volkan Gurel - Engineering Manager

50

Deployment Dashboard (Vili)

Problem:● Manage and deploy:

○ Many microservices○ In many environments○ With many versions

● Access control for deployments● QA gating for production deployments

Solution: Vili

51

Vili Overview

Kubernetes- Controllers for apps- Pods for jobs- Rolling deploys

Docker Repo- Many apps- Many versions

Environments- Different variables- Need source control

Notifications- Slack

Authentication- Okta- Extensible

Approvals- Only QA can approve- Required for prod deploy

VILI

52

Live Demo

53

Follow the Airware github repo to be notified:

https://github.com/airware

(About to be) Open Sourced

The Team● Bj Gopinath - Guidance and support● Lucas Doyle, Nick Italiano - co and Node.js generators, locator sharing strategy with frontend● Phil Kates - Countless nights/weekends on infrastructure work● Eric Johnson - Guidance and support. On coming from Java to Javascript :

“Yeah, probably some unlearning going on. JS is crazy, but I’ve rarely had more fun”

Meetup Folks● Marcel Erz, Yahoo - Feedback on implementations of Webdriver-sync● Mary Ann May-Pumphrey - Nightwatch feedback

Special Thanks

SaucelabsInitial feedback on selenium node.js

● Neil Manvar● Kristian Meier● Adam Pilger● Christian Bromann - WebdriverIO

ApplitoolsTrial POC with Applitools Javascript bindings and performance

● Moshe Milman● Matan Carmi● Adam Carmi● Ryan Peterson

Q & A

Questions

Thank you