1

Subline

HEADLINE

• whoami?

• How we use Selenium

• Selenium as a Service (Saucelabs / BrowserStack / …)

• Building your own Selenium GRID on AWS EC2

Selenium Meetup

What are we gonna talk about?

WHOAMI

Harry Urban

2

Harald.Urban@WeltN24.de

QA Lead @HarrysIdeAS

3

ABOUT MY COMPANY

WeltN24 GmbH - Axel Springer SE

APPSMOBILE

PRINT ONLINE

SOCIALMEDIA TV

USE CASES

Selenium GRID

4

www.welt.de m.welt.de

Des

ktop

Pag

e

Mob

ile R

espo

nsiv

e

SingleSignOn

Pay

men

t Wor

kflo

ws

CUCUMBER

BDD with Gerkhin

5

6

CUCUMBER

Step Implementation

CUCUMBER

Page Objects (e.g. with Fluentlenium)

7

FROM LOCALHOST TO CLOUD

First steps - Localhost

8

• „chief cook and bottle washer“ server (eierlegende Wollmilchsau Server) • Jenkins for Test Execution • Tomcat (webapp) for collecting and presenting test results • Firefox instances as many as Jenkins Executors • Selenium Standalone

• -> Left-over Firefox Instances • -> Full Hard Drives • -> Other Services not being able to run • -> Resource Conflicts • Either everything works or nothing

9

HEADLINE

Subline

FROM LOCALHOST TO CLOUD

First steps - Saucelabs

• Account at Saucelabs

• - Firewall Exclusions for Tests on Staging Systems • wide ip range • with changes every couple of months

• - Sometimes long response times -> failing tests

• + Throw-away Servers == no Re-Usage of VMs • + Video Recordings of Sessions • + No Stuck Browsers

• >1000$ / month

HEADLINE

Subline

10

FROM LOCALHOST TO CLOUD

First steps - BrowserStack

• Evaluation of BrowserStack

• - Hosting on AWS • -> Wide AWS IP Range had to be whitelisted in Firewall for Stages • -> Security / Privacy Issue

• - Re-Usage of VMs • - A lot of False Positives • - NO Video Recordings of Sessions • - In the end more expensive than Saucelabs

11

FROM LOCALHOST TO CLOUD

Make or buy

-> Best satisfaction with Saucelabs

HEADLINE

Subline

12

FROM LOCALHOST TO CLOUD

Chef Cookbooks for Infrastructure

vagrant-aws + chef + selenium cookbook = GRID @ AWS

HEADLINE

13

AWS Architecture FROM LOCALHOST TO CLOUD

14

HEADLINE

Subline

HEADLINE

Subline

FROM LOCALHOST TO CLOUD

Setting up AWS Infrastructure

HEADLINE

EC2 Instance Overview

15

FROM LOCALHOST TO CLOUD

16

HEADLINE

SublineAuto Scaling Group for Selenium Nodes FROM LOCALHOST TO CLOUD

HEADLINE

Subline

17

Auto Scaling Group for Selenium Nodes FROM LOCALHOST TO CLOUD

HEADLINE

Subline

18

CloudWatch - Custom Metrics FROM LOCALHOST TO CLOUD

Upscaling • Pushing Pending Requests with cron job from HUB -> increases ‚Desired‘ Size at AutoScalingGroup

Downscaling • AWS EC2 pay at least 1h • Nodes check from 50 to 60min if they are idle

if idle -> a) detach from Grid b) self destroy c) decrease Desired Size at ASGroup

• if they are not idle in that timespan they got to live for another 50-60min

HEADLINE

Selenium Grid Console

19

FROM LOCALHOST TO CLOUD

20

LIMITATIONS OF AWS

Auto Scaling + AMIs

• Analytics showed that >50% of our users are on Windows • -> right now 100% Windows Nodes • unix nodes planed for DesiredCapability Platform=ANY • Unix nodes are even easier, because they can be completely provisioned

• Challenges of Windows Nodes -> No easy provisioning with Chef

• so far manually configured Windows AMI, that evolved over time • (not yet time invested in provisioning naked Windows,

e.g. Windows Templates for Packer)

• AutoScalingGroups work with AMIs • ASGroups are based on LaunchConfigurations • in LC’s you have to configure an AMI to launch • -> no downside of Windows Nodes for our setup

21

LEARNINGS

Limitations of Webdriver and Selenium

• Keep requests to the DOM at a minimum • e.g.: Don’t iterate through an XML file with WD requests

• No matter what, always stop the Browser at the end of your test

• Clean temporary files often

• Have a TimeOut in your code for interrupting page loads being stuck

FUTURE CHALLENGES

Testing Microservices

22

• WELTN24 is going to make a big switch from a monolith to micro services:

• Advantage: Teams can deploy independently and more often • Requirement: Fully automated deployment

• Limitations of GUI-Tests (End2End Tests): • The higher you get in the testing pyramid the more flaky your tests get • -> Flaky tests don’t suit a fully automated deployment pipeline!?

• Challenge: Can GUI Tests tests be integrated in automated deployments?

• Proposal: Stable and smart Test Journeys through GUI for business critical paths • combined with Contract Based Testing for ensuring functional behavior?

REFERENCES

LINKS AND TOOLS

23

• https://github.com/dhoer/chef-selenium/ • https://github.com/mitchellh/vagrant-aws • https://github.com/agileorbit-cookbooks/java • https://github.com/FluentLenium/FluentLenium/ • https://github.com/cucumber/cucumber-jvm/ • http://aws.amazon.com/de/ec2/

Worth a look at: https://github.com/groupon/Selenium-Grid-Extras

24

THANK YOU

STAY IN TOUCH

• Slides will be on: • http://de.slideshare.net/urbanharry

• STAY IN TOUCH • @HarrysIdeAS

• „Harry Urban“ -> • LinkedIn • XING

Questions?