Distributed Testing Environment

Distributed Testing EnvironmentŁukasz Morawski

The Stories

• John The Tester is a manual tester,

• He is ordered to do an automation,

• So he does,

• He develop some tests, he runs iton his box

The Stories

• The more tests he develops, the more time tests lasts,• At first, it takes couple minutes,• Later, it takes dozen,• It is a lot of time!

He cannot afford to stare at running tests

...and the crucial is...

...and the crucial is

The problem

• QA team needs to provide test feedback as soon as it’s possible,

• Automated tests need to be “Fast and Furious”.

Distributed testing environment on virtual machines.

...but at the beginning there was Eternity...

• server hosting Hudson,• test execution testbed for Selenium IDE tests,

• but one executor means one test at a time.

VM and Hyper-V• QA team uses Hyper-V virtualization software,

• It let us to run ~14 VM,• CPU is not the problem,• Memory is the problem,• 80% free memory rule – swapping leads to serious

performance lost,• We have GEB and BES servers available for us:• BES – manual tests environments for testers and developers

(different OS, different browsers, mobiles),

VM and Hyper-V• GEB – automated tests environments.• Slaves for us and our clients,• Jenkins,• Applications – QA Lab uses this machine to provide tools and

services for our test framework. So far we have W3C Markup Validation service – our Validation Tool uses it.

VM and Hyper-V

All machines are under Nagios supervision, It let us to control if the environment is up and ready to go, Notifies by e-mail if something is wrong.

VM Future

• More tools available for our testing framework (W3C tools),• Different browsers/browsers version,• Maybe different virtualization system – bare metal hyper-

visors seems to look promising,• Cloud – as a main resource, as a backup resource.

Centralized Automated test driving with Jenkins CI

• QA Team use Jenkins CI tool to drive tests,• We have Jenkins deployed in a Tomcat container,• We use SLAVE AGENTS as a communication channel,• It’s not greatest solution because CI tools are rather build

tools not testing tools.

How we use Jenkins...

• We use Views to manage projects


• We use “distributed builds” feature to manage parallel testing (speed up) with several slaves,


• We use labels to nickname a slaves (ff36, ff4, winxp).

• We use “Restrict where this project can be run” to configure a test job run.


• For performance testing we use JMeter and JMeter plugin for Jenkins.

How we use Jenkins...• We use “Configuration Matrix” for some jobs.

How we use Jenkins...• We use “Discard old builds” to save disk space.


• We use “Trigger build remotely” to run build from outside:

• ...and tools like curl/wget to trigger it:

curl --user user:password JENKINS/view/project/job/build?token=token

• We use Jenkins Remote Access API,


00:00 to 08:00 - Automated/scheduled tests, nightly test,08:00 to 21:00 - work day, manual/scheduled test,21:00 to 00:00 - daily/weekly maintenance time,

• We try follow some time line guidelines:

Maintenance problem

• We suffer with maintenance problem while managing so many VM – we use STAF for performing actions on all machines.

Maintenance problem

• These machines lives and to prevent deterioration we enforce some maintenance tasks.

Maintenance problem

• Daily tasks (21:00):• Restart,• Set screen resolution,• Resources.

Weekly tasks (21:00 Sunday): Cleaner, Defragmentation, Anti-Virus Scan, Windows updates.

Jenkins future

• Browsers auto updating,• Dynamic slave management,• Custom framework – leave Jenkins bottlenecks behind and

build your own,

The answer

• Distributed testing environment on virtual machines,• Centralized automated test driving with Jenkins CI.

Benefits of presented solution

• Easy way to extend test environment (just clone VM and voila),

• ...which let us to speed up tests by splitting and paralleling,• Easy way to restore corrupted system (revert snapshot, copy

VM disk file),

• HQ for managing tests with different tools (Selenium, Webdriver, Jmeter, Wapiti, Test Complete, AutoIt...),

• Built-in features and plugins (e.g.: SVN client),

Technology

Distributed Testing Environment