iOS Parallel Automation: run faster than fast — Viktar Karanevich — SeleniumCamp 2017

iOS Parallel Automation:Run faster than fast!

Viktar Karanevich

Content:

I. History of iOS Automation Frameworks (5 min)

II. Path to Current Approach at Badoo (10 min)

III. Summary (5 min) IV. Movie: Test Run (3 min)

self.introduce =>

{ name: "Viktar Karanevich", title: ["SSTAE", "SSDET", "SQAAE"], languages: ["Java", "Ruby"], experience: "iOS Automation 7+ years" }

The original, largest and leading dating network:

330M+ registered users

800+ automated tests on iOS platform

50M+ downloads on iOS

Steve Jobs

Steve Wozniak

I. History

Jun 2010UI Automation

Apple


Sep 2014UI Automationxcrun simctl

Apple

"xcrun simctl" is introduced in xCode 6

It is command line utility to control the Simulator(create, delete, erase, install, uninstall, ...).


Sep 2015UI Automationxcrun simctlUI Testing

Apple

UI Testing is introduced as replacementfor UIAutomation in xCode 7



Sep 2016UI Testing

xcrun simctl

Apple



UI Automation removed in xCode 8


Sep 2014xcrun simctl

Sep 2016UI Testing

Sep 2016UI Testing

Apple

Calabash

Nov 2009UI Spec

Calabash

Nov 2009UI Spec

Feb 2011Frank

Calabash

Nov 2009UI Spec

Feb 2011Frank

Jan 2012Calabash

Step DefinitionsRuby Client Library

Features

Developer host

Calabash server

Simulator

Your app

UIQuery

Touch Actions

Calabash

Apr 2013Calabash +

UIAutomationLink: https://github.com/calabash/run_loop


Features

Developer host

Calabash server

Simulator

Your app

UIQueryRun loop:

UIAutomation

https://github.com/calabash/run_loop

Calabash

Sep 2016Calabash +

DeviceAgentLink: https://github.com/calabash/run_loop/tree/develop/lib/run_loop/device_agent

Developer host Simulator


XCTest Runner

Features

Calabash server

Your app

UIQuery

Device Agent

https://github.com/calabash/run_loop/tree/develop/lib/run_loop/device_agent

Calabash

Nov 2009UI Spec

Feb 2011Frank

Jan 2012Calabash

iOS Automation Frameworks

Apr 2013Calabash +

UIAutomation

Sep 2016Calabash +

DeviceAgent


Sep 2016UI Testing

xcrun simctl



Apple

3 years 1 year

Calabash

Nov 2009UI Spec

Feb 2011Frank

Jan 2012Calabash

iOS Automation Frameworks

Apr 2013Calabash +

UIAutomation

Sep 2016Calabash +

DeviceAgent


Sep 2016UI Testing

xcrun simctl



Apple

3 years 1 year

II. iOS Automation at Badoo

Approach

~80flaky tests

80data

fail tests

II. Before Parallel RunTest Suite RunNightly Cucumber ReportNightmare!

15htotal time

1simulator

80data

~500tests

II. Before Parallel RunTest Suite RunNightly Cucumber ReportNightmare!

80data

fail tests

1simulator

Key Issue: How to speed up Test Run?

15htotal time

~500tests

1simulator

Let's Parallel!

Let's Parallel!

Blocker Issue: - Calabash does not support Parallel run on 1 desktop

Let's Parallel with multi-desktops!

https://techblog.badoo.com

Tim's talk at Badoo: "Parallel Calabash Testing on iOS"

https://techblog.badoo.com/blog/2015/10/22/parallel-calalabash-testing-on-iOS

II. Test Result: multi-desktops

~4 hours regression suite run time

5 Desktops

II. Test Result: multi-desktops

Mac PRO (24 cores)

~2 hours regression suite run time

10 Desktops

X Desktops: host limit reached

II. Clustering Hosts


- Parallel run on multi (virtual) hosts

- Mac Pro and hosts: test suite - ~1h 30 min


Issues:- multi-desktops start up is Flaky- simulators can get stuck


Is it possible to automate multi-simulators on one desktop?

III. Facebook: Web Driver Agent


Selenium DriverWDA Runner Your app

FB UIA WDA

Key Change: Reimplemented UIAutomation dependencyLink: https://github.com/facebook/WebDriverAgent

https://github.com/facebook/WebDriverAgent

Profit:- 7 simulators on single desktop- More stable! more efficient!

III. Full Migration to FB UIA WDA

Issues:- all tests have calabash code


Issues:- all tests have calabash code- custom app backdoors


Issues:- all tests have calabash code- custom app backdoors- workarounds and other


Full Migration Costs: ~x hours

Is it possible to do Fast Migration?

III. Calabash + FacebookFast migration: Facebash!


Features

Developer host

Calabash server

Simulator

Your app

UIQueryRun loop:

UIAutomation

III. Calabash + FacebookFast migration: Facebash!


Features

Developer host

Calabash server

Simulator

Your app

UIQueryRun loop:

UIAutomation



WDA Runner

Features

Calabash server

Your app

UIQuery

FB UIA WDA

IV. iOS 10 Update:from UIAutomation WDA to XCTest WDA

Link: https://github.com/facebook/WebDriverAgent


Selenium DriverYour app

FB UIA WDA

WDA Runner


IV. iOS 10 Update:from UIAutomation WDA to XCTest WDA

Link: https://github.com/facebook/WebDriverAgent


Selenium DriverYour app

FB UIA WDA


Your app

FB XCTest WDA

WDA Runner

Selenium Driver

FB simctl


IV. iOS Hybrid 2.0iOS 10 update



WDA Runner

Features

Calabash server

Your app

UIQuery

FB UIA WDA

IV. iOS Hybrid 2.0iOS 10 update



WDA Runner

Features

Calabash server

Your app

UIQuery

FB UIA WDA



FB simctl

Features

Calabash server

Your app

UIQuery

FB XCTest WDA

V. Summary: multi-desktops vs multi-simulators

Test Suite: 500 testsNightly Cucumber ReportHappy morning!

80data

Mac PRO

Test Suite Run Time

0 min

225 min

450 min

675 min

900 min

1 sim 10 sims 16 sims

75 min120 min

900 min

MultiDesktops

MultiSimulators



FB simctl

Features

Calabash server

Your app

UIQuery

FB XCTest WDA

"Face"book + Cala"bash" = Facebash

V. Summary: current approach

- easy to set up Parallel Run on 1 Desktop- keep using of backdoors- full access to internal AUT state- cross iOS app automation- iOS simulator control with FBSimCtl- FB WDA supports Selenium protocol- FB WDA independent of Calabash/Appium

Facebook

CalabashCalabashFacebookFacebookFacebookFacebook

V. Summary: PROs

- support framework parts: calabash server, FB WDA, fbsimctl, ...- "poor" documentation- using of mixed locator strategies

V. Summary: PROs and CONs

- Parallel Run in Headless Mode- Simulator Start/Shutdown/Reset optimization- Cold/Hot start iOS app with relogin- record video per the each test in Parallel Run

V. TODO

Thank You!

[email protected]

Questions?

mailto:[email protected]?subject=

Technology

iOS Parallel Automation: run faster than fast — Viktar Karanevich — SeleniumCamp 2017