User Interface Test Automation Tools for iOS Applications

User Interface Test Automation Tools for iOS

Applications

Alejandro Echeverri Romero

Diciembre, 2019Version: 1.0

Bogotá, Colombia

Departamento de Ingeniería de Sistemas y ComputaciónFacultad de Ingeniería

The Software Design Lab

Proyecto de grado

User Interface Test Automation Tools for iOSApplications


Asesor Ph.D. Mario Linares Vásquez

Diciembre, 2019


User Interface Test Automation Tools for iOS Applications

Proyecto de grado, Diciembre, 2019

Asesor: Ph.D. Mario Linares Vásquez

Universidad de los Andes

The Software Design Lab

Facultad de Ingeniería

Departamento de Ingeniería de Sistemas y Computación

Cra 1 # 18A - 12

111711 and Bogotá, Colombia

Abstract

Over the past years, mobile applications have experienced a tremendous complexityincrement, they may now represent the only channel to a service or they intend tooffer at least the same features web and desktop systems currently provide. Thismeans, mobile development teams have to step up their quality assurance processthrough the introduction of test automation into their projects in order to maintainthese highly complex and user demanding apps. This project pretends to comparethe most relevant attributes and characteristics of the currently existing tools foruser interface mobile test automation over the iOS environment, with the purpose ofeasing the selection of an appropriate automation tool based on the projects specificneeds. On the other hand, during the development of this project, a case study willbe designed whereby the selected automation tools will be used with the intent ofrecognizing and documenting their strengths and weaknesses.

v

Contents

1 Introduction 1

1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.2 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.2.1 Specific Objective: Tool Research . . . . . . . . . . . . . . . . 2

1.2.2 Specific Objective: Case Study . . . . . . . . . . . . . . . . . . 2

1.2.3 Specific Objective: Documentation . . . . . . . . . . . . . . . 2

1.3 Expected Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Related work 5

2.1 State of the Art . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.2 Test Automation Tools . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.2.1 XCUITest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.2.2 Appium . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.2.3 Calabash . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.2.4 EarlGrey (2.0) . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.2.5 Ranorex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.2.6 KIF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.2.7 Detox . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.2.8 TestComplete . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.2.9 MonkeyTalk . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.2.10 Quantum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.3 Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.4 Comparison Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.5 Categories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.5.1 GUI Components . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.5.2 Gestures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.5.3 Framework Testing Capabilities . . . . . . . . . . . . . . . . . 11

2.5.4 Automation Tool Characteristics and Features . . . . . . . . . 12

3 Case Study 13

3.1 Firefox iOS App . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

3.2 Test Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

3.2.1 XCTestUI Test Cases . . . . . . . . . . . . . . . . . . . . . . . 14

3.2.2 KIF and EarlGrey Test Cases . . . . . . . . . . . . . . . . . . . 18

vii

3.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193.3.1 XCTestUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203.3.2 KIF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213.3.3 EarlGrey . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

4 Conclusion 234.1 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234.2 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

Bibliography 25

viii

1Introduction

1.1 Introduction

During the development of a system, performing tests is needed in order to validatethat it behaves as intended. Therefore, through testing, it is possible to verify many ofthe functional and nonfunctional characteristics of your system. However, hundredsof testing types and techniques exist for this purpose.

Regression testing is a common type of testing that often faces complications. Itinvolves performing a series of previously executed tests in order to validate that achange in the system has not affected any of its dependencies. In situations where thesystem is relatively large or it experiences frequent changes, performing regressiontesting can become an issue. This is due to the fact that the bigger the project or thehigher the frequency of changes in it, more testing will be required. Which leadsto an increase in costs and delays adapting the changes made [8]. Here is wherethe concept of test automation becomes quite beneficial, especially in continuousintegration projects [5].

It is very common to see the previous situation during mobile application develop-ment. This is due to the fast-paced and increasing size and complexity of projectsin the current mobile application market. Not only do users expect frequent im-provements, but they expect high quality apps. This can considerably influence thenumber of installs, user rating and user retention. Therefore, it is crucial to assessthe quality of your app with an efficient testing process [9].

A proper and efficient testing process on a mobile application development projectwill most likely require the automation of several tests [18]. Before commencingwith the test scripting, a series of important decisions regarding the design of the testautomation architecture must be made. These commonly involve choosing a testingframework and any other tools to be used. A common issue here is that this decisionmight not be as straight forward as one would desire, since it actually depends ona series of factors related to the project and the tool characteristics. Taking thewrong decision at this level may result extremely expensive and time consumingsince it could unnecessarily increase the difficulty level, the time needed at the test

1

automation implementation or, even worse, it could require an architectural changemid-project.

For the purposes of this project, an approach will be made into the user interfacetest automation tools available for the iOS platform. It is specially attractive toconduct a research on this area due to the fact that this platform is way more limitedcompared to the Android platform in terms of available tools and operating systemrestrictions.

1.2 Objectives

Studying and analyzing a series of existing tools for the development of automatedinterface tests in mobile applications that run over the iOS operating system. Thiswould allow identifying weaknesses, strengths, characteristics and limitations ofexisting tools in order to take the most convenient choice when designing thearchitecture of a test automation project.

1.2.1 Specific Objective: Tool Research

• Conducting a research on the currently available tools that allow the automa-tion of user interface tests in iOS applications. In other words, defining whattools will be taken into account in the development of this project and whichof their attributes are more relevant to compare and analyze.

1.2.2 Specific Objective: Case Study

• Designing a case study that will allow the evaluation of the attributes definedin the previous phase of research, as well as to include any other attribute thatis considered relevant during the development of this phase.

1.2.3 Specific Objective: Documentation

• Publishing a report with the results of the case study in a document that de-scribes the differences between the automation tools. As a result, this documentwould expose their weaknesses, strengths, characteristics and limitations.

2 Chapter 1 Introduction

1.3 Expected Results

Upon completion of this project, a detailed documentation on a series of toolscurrently available for user interface test automation in iOS applications will beavailable. This would have a comparison of the most relevant characteristics for eachof the selected tools. On the other hand, any code developed for the case study willbe published as open source in order to collaborate with the community. These twoshould ease the decision making when designing the architecture of an automationproject.

1.3 Expected Results 3

2Related work

2.1 State of the Art

For mobile systems there are only two main platforms: Android and iOS. SinceAndroid is an open source project, a vast number of smartphone manufacturingcompanies provide this base operating system and some have even implementedtheir own wrapper around it. As a result, the device suite for Android has surpassed7000 devices [10]. On the other hand, the iOS platform is much more restrictiveand only Apple is allowed to manufacture devices with iOS, which results in thetesting advantage of having a smaller device suite [4]. However, due to the secrecyand restrictions of the iOS system (as opposed to Android), there are fewer toolsavailable that may perform automated tests, making test automation fairly moreuncommon on this platform. Consequently, quality control might be performedthrough manual testing instead of automated testing, even if it is not the properapproach.

There are currently a few tools that allow the development of automated userinterface tests for iOS. The feasibility of developing tests through one or the othervaries depending on the compatibility of the tool with the characteristics of theproject. This refers to factors such as: project budget, programming languages withwhich the team to perform the automation is familiar with, whether the applicationis hybrid or native, testing load and speed requirements, among many others.

The two types of tools used to create user interface automated tests are: automationAPIs/frameworks and record and replay tools [13]. Both types are within the scopeof this project. It is also worth noting that there are several other test supporttools like: bug and error reporting/monitoring tools, device streaming tools andautomated test input generation tools. It is important to understand at a high levelhow these two tool types work and how they are used to create automated tests.

They both attempt to provide an interface for the user to create automated tests.Which is why, in some cases, test automation frameworks provide both types. Eventhough the goal is the same, the way the actual automated test is created is wherethey differ. On the one hand, automation APIs/frameworks require a manuallywritten test script which contains all of the code that makes up the automated test.

5

On the other hand, record and replay tools only require for the test to be manuallyexecuted and the tool itself will record all the interactions that where performed,automatically generating the automated test.

During the execution of an automated test, the tool must be capable of interactingwith the smartphone GUI elements as a real user would, replicating the test casescenarios. This includes performing GUI interactions such as: swiping, tapping,dragging, pinching, among others.

2.2 Test Automation Tools

There are several different tools that can be used to develop automated tests, rangingfrom paid license all-in-one tools like Ranorex [17] to open source client-server toolslike Appium [1].

Getting to know all of the available tools, how they work and how they are usedcan become a very time-consuming task. Besides, when performing research abouttest automation tools for iOS, you generally find a small list of tools with a briefdescription. It turns out, the actual information you can find on automation toolsfor iOS is very limited and as a result you can easily have a hard time trying tochoose the appropriate tool for your project. As a consequence, this project pursuesto reduce the conceptual gap present when trying to select the most appropriateuser interface test automation tool that supports iOS applications.

Here, a list and a brief description with the most relevant and currently availabletools that perform user interface test automation over the iOS platform is shown.This will provide a high level view on what is available and may help you discardsome options right away.

2.2.1 XCUITest

Xcode comes with the testing framework XCTest. This testing framework will notonly allow you to code your unit tests, but you may also record and write UI tests.This testing tool has seen mayor upgrades since it was released a few years ago andhas now become a really solid option. [3]

6 Chapter 2 Related work

2.2.2 Appium

Appium has now become one of the most popular mobile test automation tools outthere. This open source project is characterized by a client server architecture whichsupports a wide range of programming languages and testing frameworks to workon. [1]

2.2.3 Calabash

Although Xamarin (Microsoft) discontinued their development for this test automa-tion framework up until iOS 11 and Android 8, it has still been maintained by itscommunity. It works with Cucumber in order to follow the BDD process plus it isa free and open source project. This means it can become fairly simple to writeautomated tests without much coding experience. [6]

2.2.4 EarlGrey (2.0)

This open source project by Google provides a testing framework works along withthe existing XCUITest in order to provide certain advantages such as synchronizationand white box testing. [11]

2.2.5 Ranorex

This paid all-in-one solution seems to be one of the most popular of its kind. Itallows the recording and writing of automated tests for many platforms and it issimple to use, for which its users do not need to be skilled with programming to useit. [17]

2.2.6 KIF

KIF, standing for “Keep It Functional” is a test automation framework that works atthe XCTest level and runs with your unit tests. Tests are written in Objective-C andrely on the accessibility attributes. [12]

2.2.7 Detox

Detox is a grey-box end to end testing framework for mobile devices (iOS andAndroid) which provides fast tests with improved reliability regarding flakiness. It

2.2 Test Automation Tools 7

is designed to support both React Native and pure native ones and it may performtesting over a real device or a simulator. [7]

2.2.8 TestComplete

This paid all-in-one solution provides everything you may need in order to build andrun your tests ranging from the testing framework to the physical/virtual devices.You can either write your own tests on any of the 7 available languages or use thebuilt-in record and replay tool. [19]

2.2.9 MonkeyTalk

MonkeyTalk is a record and replay open source mobile app automation tool. It isdesigned to be simple to setup and use. It provides support for both android andiOS. [14]

2.2.10 Quantum

The Quantum framework allows you to build BDD automated tests using Cucumber.It supports both iOS and Android. Tests are written in Java. [16]

2.3 Classification

Conveniently, it is possible to classify these tools in groups from a high-level stand-point:

1. First-party: Here we have the well known XCTestUI which is easy to setup,reliable to use and requires a moderate amount of experience to work with. Themain downside is it does not provide any compatibility with other mobile operatingsystems like Android.

2. XCTest based: These act as libraries that provide some extra features besides whatXCTestUI has to offer. Tools in this group include EarlGrey and KIF.

3. Third Party: Third party tools like Appium, will require the most experience towork on but provide great flexibility and interoperability.


4. All-in-one paid: These are usually the easiest to use but generally require a paidlicense. Tools in this group include TestComplete and Ranorex.

2.4 Comparison Table

In the following table, you will be able to visualize a comparison between the maincharacteristics of the previously mentioned tools. Further on, the case study willprovide even more specific information about a few of these tools.

2.4 Comparison Table 9

2.5 Categories

In order to compare these test automation tools, it is important to understand thedifferent elements, interactions and functionality a mobile application utilizes. Thesewill be broken down in three categories: GUI components, Gestures and TestingCapabilities. The first makes reference to the all the elements that make up thegraphical user interface, which can be divided into three categories: Bars, Views andControls. Gestures are the main way users provide input to their mobile devices.Finally, testing capabilities make reference to the various APIs the test automationtool could provide in order to manipulate GUI elements, operating system andhardware configurations.

The following is a list of GUI components and gestures [2], as well as some of thetesting capabilities that one may need while automating tests.

2.5.1 GUI Components

Bars

• Navigation Bar

• Search Bars

• Status Bars

• Tab Bars

• Toolbars

Views

• Action Sheets

• Activity Views

• Alerts

• Collections

• Image Views

• Pages

• Popovers

• Scroll Views

• Split Views

• Tables


• Text Views

• Web Views

Controls

• Buttons

• Context Menus

• Edit Menus

• Labels

• Page Controls

• Pickers

• Progress Indicators

• Refresh Content Controls

• Segmented Controls

• Sliders

• Steppers

• Switches

• Text Fields

2.5.2 Gestures• 3D Touch

• Tap

• Drag

• Flick

• Swipe

• Double tap

• Pinch

• Touch and Hold

• Shake

• Rotate

2.5.3 Framework Testing Capabilities• Element handling: Element searching and Element attributes

2.5 Categories 11

• Authentication: Touch ID and Face ID

• Toggle hardware configurations: Wi-Fi, Data, Bluetooth, . . .

• Access system configurations: Language, Region, Time, Text Size, . . .

• Change device orientation

• GPS mocking

• Camera mocking

• Application state (set app to background, reopen app)

• Multitasking

• Notification handling

• Access app cache data

• File handling

2.5.4 Automation Tool Characteristics and Features

Other important aspects to take into account when evaluating the suitability of a testautomation tool for a project are the characteristics and features provided by theframework. Some of these may become invaluable depending on the project.

Examples for such are:

• Performance: Test execution time, memory use, . . .

• Maintainability

• Test parallelization

• Emulated and Real Device compatibility

• Ease of development (skill required)

• Android (hybrid) project compatibility

• Programming Languages Supported

• Developer Support

• Price


3Case Study

For this project, a case study focused on taking a closer look into the popular XCUITesttest automation framework and other XCTest based user interface test automationframeworks will be performed. In order to achieve this, an iOS application, a set oftest cases and automated tests are needed.

3.1 Firefox iOS App

After exploring several open source applications, the Firefox project for iOS seemedto be the best fit, since it provides a big enough and stable application as well ashundreds of working automated UI tests. These automated tests make use of theXCUITest framework for black box testing along with KIF and EarlGrey for greybox testing. On the other hand, the project is currently up to date regarding thesoftware versions used and it is constantly been worked on, which greatly reducescompatibility issues. [15]

3.2 Test Cases

Considering the objective of this case study is to compare the different test automa-tion tools, the following set of test cases aims to cover as many different situationsone could encounter while testing, and not providing a coverage for the applicationitself. This means, they will try to cover as many of the GUI components, gesturesand testing capabilities as possible within the existing automated tests in the FirefoxApp. Automated tests available at fork: https://github.com/aecheverrir/firefox-ios

13

3.2.1 XCTestUI Test Cases

14 Chapter 3 Case Study

3.2 Test Cases 15


3.2 Test Cases 17

3.2.2 KIF and EarlGrey Test Cases


3.3 Results

The following tables provide, for each tool, a view which matches a specific GUIcomponent, gesture or testing capability with a test case that involves it. This isparticularly useful if you desire to explore a real-world example of an automatedUI test for some of the different GUI components, gestures and framework testingcapabilities or to check out the coverage of the tool.

3.3 Results 19

3.3.1 XCTestUI


3.3.2 KIF

3.3 Results 21

3.3.3 EarlGrey

After some extra research, the coverage of several GUI components, gestures andframework testing capabilities could be exhibited. However, these do not have arelated test case as there is no automated test in the Firefox App to associate themwith. It is also important to take into account the fact that not all GUI components,gestures and framework testing capabilities were evaluated.


4Conclusion

During the development of this project, a series of discoveries were made about thecurrently available mobile test automation tools that support iOS. First, it came torealization that there are a lot of variables to consider when choosing a test automa-tion framework. However, one can start by selecting the right type of test automationframework to work with given the characteristics of the project and the expertiseof the team that will perform the test automation. Second, actually understandingthe different elements, interactions and functionality a mobile application utilizesis key when working on test automation. Therefore, a detailed research on thepieces that are involved in a test automation tool was made. This produced a listof GUI components, gestures, testing capabilities, framework characteristics andfeatures. Finally, the developed case study allowed the examination of the previouslymentioned lists over the Firefox App providing real world examples of automatedtests for some of the evaluated tools. Reading the code and watching the test executein real time considerably contributes to the understanding of the test automationenvironment via a practical approach.

4.1 Limitations

• Limited options for iOS open source apps.

• Time scope of the project was not enough to actually develop the case studywith third party and all-in-one test automation tools.

• All-in-one tools can be very expensive. For example, TestComplete will easilycost more than 6,000 USD a year with no add-ons.

4.2 Future work

This project has several opportunities for improvement:

• Performing the same case study content with more test automation tools.

23

• Making use of more iOS applications in order to reference more real world testcases and automated test scripts.

• Expanding the coverage tables, including more GUI components, gestures andtesting capabilities for the used tools.

24 Chapter 4 Conclusion

Bibliography

[1] Appium. Introduction to Appium (cit. on pp. 6, 7).

[2] Apple. Human Interface Guidelines (cit. on p. 10).

[3] Apple. IUser Interface Tests (cit. on p. 6).

[4] Apple. Supported iPhone models.

[5] International Software Testing Qualifications Board. Certified Tester Founda-tion Level Syllabus. 2018 (cit. on p. 1).

[6] Calabash. Calabash iOS (cit. on p. 7).

[7] Detox. Detox (cit. on p. 8).

[8] Google. Automate user interface tests.

[9] Google. Core app quality.

[10] Google. Devices.

[11] Google. EarlGrey (cit. on p. 7).

[12] KIF. KIF iOS Integration Testing Framework (cit. on p. 7).

[13] Mario Linares-Vásquez, Kevin Moran, and Denys Poshyvanyk. “Continuous,evolutionary and large-scale: A new perspective for automated mobile apptesting”. In: Proceedings - 2017 IEEE International Conference on SoftwareMaintenance and Evolution, ICSME 2017. 2017 (cit. on p. 5).

[14] Harish Mohan. How to do mobile app automation testing using MonkeyTalktool? 2016 (cit. on p. 8).

[15] Mozilla. Firefox for iOS (cit. on p. 13).

[16] Quantum. Quantum Starter Kit (cit. on p. 8).

[17] Ranorex. iOS Automation Testing with Ranorex Studio (cit. on pp. 6, 7).

[18] Gaurav Salwan. Why You Need Automation Testing For Mobile Apps & TheRight Way To Do It. 2016.

[19] SmartBear Software. Mobile Application Test Automation (cit. on p. 8).

25

Colophon

This document was written in LATEX 2ε. It utilizes the stile Clean Thesis developed byRicardo Langner.

Documents

User Interface Test Automation Tools for iOS Applications