HAL Id: hal-02909787https://hal.archives-ouvertes.fr/hal-02909787

Submitted on 31 Jul 2020

HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, estdestinée au dépôt et à la diffusion de documentsscientifiques de niveau recherche, publiés ou non,émanant des établissements d’enseignement et derecherche français ou étrangers, des laboratoirespublics ou privés.

Codeless web testing using Selenium and machinelearning

Duyen Phuc Nguyen, Stephane Maag

To cite this version:Duyen Phuc Nguyen, Stephane Maag. Codeless web testing using Selenium and machine learning.ICSOFT 2020: 15th International Conference on Software Technologies, Jul 2020, Online, France.pp.51-60, �10.5220/0009885400510060�. �hal-02909787�

Codeless Web Testing using Selenium and Machine Learning

Duyen Phuc Nguyen a and Stephane Maag b

Samovar, CNRS, Telecom SudParis, Institut Polytechnique de Paris, Francenguyenduyenphuc@gmail.com, stephane.maag@telecom-sudparis.eu

Keywords: Codeless Testing, Web Testing, Automation Testing, Selenium, Machine Learning, SVM.

Abstract: Complexity of web systems lead to development processes always more tough to test. Testing phases arecrucial in software and system engineering and are known to be very costly. While automated testing methodsappear to take over the role of the human testers, the issues of reliability and the capability of the testingmethod still need to be solved. In our paper, we focus on the automation of functional tests of websites. Asingle web page may contain a huge set of important functionalities leading to the execution of critical webservice operations. Besides, testing all of these functionalities implemented in a web page service is highlycomplex. Two current popular research areas for automation web-based testing are Codeless Functional TestAutomation and Machine Learning/Artificial Intelligence (ML/AI) in test automation. We therefore defineand implement a framework to figure out how to automate the web service product under test, the machine candetect or predict the change and adapt those changes to suitable generic test cases. In our work, we examineon Selenium and the benefits of using machine learning in automated web application testing.

1 INTRODUCTION

Software systems and their complexity distributedthrough diverse components, sometimes virtualizedusing different platforms (e.g., Xaas) (Patel et al.,2016), lead to development processes always morecomplicated to test. However, testing phases are cru-cial in software and system engineering (Kassab et al.,2017). They are known as very costly necessitatingresources (including time and people) spent to pre-pare testing architectures, test case scenarios and theirexecution. That cost is estimated to be between 40%and 80% of the total cost of the service/system devel-opment (Heusser and Kulkarni, 2018). While auto-mated testing methods appear to take over the role ofthe human testers, the issues of reliability and the ca-pability of the testing method still need to be solved.In our paper, we focus on the functional tests of web-sites. A single web page may contain a huge set ofimportant functionalities leading to the execution ofcritical web service operations. Besides, testing all ofthese functionalities implemented in a web page ser-vice is highly complex. For that purpose, several au-tomation tools are available to help the testers and tofacilitate the execution of testing processes (Hynninenet al., 2018; Raulamo-Jurvanen et al., 2019). Among

a https://orcid.org/0000-0003-3740-3586b https://orcid.org/0000-0002-0305-4712

them, we may cite the well known and adopted toolsin the industry: Selenium and OpenScript. Recently,it has been shown that Selenium (Selenium, 2020)is very efficient for testing applications which arebrowser based (Jain and Rajnish, 2018). It runs spe-cific tests on diverse installed browsers and return re-sults, alerting you to failures in browsers as they cropup.

However, automated web-based testing tools likeSelenium still have major drawbacks. Indeed, testshave to be repetitive and require resources and a lotof maintenance. A very small change or modificationin the content of a web page may cause a test fail.Furthermore, many tests scripts are manually handledwhich increases the costs and time for testing (Shariffet al., 2019; Ameur-Boulifa et al., 2019). Researchersand industrials are studying and defining optimizedapproaches to deal with this challenge. Two cur-rent popular research areas for automation web-basedtesting are Codeless Functional Test Automation andMachine Learning/Artificial Intelligence (ML/AI) intest automation. In here, algorithms are defined andimplemented to figure out how to automate the soft-ware/web service product under test, the machine candetect or predict the change and adapt those changesto the suitable test cases. Thanks to codeless testing,testers can drastically reduce the cost of test automa-tion creation, test execution, and maintenance. In our

work, we examine on Selenium and the benefits ofusing machine learning in automated web applica-tion testing. The motivation is to utilize the seleniumframework in the world of automated testing and uti-lize a machine learning approach for codeless testing,enabling to test multiple functional web pages with-out using code.

The main contributions of our paper are:

• Defining a framework named “codeless testing”by combining Selenium and machine learningtechnique,

• Implementing and applying our framework to rungeneric test cases in order to test the functionalitysearch that is present in multiple web pages,

• Through our experiment results, we show that thesuccess of generic test cases may help to con-tribute in decreasing implementation and main-tenance costs of automated tests of functionalbrowser based applications.

Finally, our paper is organized as it follows. In Sec-tion 2, we present the state of the art about automatedcodeless testing techniques and tools. Then in Sec-tion 3, we depict the basis used in our paper. Our MLbased web testing framework is defined in the Sec-tion 4 and evaluated in the Section 5 in which relevantexperimental results are presented. We conclude andgive perspectives in Section 6.

2 RELATED WORKS

In the literature, there are several papers introduc-ing approaches for automation web-based applica-tions testing by applying machine learning tech-niques (RJ Bhojan, 2019) (Rosenfeld et al., 2018)(Nguyen et al., 2018) (Joshi, 2016). Although wegot inspired of these works, none of them has pro-posed novel codeless web service testing. With thesame observation, codeless testing approaches havebeen recently proposed. We may cite (Isha and Re-vathi, 2019) in which the authors proposed an auto-mated API testing tool in providing a GUI with sup-port for codeless testing. However, they do not in-tend to use any learning techniques that could im-prove their work and the problematic we are tacklingin our paper. We may note the same lack in (Brueck-mann et al., 2017). Though the technique proposed isnot entirely codeless, the authors proposed a frame-work to implement a simple business process by in-tegrating existing services in a mashup with codelessengineering processes. However, they still do not in-tegrate any learning techniques.

Moreover, in the market area, many start-up com-panies with several tools have entered the market re-cently, all with the promise of solving the coding skillconundrum. While no-code or low-code conceptsbecome a trend (following the recent Google patent(Arkadyev, 2017)), companies such as Salesforce pro-mote plug-and-play offers. Some testing tools suchas Testcraft1, Ranorex2, Tricentis3, Leapwork4, etc.advertise that they can provide this functionality bybuilding a user-friendly UI on top of the code layerand also enabling switching between two modes. Thecompanies promote that testers can still write scriptsand receive detailed feedback but skip the coding part.However, when it comes to test automation, an issueis often raised on how codeless is being interpreted.Moreover, none of these codeless testing tools pro-posed by those companies are open-source.

In our approach, we cope with these above men-tioned issues by defining a machine learning basedapproach for codeless web automation testing.

3 BASICS

3.1 Selenium

Selenium (Selenium, 2020) is a suite of automationtesting techniques and tools which allows the testersto run the tests directly on the target browser, drivethe interactions on the required web pages and rerunthem without any manual inputs. Selenium has be-come very popular among testers because of the var-ious advantages it offers. With its advent in 2004,Selenium made the life of automation testers easier,highly improved the testing time, costs, detected fail-ures and is now a favorite tool for many automationtesters. Selenium was invented with the introduc-tion of a basic tool named as JavaScriptTestRunner,by Jason Huggins at ThoughtWorks to test their in-ternal Time and Expenses application. Now it hasgained popularity among software testers and devel-opers as an open-source portable automation testingframework. Nowadays, Selenium is currently usedin production in many large companies as Netflix,Google, HubSpot, Fitbit, and more. According to(Enlyft, 2020), there are 42,159 companies that useSelenium, taking 26.83% market share in softwaretesting tool (see Figure 1).

The primary reason behind such overwhelming

1https://www.testcraft.io/2https://www.ranorex.com/3https://www.tricentis.com/4https://www.leapwork.com/

Figure 1: Selenium used by company size.

popularity of Selenium is that it is open source. Thisnot only helps keep the costs in check but also en-sures that companies are using a tool that will getcontinually updated. Other reasons include the multi-dimensional flexibility that it offers in terms of script-ing languages, operating systems, browsers, and in-tegration with other tools. This widens the scale ofreach and test coverage, enabling enterprises to de-liver a web application that is highly reliable andfunctional. Selenium test scripts can be written inJava, Python, C#, PHP, Ruby, Perl and .Net. This al-lows a large number of testers to easily use it withoutany language barriers. It can be carried out on Win-dows, MacOS, and Linux, using any browsers out ofMozilla Firefox, Internet Explorer, Chrome, Safari,and Opera. This enables a thorough cross browsercompatibility testing with strong environment sup-port.

Selenium suite includes three major components(Figure 2), each tool has its own approach for au-tomation testing. The testers or developers can choosetools out of it depending upon the testing require-ments.• Selenium IDE.

Selenium IDE, earlier known as Seleniumrecorder, is a tool used to record, edit, debug andreplay functional tests. Selenium IDE is imple-mented as an extension to the Chrome browserand add-on in Firefox browser. With SeleniumIDE plugin, the testers can do simple record-and-playback of interactions with the browser, theycan also export tests in any of the supportedprogramming languages like Ruby, Java, PHP,Javascript, etc.

• Selenium Grid.Selenium Grid allows the testers to run paral-lel automated tests on multiple machines andbrowsers at the same time. The main functionof this tool is to save time. If the test suite islarge, the testers can use Selenium Grid to reducethe time running. Considering how scripts nor-mally run slow on a browser, using performance-

improving techniques such as parallel testing canhelp with the problem. Testers can also use it totest one application in different browsers in par-allel, when one machine is running Firefox, theother Chrome, and so on. Testers can also cre-ate different configurations with Grid, combiningdifferent versions of browsers and operating sys-tems. Needless to say that when used in large pro-duction environments, Grid is a huge time-saver.

• Selenium Webdriver.Selenium Webdriver (which is also known as Se-lenium 2.0 or 3.0 currently version) is an en-hanced version of Selenium RC and the most usedtool. Selenium Webdriver is by far the most im-portant component of Selenium Suite. It pro-vides a programming interface to create and ex-ecute automation scripts. It accepts commandsvia client API and sends them to browsers. Se-lenium WebDriver allows testers to choose a pro-gramming language of their choice to create testscripts. Test scripts are written in order to iden-tify web elements on web pages and then desiredactions are performed on those elements. Sele-nium Webdriver currently supports most popularbrowsers (Chrome, Firefox, Opera, etc.). Everybrowser has different drivers to run tests. In our”codeless testing framework”, we use SeleniumWebdriver to conduct the automated tests on mul-tiple popular web browsers.

Figure 2: Selenium suite components.

3.2 Codeless Testing

Despite the advantages above listed, Selenium stillhas limitations. The most challenge of using Sele-nium in automation testing is steep learning curve.Testers require high technical skills to accuratelydesign and maintain test automation. Maintenanceincluding modifications and updating Selenium testcode in an efficient way is a very common problemin automated test. The impact of the changes occurin the web page or application under test could suf-fer during its development. Changes or modificationsfrom web User Interface (UI) or from its structure toits elements or its attributes could make the whole

test suites collapse. For that purpose, one of auto-mated testing trends, namely Codeless Testing, wasintroduced to try resolving those issues.

Codeless Testing for web services refers to themethodology which utilizes a generic test case totest multiple websites. This approach allows anytester without deep programming knowledge to per-form tests. Organizations started adapting tools andapproaches to simplify test automation and empowerteam members who lacked sophisticated program-ming skills. Codeless tools were originally meant tohelp the tester avoid the hours of programming thatare usually necessary to get the most out of testinglogic. While their objective was to address program-ming complexity, most tools in the market adapted ano-code approach by avoiding the code, but not reallyaddressing the logical complexity in testing. A com-mon misconception is that codeless test automationtools should completely avoid code. We believe thisis actually a disservice, as very soon users will starthitting roadblocks. Testing requirements are typicallyas vast as application development. It is hard to be-lieve that all testing requirements could be addressedwith some canned, pre-packaged solution. Some levelof logic development flexibility is required, but in away so that the user does not get bogged down bythe syntactical complexities. Note that, though thename codeless testing, it does not mean it is com-pletely code free. While a tester can generate most ofthe tests code free, certain tests may still need somecoding. Testers can use codeless testing for keepingup with the deployment needs.

3.3 Support Vector Machines

A support vector machine (SVM) (Vapnik and Vap-nik, 1998) is a linear classifier defined by a separat-ing hyperplane that determines the decision surfacefor the classification. Given a training set (supervisedlearning), the SVM algorithm finds a hyperplane toclassify new data. Consider a binary classificationproblem, with a training dataset composed of pairs(x1,y1), . . . , (xl,yl ), where each vector xi ∈ Rn andyi ∈ {−1,+1}. The SVM classifier model is a hyper-plane that separates the training data in two sets corre-sponding to the desired classes. Equation (1) definesa separating hyperplane

f (x) =wTx+b = 0 (1)

where w ∈ Rn and b ∈ R are parameters that controlthe function. Function f gives the signed distancebetween a point x and the separating hyperplane. Apoint x is assigned to the positive class if f (x) ≥ 0,and otherwise to the negative class. The SVM al-gorithm computes a hyperplane that maximizes the

distance between the data points on either side, thisdistance is called margin. SVMs can be modeled asthe solution of the optimization problem given by (2),this problem maximizes the margin between trainingpoints.

minw,b

12‖w ‖2

subject to: yi(wTxi+b) ≥ 1, i = 1, . . . , l(2)

All training examples label −1 are on one side of thehyperplane and all training examples label 1 are onthe other side. Not all the samples of the training dataare used to the determine the hyperplane, only a sub-set of the training samples contribute to the definitionof the classifier. The data points used in the algorithmto maximize the margin are called support vectors.

4 A ML-BASED WEB TESTINGFRAMEWORK

The architecture overview of our codeless testingframework is illustrated in Figure 3.

Figure 3: Our architecture of codeless testing framework.

As mentioned in the previous sections, our frame-work is combined of Selenium and a machine learningalgorithm. It is composed by four main components.• Selenium Webdriver Component.

This is the major module, it plays a role as an en-gine to drive the browser for automating websiteand for testing. As referring in Figure 2, Sele-nium Webdriver (web, 2020) is the core moduleof our testing framework. Selenium WebDriverplays as a browser automation framework that ac-cepts commands and sends them to a browser. Itis implemented through a browser-specific driver.It controls the browser by directly communicatingwith it. The flexibility that Selenium Webdriverprovides is almost unmatched in the test automa-tion world.

• Scraping Data Component.We implemented our scraping tool in Python.Besides, we utilize BeautifulSoup (Richardson,2020) as a web scraping library to pull data outof DOM in HTML or XML format.

• Processing Data Component.This component has a role to process the data af-ter pulling from HTML DOM file, its main goal isto clean up and extract the useful data, then trans-form the cleaned data to the next stage for train-ing.

• SVM Model.A machine learning model to recognise and/orpredict the search element pattern which appearsin HTML data of testing website.

4.1 Scraping and Processing Web Data

Figure 4: The structure of our scraping tool.

The general architecture of our scraping tool isshowed in Figure 4. Its main target is exactly as itsname - to collect HTML data and their position in theDOM tree from the website which is driven by Se-lenium Webdriver. We use both Selenium Webdriverand Request library to send a HTTP request to theURL of the webpage we want to access. We discoverthat some websites with heavy render JavaScript willnot response to the request module. Also, some web-sites can detect the request is performed in an auto-mated way (not by human browsing but expected tobe a bot) by their tool, it will deny to establish the con-nection. Selenium Webdriver works fine with almostall major websites, however the speed is recordedas very slow since Selenium needs to open the realbrowsers to perform tasks. Moreover, using differ-ent browsers (Chrome, Firefox, Opera, etc.) maylead to obtain different behaviours of navigation web-sites. For example, some websites are optimized forChrome but not Firefox or vice versa, or some websitemay crash down on Microsoft Edge browser. There-fore, we use both modules Selenium Webdriver andRequest library (Pypi, 2020) to ensure that our scrap-ing tool can adapt with most of the websites to col-lect as most as data it can. First, using request libraryfor speeding up the test, if a request fails, then Sele-nium Webdriver will easily detect it. Moreover, eachwebsite has its different DOM structure, thousandsof websites will have thousands of DOM structures,therefore it is a challenge process to retrieve in an au-tomated way the data from multiple of websites con-

currently. Nevertheless, our scraping tool enables toscrape and collect variety of websites. Our scrapingtool follows two tasks:1. Query the website using requests or Selenium

WebDriver and return its HTML content.2. Using BeautifulSoup (Richardson, 2020) and

LXML (LXML, 2020) libraries to go through theHTML structure of each website, parse HTMLcontent and extract the expected data useful totested the targeted functionality embedded in thewebsite.

4.2 Our SVM Model

In considering that web HTML elements are keepchanging dynamically in both structure and attributevalues, it is a very challenging task to adapt test suiteswith the websites changes. In our proposed frame-work, the objective of SVM model is to recognize thepattern of web elements corresponding to the searchbox in each website as illustrated in Figure 5. Specif-ically, the SVM model will learn the HTML structureof each website, in case of any modification in termsof web elements, the SVM model will find the similarpattern of HTML web and adjust its model accordingto the changes.

Figure 5: SVM model detects search box pattern.

For the next testing step, if the website haschanged, the SVM model detects its changes andguides the Selenium code to adapt automatically withit. In this case, the test code will not be broken andcan be reused. More specifically, the goal of the SVMis to train a model that assigns new unseen web ele-ments into a particular category. It achieves this bycreating a linear partition of the feature space intotwo categories. In each website, structured data isextracted for each HTML element, they have to beturned into proper feature vectors. We use SVMmodel to recognize the relevant feature of each webelement corresponding to its density and apply in thefeature importance property of our model. Many fea-tures and preprocessing steps are performed as part

of our cross-validation process, and we kept the onesyielding best performance and moved them in the fi-nal model. Feature importance gives a score for eachfeature attribute, the higher of the score, the more im-portant or more relevant of the feature is. Therefore,the weight for each element attribute is assigned ac-cording to its frequency and informative ranking.

4.3 Data Collection and Analysis

In order to have the dataset for our training SVMmodel, we need to gather the large amount of data. Byusing the scraping tool described above, we are suc-cessful to retrieve the data from the list of 5000 web-sites provided by the Alexa database. This databasecontains more than one million URLs on top rank-ing of Alexa (Helme, 2020). Our tool can scrape anywebsite that has the search functionality, it pulls downall the web elements corresponding to the search box.

However, these elements are collected as raw datain an undesired format, unorganized, and extremelylarge. Furthermore, these elements may be differentfrom one page to another. We need, for further steps,to use the processing tool to enhance the data qualityand, related to our experiments, in particular to extractthe elements linked to the search box (our testing pur-pose).

The main steps for preprocessing data are format-ting and cleaning. Data cleaning is applied to removemessy data, duplicate data and manage missing val-ues. Formatting is required to ensure that all vari-ables within the same attribute are consistently writ-ten. Once structured data is extracted for each HTMLelement, they have to be turned into proper featurevectors.

After this preprocessing phase, vectors with rele-vant features is obtained, our training dataset is built.For that purpose, an important task of labelling isthen performed to classify our vectors depending onthe structure of the ”search box”. We labeled 1000webpages to provide three classes according to thethree cases detailed in Section 5.1. From this trainingdataset, an SVM model has been obtained as aboveexplained.

5 EXPERIMENTAL STUDIESAND RESULTS

5.1 Use Case Studies

To test our codeless testing framework, our expe-riences focus on testing the search-functionality of

websites. To be concrete, let us follow the followingscenario: if the users want to use Google to search forthe term Codeless Testing, they normally open theirfavourite browser, let say Chrome, to navigate to thewebsite of Google. After the official site of Googlepage fully loaded, the users need to locate the searchbox, then type the search term Codeless Testing. Allof this manual steps performed by human can be com-pletely replaced those steps by our codeless testingframework. However, although these steps can be in-deed easily automated for this scenario, our codelesstesting approach intends to make it generic enoughto test the search functionality on diverse websites.By analyzing the multiple search activities on variouswebsites, we group the search scenario in three usecases:• Case 1 (Traditional Search): we call it traditional

because the search box is appeared directly in thewebsite interface, the user can easily locate it andenter the search query. We studied that 80-90%search websites are designed in this case 1 (Fig.6). For example: google, youtube, yahoo, ama-zon, etc.

Figure 6: Case 1 search type: traditional search.

• Case 2 (Hidden Bar Search): The search bar ishidden, the users are required extra step by click-ing the search button to activate/open the searchbar. Once the search bar appears, the user can en-ter his search query (Fig. 7).

Figure 7: Case 2 search type: hidden search box.

• Case 3 (Pattern/Condition Search): the websiteshave many filters/criteria for search, the users

must select the right criteria to proceed. For in-stance, some hotel or housing websites only al-low users to select the special search key in theirpattern database (region, type of house, size ofhouse, price, number of rooms, etc.). Further-more, particular websites only allow the users toperform the search on specify content provided intheir database. If the users type a random searchquery, or the query does not correspond to a spe-cific form, nothing is returned (Fig. 8).

Figure 8: Case 3 search type: filters/criteria search.

In order to test the search functionality in a website,we defined three verdicts within our generic test case:pass, fail, error, as it follows.

• The test is considered as pass if a browser hasfully loaded the website, the “search term” is en-tered in the search box, the website returns the an-swer corresponding to the “search term”.

• The test is considered as fail if the browser wasunable to load the website under test and to navi-gate to the search box.

• The test is considered as error if the website un-der test does not contain any search box or doesnot have the search functionality in its services.

Based on these verdicts and the three above men-tioned use cases, we define other verdicts as it fol-lows:

• When entering “search term” in the search box ofthe webpage under test, “search term” has to ap-pear into the targeted search box, if not, we deliverthe verdict fail.

• The targeted search box has to accept any type ofquery as input (number, text, special characters,etc.). If not, we deliver the verdict error.

• We do not consider webpages where thereare condition/limit/boundary/criteria for the inputsearch (case 3 as described above). In this case,we deliver the verdict error.

• If a website requires more than a step rather thantype search term (case 3), we deliver the verdicterror.

• A website has to provide the response for thesearch term. If an answer belonging to the class“no result found or nothing found” is returned, wedeliver the verdict pass.

5.2 Experiments Setup

In the experiment phase, we test our framework on thedataset as described in Section 5.3.1.Testbed.• OS: x86-64 Microsoft Windows 10 Pro, version

10.0.18363, Build 18363

• Processor: Intel Core i7-7500U CPU @2.70GHzx2

• Memory: 16GB

• SSD: 512GB

• Dataset:

– Metadata: 1.3 Mb in csv format. 1000 rows x 7columns

– Training set: 70% metadata in npy format(numpy array)

– Test set: 20% metadata in npy format (numpyarray).

From this training dataset, our tool processed theSVM model in average 7 minutes using the de-pendencies below mentioned.

• Browsers are detailed in the Table 1

Table 1: Browser Version.

Chrome 78.0.3904.70 (64-bit)Firefox 70.0 (64-bit)Opera 76.0.3809.132

Internet Explorer 11.418.18362Microsoft Edge 44.18362.387.0 Build 18362

Project Tools.In order to run the experiments, we setup our systemenvironment as in the following:

• Python 3.6.8,

• Dependencies: Keras/Tensorflow, scikit-learn:core platform to analyse/process data and trainingSVM model,

• Libraries: Selenium, Request HTTP, BeautifulSoup, LXML,

• Jupyter Notebook App: online web-based plat-form to visualize the data.

5.3 Results and Discussions

5.3.1 Automation Testing against MultipleWebsites

To evaluate the efficiency of our framework, we con-ducted the tests using a generic test case through 1000websites chosen randomly on the list of Alexa de-scribed above. First, the learning approach has beenprocessed. A sample of raw data with 10 features hasbeen scraped (Fig. 9).

Figure 9: Sample of raw data.

The preprocessing phase and specifically thecleaning step allowed to obtain the seven features andone thousand lines of our training dataset. An illus-tration of these lines are presented in the Fig. 10.

Figure 10: Training dataset sample.

Our testing search functionality has been imple-mented in Selenium, the testing results are showed inthe table below (Table 2).

Table 2: Testing against 1000 web sites.

Pass Fail ErrorChrome 48% 18% 34%Firefox 57% 16% 27%Opera 47% 12% 41%

Internet Explorer 9% 19% 72%Microsoft Edge 17% 19% 64%

First, we may be surprised by the number of failand error verdicts. Nevertheless, the high percentageof results getting error does not mean that our frame-work is defective. After analysis, we noted that it isdue to the fact that we did the test on 1000 websiteschosen randomly. Therefore, there are websites thatdo not have the search functionality. From Table 2,we can see that Internet Explorer and Microsoft Edgeperform the worst. It makes sense since Microsoft

has stopped support for Internet Explorer. MicrosoftEdge is replaced as the main and fast browser for theWindows OS. However, the Selenium Webdriver forMicrosoft Edge is not fully supported and still underdevelopment. We encountered that Internet Explorerand Microsoft Edge crashed when testing more than20 websites concurrently.

Figure 11: Automation testing against multiple web sites.

Through our experimental results for automationtesting the search functionality showed in Figure 11,Firefox performs the best in terms of pass cases.Chrome is the best browser to handle the verdicts er-ror. We experienced that Chrome is a good choicewhen testing the websites which are rendered heavilywith JavaScript, while Firefox still suffers for thosesites. Opera is fairly good in our test. We were alsosurprise with the performance of Opera consideredthat it is not as popular as Chrome and Firefox.

5.3.2 Testing with Extension (Add-on)

During our experiments, we noticed that our testswere interrupted if a pop-up windows suddenly ap-pears during the test. The pop-up can be of any type(advertisement, data privacy, etc.) but the main metone was the pop-up which requires to accept the termand privacy when browsing the website as seen in Fig-ure 12. Without clicking the accept button, there is noway to browse the website further. Considering thatthe test is running automated under Selenium code,there is no interference by manual hand to click theaccept button. Therefore, the tests were most of thetime broken when facing with pop-up windows.

In order to solve this problem, we decided to adda blocking pop-up extension to the browsers. This ledto a trade-off, adding this extension allows to test pop-up websites, but in return, the test speeds decreased.The reason is that we modified our Selenium scriptsin order to detect such pop-up windows and then toadd such an extension when it occurs. Therefore, thewebdriver had to trigger automatically the extension

Figure 12: Pop-up blocks the browser in Yahoo website.

in the targeted browser, reducing at the same time theperformance of our tests (in terms of response time -obtained verdicts).

For the experiments, we chose two extensions be-low due to their efficiency in blocking pop-up win-dows, there are more than 10 millions users usingthose extensions:

• Ublock: (ubl, 2020) is a free and open-source,cross-platform browser extension for content-filtering, including ad-blocking.

• I dont care about cookie: (Kladnik, 2020) Thisbrowser extension removes cookie warnings fromalmost all websites and saves thousands of unnec-essary clicks, very useful for our codeless testingapproach.

Since the above extensions are only fully supportedfor Chrome and Firefox, the experiments in this sec-tion were only run on these two browsers. The testswere run on 20 random websites concurrently on eachbrowser in two modes: with extension and without ex-tension. The Table 3 and Fig. 13 illustrate our results.

Table 3: Verdict results when testing with/without exten-sions.

With Extension Without ExtensionPass Fail Error Pass Fail Error

Chrome 12 2 6 9 5 6Firefox 11 3 6 9 5 6

Figure 13: Running time with extensions testing.

Within Table 3, we can note that using extensionto block the pop-up is very effective during the tests.It helps to enhance the pass case and reduce the failone. However, the running time when executing thetests with addons is reduced (Fig. 13). We also notethat Firefox is more efficient than Chrome in termsof time performance in both cases (with and withoutextensions).

6 CONCLUSION ANDPERSPECTIVES

In this paper, we proposed a framework to test multi-ple websites in an automation way. The approach isable to use a generic test case to test a single func-tionality through multiple websites by leveraging ma-chine learning technique and selenium on web ele-ment locators. Our technique can adapt dynamicallyto the changes of the website. This framework called“codeless testing automation” aims to help the testerin reducing the time and efforts that are commonlyspent to change or modify the test codes. In our exper-iments, we focus on testing the search functionality ofa web service. The test scenario is the following: us-ing our framework, without modify the test code, theframework must be able to test the search functional-ity of multiple websites. Note that the categorizationof websites is not a matter, the framework enable tocheck with any website. From the experiments, theresults show that our framework can be efficient toperform the automation testing with most of standardwebsites by using the generic test case without rewrit-ing the test code into Selenium.

From this work, we intend several perspectives.First, we have to tackle all the use cases mentionedin our paper and in particular the third case in whichconstraints are given on the search functionality. Fur-thermore, we aim at testing several behaviors at thesame time. This aspect will need another testing ar-chitecture to consider the huge amount of data neededto be trained and analysed through our framework.Besides, we expect to propose APIs and generic testscripts to the community in order to assess our tech-nique on diverse functional tests. This is will lead tointeresting and relevant comparisons with other ap-proaches. Finally, other learning techniques such asneural based mechanisms are studied.

REFERENCES

(2020). Selenium webdriver document. https://selenium-python.readthedocs.io/locating-elements.

html. Accessed: 2020-04-02.(2020). Ublock blocking popup extension. https://github.

com/gorhill/uBlock. Accessed: 2020-04-02.Ameur-Boulifa, R., Cavalli, A. R., and Maag, S. (2019).

Verifying complex software control systems from testobjectives: Application to the ETCS system. In Pro-ceedings of the 14th International Conference on Soft-ware Technologies, ICSOFT 2019, Prague, Czech Re-public, July 26-28, 2019, pages 397–406.

Arkadyev, A. (2017). Codeless system and tool for testingapplications. US Patent 9,697,110.

Brueckmann, T., Gruhn, V., Koop, W., Ollesch, J., Pradel,L., Wessling, F., and Benner-Wickner, M. (2017).Codeless engineering of service mashups-an experi-ence report. In 2017 IEEE International Conferenceon Services Computing (SCC), pages 402–409. IEEE.

Enlyft (2020). Companies using selenium. https://enlyft.com/tech/products/selenium. Accessed: 2020-04-02.

Helme, S. (2020). List of top one million websites on alexaranking. https://crawler.ninja/files/https-sites.txt. Ac-cessed: 2020-04-02.

Heusser, M. and Kulkarni, G. (2018). How to reduce thecost of software testing. CRC Press.

Hynninen, T., Kasurinen, J., Knutas, A., and Taipale, O.(2018). Software testing: Survey of the industry prac-tices. In 2018 41st International Convention on In-formation and Communication Technology, Electron-ics and Microelectronics (MIPRO), pages 1449–1454.IEEE.

Isha, A. S. and Revathi, M. (2019). Automated api testing.International Journal of Engineering Science, 20826.

Jain, V. and Rajnish, K. (2018). Comparative study of soft-ware automation testing tools: Openscript and sele-nium. Int. Journal of Engineering Research and Ap-plication, 8(2):29–33.

Joshi, N. (2016). Survey of rapid software testing usingmachine learning. International Journal of Trend inResearch and Development.

Kassab, M., DeFranco, J. F., and Laplante, P. A. (2017).Software testing: The state of the practice. IEEE Soft-ware, 34(5):46–52.

Kladnik, D. (2020). Idontcareaboutcookie blocking popupextention. https://www.i-dont-care-about-cookies.eu/. Accessed: 2020-04-02.

LXML (2020). Lxml toolkit. https://lxml.de/. Accessed:2020-04-02.

Nguyen, D. M., Do, H. N., Huynh, Q. T., Vo, D. T., andHa, N.-H. (2018). Shinobi: A novel approach forcontext-driven testing (cdt) using heuristics and ma-chine learning for web applications: An analysis ofchemosensory afferents and the projection pattern inthe central nervous system. In Topographic Organiza-tion of the Pectine Neuropils in Scorpions. INISCOM.

Patel, M., Patel, M., et al. (2016). Survey paper on analysisof xaas. International Journal For Research In Ad-vanced Computer Science And Engineering, 2(2):13–18.

Pypi (2020). Request library. https://pypi.org/project/requests/2.7.0/. Accessed: 2020-04-02.

Raulamo-Jurvanen, P., Hosio, S., and Mantyla, M. V.(2019). Practitioner evaluations on software testingtools. In Proceedings of the Evaluation and Assess-ment on Software Engineering, pages 57–66.

Richardson, L. (2020). Beautiful soup documentation.https://www.crummy.com/software/BeautifulSoup/.Accessed: 2020-04-02.

RJ Bhojan, K Vivekanandan, R. G. (2019). A machinelearning based approach for detecting non- determin-istic tests and its analysis in mobile application test-ing. International Journal of Advanced Research inComputer Science.

Rosenfeld, A., Kardashov, O., and Zang, O. (2018).Automation of android applications functional test-ing using machine learning activities classification.IEEE/ACM 5th International Conference on MobileSoftware Engineering and Systems (MOBILESoft).

Selenium (2020). Selenium automates browsers. https://selenium.dev/. Accessed: 2020-04-02.

Shariff, S. M., Li, H., Bezemer, C.-P., Hassan, A. E.,Nguyen, T. H., and Flora, P. (2019). Improving thetesting efficiency of selenium-based load tests. In2019 IEEE/ACM 14th International Workshop on Au-tomation of Software Test (AST), pages 14–20. IEEE.

Vapnik, V. N. and Vapnik, V. (1998). Statistical learningtheory, volume 1. Wiley New York.