A Workflow-Based Test Automation Framework for Web Based Systems

Weider D. Yu Giri Patil

Computer Engineering Department, San Jose State UniversitySan Jose (Silicon Valley), California, USA, 95192-0180, Email.

Abstract

Due to the high complexity involved in Web baseddistributed systems, it has been difficult to effectivelyperformthe testing process with limited resources to obtain a morethorough coverage of testing. This might open the doors forpotential security attacks. Testing such a system iscomplicated and expensive, and sometimes some tests maynot be possible to perform due to the availability ofresources. A workflow based test automation frameworksystem was designed and implemented to conduct the testcases and demonstrate the concept. A tester can launch atesting process against a target system by clicking a fewbuttons on the user screens, and the framework of theworkflow test automation system performs the entire testingjob in the background. The workflow frameworkcommunicates with the participating testers performingtesting processes, andprovides feedback ofeach test in realtime. The framework also allows plug-ins of a variety oftesting workflow definitions for diferent testing processes.

1. IntroductionThere are lots of technologies involved developing a

Web based system. Technologies provide flexibility in howthey can be used. But if they are not properly used, then itopens the door for the malicious attackers. Hence the testingteam needs to be aware of all the possible ways thattechnology might fail and ensure that the system under testdoes not compromise on data integrity and security.

When a build is delivered to the test team, theorganization expects that all the bugs are found and once theyare fixed, the product can go out of the door. This is notpractical. The reason for not catching all the bugs can be lackof resources, lack of time available for testing, lack ofequipment, etc.

One of the key approaches to testing is to define andexecute a finite number of test cases. Otherwise we wouldend up with performing testing for a very long time. In mostof the test organizations, people who write test cases are theones who execute the test cases as well. So test personnel arenot properly utilized. If the test cases are automated, then thetest

personnel can spend time in designing more test cases toprovide better coverage in testing.

This research focuses on workflow based testframework systems that perform all the testing activities asspecified in the Software Engineering Body of Knowledge(SEBOK) [7]. An easy to use user interface puts the user inthe driver seat. Using the workflow management userinterface, user can easily create test session which is basicallya collection of tests. This is sometimes also called as testsuites. One the user creates the test session, and then he/shecan easily launch the application just by a click on button.Users can then go and do other tasks leaving the testingcompletely in the hands of the workflow based test system.User can at any point of time monitor the status of the testexecution and all the data are updated to the centralizeddatabase in real time.

The workflow based test framework is comprised of theWorkflow Manager and the Workflow Agent. Theseframework components are implemented as Windowsservices. The Workflow Agent launches the test applications,and establishes communication with them. The testapplications communicate with the Workflow Agent andhence with the Workflow Manager in real time. The testapplications communicate the current executing status aswell as the paths of the work products generate by theapplications. Thus workflow based test framework providesa good platform to automate the testing process.

2. Related technologies and workWorkflow Management Coalition (WfMC) has defined

the Workflow Reference Model that identifies the standardcomponents and interfaces involved in any typical workflowsystem [1], [2]. The workflow system is defined by interfaceswhich allow the implementers of the interfaces and the usersof the interfaces to evolve independently [4], 5], [6].

3. Background

3.1. Software testing processSoftware Engineering Body of Knowledge [7] defines

testing as an activity performed for evaluating productquality, and for improving it, by identifying defects andproblems. Software testing consists of the dynamicverification of the behavior of a program on a finite set oftest cases, selected from the usually infinite executions

1-4244-1521-7/07/$25.00 ©2007 IEEE333

domain, against the expected behavior. Dynamic means thatthe systems would be subjected to inputs and these inputswould vary to exercise the various execution paths.

The SEBOK [7] identifies the following test activitiesin the testing process. The following list shows the variousactivities involved in the testing phase of the SoftwareDevelopment Life Cycle (SDLC).

* Planning: This includes coordination of testingpersonnel, availability of test machines, testequipments, test plan creation and documentation.This also includes planning for the time and effort tosetup the test environment.

* Test Case Generation: Test cases are generated basedon the intention of testing. They also depend on thetesting technique to be used for testing. All the testcases generated should be placed under a SoftwareConfiguration Management (SCM). Test sases shouldinclude the steps to execute the tests and also theexpected outcome for each test.

* Test Environment Development: The environmentshould be prepared so that the test tools can beexecuted properly.

* Execution: Test process and the outcome should berepeatable. The process should be well documented sothat any person can perform the test by referring to thetest document.

* Test Result Evaluation: Test result should beevaluated or analyzed in order to determine if theperformed test was successful or not. This is done bycomparing the actual output of the test with theexpected output.

* Problem Reporting and Defect Tracking: All theproblems encountered during the testing should bereported to a SCM system so that the incidents aretracked and managed.

A good Workflow Management System (WFMS),which provides the foundation for defining and executingprocesses, would help in automating the above activitiesinvolved in software testing [14].

3.2. Workflow Management SystemWorkflow at its simplest is the movement of

documents and/or tasks through a work process. Morespecifically, workflow is the operational aspect of a workprocedure: how tasks are structured, who performs them,what their relative order is, how they are synchronized,how information flows to support the tasks and how tasksare being tracked.

Workflow Management Systems can be used asenterprise application integration platform. In mostorganizations, the infrastructure would have variousheterogeneous applications and databases. Theseapplications have a clear purpose when they are introducedin the organization infrastructure. But as time passes byand as the organization grows, newer applications get

intranet applications. But since they all come fromdifferent vendors, there are very little chances that theseapplications could interact with each other and exchangedata or information. Examples of such systems are:customer management, document management, sourcecontrol systems, configuration management systems,incident management, resource management, etc. Thesededicated applications only know about certain businessprocesses they have to support. The enterprise fits theseautomated processes of the software components into theiroverall non-automated business processes [4], [9], [13].Once such dedicated applications are installed andrunning, opportunities arise for new feature requirementsthat span over multiple applications. EnterpriseApplication Integration (EAI) is the discipline ofimplementing software requirements that involve multiplededicated applications. Dedicated applications incorporatea number of hard coded business processes. A WorkflowManagement System on the other hand has got no priorknowledge of the domain. [6]

A WFMS takes a description of a business process asinput and manages the executions of the process instances.That makes it much more flexible than dedicatedapplications, but it is required to create a formal descriptionof the business process. That is why WFMS and dedicatedapplications are complementary. A WFMS can be used tomanage the overall process [4], [8], [11], [12]. As a rule ofthumb, we should use a dedicated application if it supportsthe processes needed. The way to use a WFMS that wasdiscussed here is to tie together all dedicated applications andcreate an EAI-platform. WfMC (a standards committee)defines workflow as an automation of a business process, inwhole or part, during which documents, information or tasksare passed from one participant to another for action,according to a set of procedural rules. [1]

Workflows are all about process automation and help toimprove efficiency and productivity of any organization.This paper will look into automating the software testingphase ofthe SDLC and will demonstrate how Workflows canbe applied to automate the testing process [19]. In addition,an extensive analysis was performed in the area ofhealthcareto understand the feasibility of use [3], [10], [15], [16], [17].

4. Testing issuesThere are specific issues related to testing process for a

test team:Sequential execution ofactivities

The testing activities are carried out sequentially. Thismeans that the time to complete the task is sum of timerequired to complete individual tasks.

Delays between activitiesConsidering practical work environment, there will be

delays between the testing tasks, though these delays aresmall, but they occur more often and the chances that theyhappen are also high.

added to the information technology infrastructure. In thecourse of time the company would end up with many

334

T (total time for Testing) = E t, (time required for eachtesting activities) + E di (delay between testing activities),

Improper utilization ofresourcesThe human resources are poorly utilized. Testers spend

the most of their time in executing the test cases. However ifworkflow is used, then the same use case activities getdistributed.

Need expert users to execute testsThe task of setting up tests is a complicated task. This

means that only skilled users can configure the system toinitiate the tests. This puts a lot of dependency on the skilledusers to carry on the execution of the tests.

It is clear that there are many benefits in automating amanual process in terms of cost and time. Workflowprovides a framework to systematically define and automatea manual process. There has been a great amount of effort indefining standards for workflow based systems.

5. Workflows framework architectureThe framework architecture shown in Figure 1 below is

a workflow system for automation of the test environment.This architecture has been developed with WfMC ReferenceModel in mind. Testing process involves intense interactionwith the system namely: setting up the test environment,running the test application, moving the data to differentmachines for processing, performing the data processing togenerate reports and finally archiving the data. Such a systemdemands that there should be an agent on each of the systemthat will help co-ordinate all system related activities. Thissystem also demands that there is a centralized server thatdrives these activities and co-ordinate with other agents [18].As shown in Figure 1, the workflow system for testautomation is achieved by two major components,"Workflow Manager" and "Workflow Agents".

Workflow Manager:This component is responsible for managing the

currently executing workflow instances. This componentmaintains a database of workflow templates, status of eachinstance and results of each execution. This componentcommunicates with the different Workflow Agents runningon different machines, and feeds them with commands, orgathers status information of each execution from them.

Workflow Agents:This component sits on each of the test bed systems. It

runs as a Windows service. Its main role is to launchapplications, feed them with commands that they aresupposed to execute, and constantly get status informationfrom them. The Workflow Agents then communicate all theinformation gathered to the Workflow Manager, which is thecentral storage repository [9].

User Intrface Web EasedManagement and Stahs tracking U!Startworif1owinstance

A*0 MlIotDtor statbs

Wkfowi103 Mana,gerl =DtbsAppi App2 App 1 App 2

1

4

Figure 1. Architecture for workflow based test automation

Agents also host some system level services like "Filemovers", "Application launchers", "Application managers","Archiving services", etc. that will aid in executing theworkflow properly.

User Interface:Using the interface, the user can submit new testing

tasks that need to be executed and determine the status ofcurrent executing instances. This interface will interact withthe Workflow Manager to update its data.

Communication Protocols:Apart from the new components mentioned, there are

new protocols so that a proper communication is establishedbetween the applications and the Workflow Agents, andbetween the Workflow Agents and the Workflow Manager.As soon as a test machine comes online, the Workflow Agentrunning on that machine reports to the Workflow Managerindicating that it is online and available. The WorkflowManager then instructs the Agent what tests to execute.

6. Experiments and related analysisThis section includes a discussion regarding the

activities involved in software testing and how theseactivities are coved in the workflow based test automation.An experiment was conducted to benchmark workflow basedtest automation. There are many reasons why delays occurbetween the activities. Some of sample cases are mentionedbelow:

Priorities: The test team might have a different higherpriority items to work on.

Member availability: Test team member cannot proceedwith the next step because some members on leave.

335

Interrupts: There might be unjust interrupts with the testteam. This would push out the tasks on hand.

Delayed Notification: At the end of each step, the testteam member may forget to notify other membersresponsible to start the next step of testing.

Figure 3. Form for test environment configuration

Figure 2. Test session creation form

6.1. Problem resolution approachHow the workflow based test automation can overcome

these problems mentioned above is specified below:

1. Planning:The workflow based test automation provides a rich user

interface where the user can define the test sessions. Once theuser defines the test sessions, then the sessions get stored onthe Workflow Manger and can be accessed. Figure 2 showsthe Test Session Generation Form.

2. Test case generation:This step is part ofthe test design activity. In this step the

test team takes the product requirements and the productdesign, and determines how the test cases should be designedso that they can produce the maximum coverage, and canmake sure that the test cases pass through most of theexecution paths.

The test team can design more test cases when using theworkflow based approach. Because most of the testexecution activities are handled by the system, the test teamcan concentrate more on the design of the test and designmore test cases.

3. Test environment developmentThe needs to make sure all the tools are available and

have necessary functionalities to test the new software.

In a workflow based system, the test tools can beinstalled and be ready to use through bookkeeping.Whenever there is a new test tool available, the user justneeds to upload the new test tool to the Workflow Manager(shown in Figure 3). The Workflow Manager would thenmake sure all the Test Machines have the latest copy of thetest tool. This installation of the test tool can be achievedthough the Workflow Agents running on each of the testmachines.

4. Test executionThis step involves actual execution of all the planned test

cases. Workflow helps to truly automate this step. The useruses the Workflow Management User Interface to launch thetest applications. The Workflow Framework will take chargeand does the application launch, execution, data moving andarchiving. The test execution, data processing and analyzingand then archiving of the data all happen automatically at thecentralized Workflow Manager. The user just needs to clickon the Execute Session button. Another advantage of theworkflow system is that the status of execution and the statusof processing are all updated in real time.

5. Test result evaluation:The workflow based test automation system (Workflow

Agent) also takes care of launching the test result processorapplication. This application takes the test data generated andparses the file to evaluate if the test was successful or not.Whenever the Agent determines that the test is complete,then it launches the test result processor application. Thus thesystem helps to automate the evaluation step.

6. Problem reporting and loggingWorkflow based test automation records all the activities

in the system event log. The log serves as the record of thetest executions. Also all the tests executed using theWorkflow Management User Interface are recorded in theworkflow database. These data records can be pulled in forreporting at any point. Since the database is also aware of the

336

test data, it can use some archiving plug-ins to archive thedata.

7. Defect trackingThis activity falls under project tracking and heavily

uses the Software Configuration Management techniques. Itwould be more logical to use external ConfigurationManagement Systems to manage and track the defects.

6.2. BenchmarkingFigure 4 shows the benchmarking on workflow based

test automation system. Some key performance measures are

defined. The system is evaluated under two conditions. Oneenvironment does not use the workflow and the other does.

Identi PeOrmance Measdtes

Ueson Surveys

AoMnfstrSWys

Ahjl{qq Data

RevwOw Individial and Conperative PResWts

Figure 4. Benchmarking processes

6.3. Comparison of two approaches

Figure 5 below shows the cost comparison of the twoprocesses. The process without workflows takes 77% more

execution time to complete.

Figure 5. Execution time comparison

Figure 6. Cost comparison

7. System designThere are many applications involved in a workflow

system. Workflow Manager and Workflow Agent are

implemented using the Remoting Objects. The WorkflowManager and Workflow Agent(s) are implemented as

Windows service. Windows service provides a backgroundtask that helps to carry out the workflow related activities.Main advantages of using the Windows service are that theycan be configured to run automatically when the machinereboots [21]. This will ensure that the workflow processes

are always running. Since the test applications need to talk tothe Agents and the Manger quite often, the Manager and theAgents are exposed as interfaces. .NET Remoting providesan infrastructure for the clients to communicate withsoftware objects on remote servers. The clients can makeremote method calls through the proxy [9], [20].

The workflow is designed in a pull model. When theagents come up, they will report to the Manger that they are

available. The Manager then will execute tests on thatagent/machine. When the test applications come up (usuallylaunched by the Agent), it will request for Agent to executethe tests. Thus the test applications query the system for tasksto execute.

Client DesignClient application communicates to Workflow Agents

through IWFRuntimeAgent interface. Basic operations are toquery the command to execute and also update the status ofthe command it is currently executing. Figure 7 shows theinterface as exposed by Workflow Agent.

Also the workflow based test automation is about 76.5%cheaper than the non-workflow based test execution (shownin Figure 6). Table 3 illustrates the comparison data based onthe three performance parameters identified previously.

Figure 7. Component view of agent

337

Execution Time comparisons between Processes

200 X g0

E 50X150 66 66

o 100 X g_~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~._Xl......... ....l.WithouWorklows Wth Wokflow

Cost Comparison between Processes

200 |

.. 150 |

0a 100

0 50-

Without Workflows With WorkflowsProcecss Type

The middle tier consists of the Workflow Managerrunning on a server and Workflow Agents that run on TestMachines (Figure 8). Workflow Manager and WorkflowAgents are both implemented as Windows services and bothof them are exposed as NET Remoting Objects. TheManager is responsible for managing the test sessions and tokeep track ofthem. The Agents facilitate in launching the testapplications and help in moving and archiving of data. TheUser Interface allows the user to drive the workflow process.

8. Conclusions~~~~~~~~~~~~Ro* flj-t

Thew rklowU framew rk LadLosusrnoeein n

m onitored ionsratmAnThereaemnbenefito sin the systemeistinimizingfthe

dependency nisystemc rsourcesTan experiet skills.Oc theiproesseisauofmated tanybodycn strtceusing thmesystems.AlTheshwoesfnot nraeedrtoaknowiswhtserst toru,adefiseanolobntongesnetofworry aboeeutehotodrun thetesuts.The resutmofitesigarrpaedin real timean th usrgs

conotinuou fenedbac ofsnal thetestslued.smiiiin hThpeneconssmresourcesaredproperlyuiizdi wokflow. basedth

troestiautomato.Tetesttamyod can focrtusinmoreonsdesigning

lngew testdcasesoand deveopingo tes autoaintooless.AlThereutoftedtious maneualatiiiedeated toea tiestn are taken aergetbycothenworkflowdsstem Compaltex webt basnced.sse etn a

be rsureautomatedlyutiusingiworkflowsTessea bautmaed

cedousmplexlsecurityierelated ttesting.As theWaebServieAPI

can be thoroughly tested using this automated system [14],[20]. The architecture of the workflow based test automationis simple and can be applied to testing process of any domain.

[1] David Hollingsworth (Jan 19, 1995), "Workflow ManagementCoalition: The Workflow Reference Model" (Retrieved July 09,2006) fromhtp/www c.r/tnad/ostO3ILd

[2] Dino Esposito of Microsoft (March 2006), "Cutting Edge:Windows Workflow Foundation". (Retrieved July 09, 2006) from

[3] Dwivedi, A.; Bali, R.K.; Naguib, R.N.G.; Nassar, N.S., "Clinicalknowledge management for healthcare," Information TechnologyApplications in Biomedicine, 2003. 4th International IEEE EMIBSSpecial Topic Conference on, vol., no.pp. 157- 160, 24-26 April2003

[4] Scott Neumann,Kamaraj Shankar,Ali Vojdan. "ApplyingWorkflow Technologies to Integrate Utility Business Processes",(Retrieved July 09, 2006) from

[5] Diimitrios Gorgakopoulos, Mark Hornick And Amit Sheth.(1995), "An Overview of Workflow Management: From ProcessModeling to Workflow Automation Infrastructure", (2000, May4). Retrieved July 09, 2006, fromhtp/cesrit

[6] Tom Baeyens (May 2004), "The State of Workflow", (RetrievedNov 2006), from

[7] SWEBOK 2004, "Guide to the Software Engineering Body OfKnowledge", ISBN 0-769.5-2330-7, 2004

[8] Workflows, from Wikipedia the free encyclopedia (RetrievedNov 2006), fromhtp/e.iiei.rwk/W kfo

[9] Qi Xu Qiu, R. Fuyuan Xu, "Agent-based workflow approach tothe design and development of cross-enterprise informationsystems", IEEE Press, page: 2633 - 2638 vol.3, ISSN:]062-922X, Oct. 2003

[10] Sachpazidis, "d4~Home: A modular Telemedicine System",

[11I] Usability, from Wikipedia the free encyclopedia(rrtrieved Nov 2006) from

L12] Definitions, from Wikipedia the free encyclopedia (RetrievedNov 2006) from wwwl-ei.o

[13] Workflow management Coalition, "Relationship betweenBusiness process elements", (Retrieved Nov 2006),

[14] Colin Wong, Daniel Grgelak, "A Web Based Security TestingFramework" ,

338

Middle-Tier Design References

[15] Mauro Regio of Microsoft Corporation, "Web ServicesEnablement for Healthcare HL7 Applications - Web ServicesBasic Profile Reference Implementation",

[16] Hudson, D.L.; Cohen, M.E., "The role of information technologyin disease management," Information Technology Applications inBiomedicine, 2003. 4th International IEEE EMBS Special TopicConference on, vol., no.pp. 62- 65, 24-26 April 2003

[17] Chao, H.M.; Twu, S.H.; Hsu, C.M., "A secure identificationaccess control scheme for accessing healthcare informationsystems," Information Technology Applications in Biomedicine,2003. 4th International IEEE EMBS Special Topic Conferenceon, vol., no.pp. 122- 125, 24-26 April 2003

[18] Michael zur Muehlen, Logos Verlag, Berlin 2004 Workflowbased Process Controlling ISBN 3-8325-0388-9

[19] [Kan99] C. Kaner, J. Falk and H. Q. Nguyen, "Testing ComputerSoftware, " Second ed: Wiley, 1999

[20] Geer. D, "Taking Steps to Secure Web Services", IEEE ISSN0018-9162, vol36 issue 10, pages 14-16, Oct 2003

[21] Bowe, P.; O'Flaherty, B., "Implementing workflow automationsystems: implications for management control," System Sciences,1996., Proceedings ofthe Twenty-Ninth Hawaii InternationalConference on, , vol.5, no.pp.13-22 vol.5, 3-6 Jan 1996

339