Permission to copy, without fee, all or
part of this material, except
copyrighted material as noted, is
granted provided that the copies are
not made or distributed for
Conference Paper excerpt
This paper describes a test harness that can be used for developing test automation for
websites. The harness provides a set of classes that a test developer can use to develop
custom coding solutions for controlling (automating) and verifying websites. The harness
is completely browser agnostic meaning a single code base is developed that can
execute on multiple browsers. The current implementation is targeted for the Windows
Last year, our department began development of a server management website. Our
team (the UI test team), was given the challenge of developing a test harness that
would allow for the development of a completely automated regression suite and
would give the capability of running against multiple browsers using a single code base.
Our initial idea was to develop a harness compatible with IE7 (Internet Explorer 7) and
Firefox. However, we wanted to make it easily extendable to other browsers and
wanted to shield the test developer from needing to write two sets of code
implementing the same test case.
We addressed this challenge by developing a set of classes that allows the tester to
author tests independent of the target browser (IE7 and Firefox in our case). The test
code implements a single test case that could be targeted towards different browsers.
The test code is completely browser agnostic. Nowhere would one read anything
specific to a particular browser. The following example shows our intention:
1. Open the browser and navigate to a live.com
2. Enter search criteria “cats”
3. Click search button
4. Verify www.catsite.com is found
5. Click on the link if found
6. Close the browser
PSNQC Conference Paper
Presenters: Jagannathan Venkatesan and Craig Merchant
Contributor: Manuel Tellez
Web UI Automation – A Browser
Agnostic harness for Web UI Testing
Excerpt from 2008 PNSQC Proceedings
Page 2 of 26
Implementation (shown in pseudo code)
//Create browser control object , browser
browser = new Browser()
//navigate to live.com
//try to find the expected search result
browserInput input = FindInputElement(“www.thecatsite.com”);
//verify the search result
If (input == null)
Microsoft has a significant number of test teams writing website automation. We define
website automation as the ability to programmatically control a website and verify
content of the website. We found that most solutions fell into one of three buckets:
The characteristics of these solutions appear below:
Record and Playback
� Easy to record a script. Just Point and click. A script of all user input
is created automatically from the recording.
� Different scripts need to be recorded for different browsers. This
type of approach is definitely not browser agnostic.
� A slight change in the UI requires scripts to be re-recorded. Thus,
recording tasks must be duplicated for each UI revision.
� Low level DOM(Document Object Model) access.
� No special development environment required other than cscript.
Notepad is the editor of choice.
level languages. We need the ability set break points for failure
investigation instead of relying on print statement debugging.
experience we have observed naturally caters to UI automation).
Excerpt from 2008 PNSQC Proceedings
Page 3 of 26
� Provides extremely low level access to all controls on the window.
� Fast and reliable.
� .Net classes available in System. Automation
� Different browsers have different automation ids for the same
control. In fact, the programmer has no control over the
automation id. Browser agnostic tests would not be possible here.
We found that no single solution satisfied all our requirements. Thus we decided to
produce a custom harness. The design of the harness is detailed in the next section.
Our test team does leverage existing tools when possible. Prior to the proposed web
based management solution, our team worked on traditional windows software. Our
team developed DFS (Distributed File System) Management Console, Share and
Storage Management Console, and some other management applications on
Windows Server 2008. From our prior work in UI automation, we had a UI automation
harness. The harness was designed with extendibility in mind. It provided a skeleton on
which to build the new web automation harness. Building off the existing harness
allowed our new tests to have an identical structure to our non-web based tests and
allowed us to reuse code for common functions like logging.
Our existing harness was designed to allow users to mix and match back end
automation techniques. Thus the browser agnostic web automation harness was a
natural extension and did not involve any re-design work. We designed two new plug
and play components for dealing with Firefox and IE7. At runtime, the harness loads the
desired plug and play component depending on the browser being tested and from
there the harness uses the loaded plug and play component to control the browser.
Additionally, our current harness is .Net based (C#). Test development and debugging
can be done in MS Visual Studio. Being implemented in a high level object oriented
language makes it accessible to a wide developer audience.
Excerpt from 2008 PNSQC Proceedings
Page 4 of 26
The harness has two main layers. They are the Test Case Layer and the External layer.
(Please see Figure 1 for a block diagram of the layers.)
The TestCase layer provides several static classes such as a logger, an environment
mapping resolver for running in particular machine/domain environments, a resource
manager for providing support for running the tests in different locales, and a
performance monitor for collecting performance statistics. It also provides several
dynamic classes for dealing with user interfaces. This is the layer at which we added an
extension to allow test cases to control the browser.
The External Layer level provides specific control to particular browsers such as IE7 and
Firefox. It is at the external layer where we added the plug and play components to
control IE7 and Firefox. When we say plug and play, we are indicating that each
component will implement the same interface. In this way, the objects in the test case
layer can instantiate the objects in the external layer and use them interchangeably to
achieve browser agnostic behavior. The test developer deals with the objects in the test
case layer only and is unaware of which plug and play component has been loaded.
He/she simply calls into the test case layer objects which handle the communication to
the external layer component.
Comparison to Existing Approaches
The following chart shows how our design compares to the existing approaches to web
automation mentioned above,
access to all
NO NO YES NO NO NO NO
YES YES NO NO NO YES YES
NO YES YES YES YES YES YES
YES YES YES YES YES YES YES
The harness is structured such that a test case derives from the BaseUITest class which
derives from the TestCaseBase class. This object oriented approach allows the tester to
immediately have access to pre-existing functionality such as logging, environment
management for running in different machine/domain settings, resource management
for localization testing, performance monitoring for tracking runtime statistics such as
memory usage and CPU utilization, test data management, and other areas. The tester
can extend or override existing code in order to