Evaluating Functional Testing Solutions

TeleconferenceEvaluating Functional Testing Solutions: The Forrester Wave™ Q2 2006

Carey Schwaber

Analyst

Forrester Research

June 27, 2006. Call in at 12:55 pm Eastern Time

2Entire contents © 2006 Forrester Research, Inc. All rights reserved.

Agenda

• The what and why of functional testing solutions

• How Forrester evaluated functional testing solutions

• The findings of Forrester’s evaluation

• Recommendations and WIM


Agenda






Definitions

Functional tests:

►Tests to verify that application functionality conforms to predefined specifications


What are the key ingredients of a functional testing solution?

• In scope:

» Manual testing

» Functional test automation

» Test management

• Out of scope:

» Unit testing

» Static analysis

» Performance testing

» Application monitoring


“Does your shop conduct any automation of functional test scripts?”

Yes, for some testing efforts

39%

No, not for any testing

efforts52%

Yes, for every testing effort

9%

Base: 74 IT decision-makersat North American and European companies


Pros and cons of manual testing

Pros• Cost of test case design is minimal

» No required tools or tool expertise

» No automation necessary

» No need for lead time prior to test execution

• Can be scripted, exploratory, or both

» Simultaneous learning, test design, and test execution (Bach)

» Useful before, during, and after design and execution of standard manual test scripts

Cons• Cost of test execution is high

» Cost = execution time X labor rate

» Execution time is high

» No efficiencies when execution is repeated

• Scripted test execution is tedious

» No creativity required

• All forms are highly error-prone

» Quality depends on tester’s attention to detail over time

» Documentation of test results is another potential source of errors


Specialized tool support can improve efficiency of scripted manual testing

• Tool support helps manual testers:

» Store test plans, test scripts, and test results in a single, secure location

» Share test components (e.g., “login”) across test cases

» Automate data entry and data verification


Pros and cons of test automation

Pros

• Frees testers for more intelligent types of testing (e.g., exploratory testing)

• Drives down time, cost of test execution

• Permits shops to expand scope of testing efforts

Cons

• Increases up-front investment in test design

• Easy to waste time automating the “wrong” tests — or the right tests in the wrong way

• Requires more technical expertise, specialized tools support than manual testing


A simplified view of the economics of test automation

If a test script will be run every week for the next two years, automate the test if thecost of automation is less than the cost of manually executing the test 104 times.

Cost of automation Cost of manually executing the test as manytimes as the automated test will be executed

Choose to automate when:

Cost of test automation Cost of tool(s)Labor costs ofscript creation

Labor costs ofscript maintenance

To calculate the cost of automating a test script:


“Why does your company not perform any test automation?”

Base: 38 North American and European IT decision-makersat companies that don’t perform any test automation

16

14

8

7

2

4

We lack the necessary tools

We don’t believe in the ROI of testautomation

We just haven’t gotten around to it

We lack the necessary skills

Don't know

Other


The right balance varies by test effort

• Testing team composition

» Programming skills vs. subject matter expertise

» Division of labor that leverages each team members’ strengths

» Assessment of development team’s own testing efforts

• Nature of the application under test

» Application technology

» Application stability

• Timeline

» Time available for creation of automated test scripts

» Expected lifetime of the application


The benefits of an integrated test management solution for manual and automated testing

• Common interface for planning and monitoring all test activities

• Change management for manual and automated test assets

• Submission of defects from manual testing and test automation tool directly to test management tool

• Incremental automation of portions of the test suite


Agenda






How did we decide which vendors to evaluate?

Criteria for inclusion:

• $10M in annual revenues

• Support for manual testing, test automation, and test management

Some excluded vendors:

• RadView and Seapine» Both included in last year’s Wave of test automation tools

• Worksoft, SDT, and LogiGear» Focus on keyword-driven test automation

• Agitar and Parasoft » Focus on developer testing


Evaluated vendors and their product offerings

Borland Software

SilkTest, SilkCentral Test Manager, and SilkCentral Issue Manager

CompuwareQACenter Enterprise Edition: TestPartner, QARun, QADirector, TrackRecord, Reconcile

Empirix e-Tester, e-Manager Enterprise

IBMRational Functional Tester, Rational Robot, Rational Manual Tester, Rational ClearQuest

MercuryInteractive

QuickTest Professional, WinRunner, Quality Center


The Forrester Wave™ evaluation process

• Evaluation conducted between February and May of 2006

» Based on product capabilities generally available by June 1, 2006

• Open process to select 87 evaluation criteria:

» Interviewed vendors, experts, outsourcers, and users

• Vendor self-evaluations

» Evaluation relies in part on data provided by vendors

• Interviews on vendor strategy

» Conversations with executives to determine how vendors will enhance their offerings in the future

• Product demonstrations

» Validate our understanding of product capabilities

• Extensive fact-checking with customer references

» Determine how vendor offerings work in practice as well as in theory


Evaluation criteria

• Forrester evaluated these five vendors’ solutions across 87 criteria

• These criteria fell into three categories (and 19 subcategories):

» Current offerings

» Strategy

» Market presence


Current offerings criteria

Solution architectureWhat is the architecture of the vendor’s functional testing solution?(Operating systems, databases, directory servers)

Life-cycle integration

With what life-cycle tools do the products in the vendor’s functional testing solution integrate?(Requirements management, issue management, SCM, performance testing, SOA management, integration interfaces)

Manual testingHow strong are the manual testing capabilities of the vendor’s functional testing solution?(Test case creation, attachments, reuse, impact analysis)

Test automationHow strong are the test automation capabilities of the vendor’s functional testing solution?(Testable applications, object recognition, script creation)

Test management

How strong are the test management capabilities of the vendor’s functional testing solution?(Scalability, supported test assets, test planning, test execution, workflow, reporting and analytics, collaboration)


Strategy criteria

Product directionHow well does the vendor’s product strategy position it for market leadership?(Product vision, product roadmap)

FocusWhat percentage of the vendor’s resources are focused on the products in its functional testing solution?

R&DHow significantly is the vendor investing in R&D for the products in its functional testing solution?(R&D resources, R&D expense ratio)

Price

What is the price of the vendor’s functional testing solution?(Test automation price per user, manual testing price per user, test management price per user, solution price for a typical deployment, maintenance price, pricing structure)

Financial resources to execute on strategy

What is the ratio of the vendor’s current assets to its current liabilities?


Market presence criteria

Installed base

How large is the vendor's installed base in terms of unique companies?(Total installed base, functional testing installed base, new functional testing customers, functional testing reference customers)

Financial strengthHow financially healthy is the vendor?(Revenues, revenue growth)

Employees How many employees does the vendor have?

TrainingWhat training is available for the products in the vendor’s functional testing solution?

Who are the vendor's partners for its functional testing solution?(Technology partners, systems integrators, channel partners)

Partnerships


Agenda






Findings

Empirix 2.11

Vendor

Mercury

IBM

Compuware

Borland

Ranking

4.39

3.23

2.65

2.64

Overall rankings

(50% current offering, 50% strategy)

One leader

Three strong performers

One contender


How to create a custom ranking

• Determine how much each of the evaluation criteria matters to you.

• Weight the evaluation criteria accordingly.

• Read the score explanation text to familiarize yourself with these tools and vendors.

• Follow up with demos, trials, and pilots.

© 2006, Forrester Research, Inc. Reproduction Prohibited

Forrester Wave™: Functional Testing Solutions, Q2 ’06May 2006, Tech Choices “The Forrester Wave™: Functional Testing Solutions, Q2 2006”


Vendor profile: Borland

• Strengths:

» Life-cycle integration

» Reporting and analytics

• Weaknesses:

» Manual testing

» Automated test script creation

» Environment support

• Best for:

» Shops with testers who have programming skills.

» Shops that use other Borland life-cycle management products


Vendor profile: Compuware

• Strengths:

» Overall breadth — though not depth — of capabilities

» Built-in support for risk-based testing

• Weaknesses:

» Weak support for hand-coding and graphical modification of test scripts

» Core test management capabilities available only to CARS customers

» Too many disparate interfaces

• Best for:

» Project-level testing efforts

» Shops that use other Compuware products


Vendor profile: Empirix• Strengths:

» Strong support for Web environments

» Specialized support for Web services

» XML-based APIs

• Weaknesses:

» e-Tester has extremely limited environment support

» e-Tester does not serve technical or nontechnical testers well

» e-Manager Enterprise has minimal support for manual testing

» e-Manager Enterprise offers only basic capabilities when it comes to test management

» The solution as a whole falls down on life-cycle integration

• Best for:

» Project-level testing efforts

» Web applications and services testing efforts


Vendor profile: IBM• Strengths:

» Support for manual testing

» Support for custom coding of test scripts

» A platform for test management

• Weaknesses:

» Nonprogrammers don't get much help on test automation

» Environment support is still limited, although it is improving

» Test execution capabilities are primitive

» The functional testing solution itself is in need of better integration

• Best for:

» Using other IBM Rational tools

» Doing a great deal of manual testing

» Having testers with programming experience


Vendor profile: Mercury

• Strengths:

» Enhanced user productivity through ease of use

» Top-notch environment support

» Proven scalability across multiple dimensions

• Weaknesses:

» Weak scripting language and scripting environment

» Limited management of changes to reused manual test components

» Corporate instability

• Best for:

» Centralized testing organizations

» Companies that use other Mercury products


Agenda






Factors to consider when selecting a functional testing solution

• Application technologies in use

» Legacy 4GL, Web services, ERP/CRM, custom controls

• Skill sets

» Strong knowledge of the business, programming experience and/or aptitude

• Organizational structure

» Centralized test organization, testers on development teams, offshore testing

• Development life-cycle tools in use

» Tools for developer testing, requirements definition and management, issue management, software configuration management

• IT operations tools in use

» Tools for deployment, performance monitoring, SOA management

• IT management tools in use


How will vendors improve their offerings?

• Better enable incremental automation of manual test cases

• Provide better facilities for graphical creation and modification of test cases

• Improve support for testing in an SOA environment

• Do more to facilitate geographically distributed testing efforts

• Improve integrations with development, operations, and management tools

• Continue to explore open standards


The next area of innovation: SOA testingJuly 2006 (Upcoming) Trends “SOA Raises The Stakes For Software Quality”


Manual and automated functional testing are good candidates for offshore outsourcing

March 2006, Trends “How To Benefit From Offshore Testing Services”


Carey Schwaber

+1 617/613-6260

[email protected]

www.forrester.com

Thank you


Selected bibliography

• May 31, 2006, Tech Choices “The Forrester Wave, Q2 2006: Functional Testing Solutions”

• March 8, 2006, Trends “How To Benefit From Offshore Testing Services”

• September 15, 2005, Quick Take “Take Careful Inventory Before Adopting Standalone Code Quality Tools”

• May 16, 2005, Best Practices “Software Quality Is Everybody’s Business”

• February 3, 2005, Tech Choices “Evaluating Automated Functional Testing Tools”

Documents

Evaluating Functional Testing Solutions