Market Guide for Mobile Appl 273008

8/17/2019 Market Guide for Mobile Appl 273008

1/33

This research note is restricted to the personal use of [email protected]


G00273008

Market Guide for Mobile Application Testing

ServicesPublished: 29 October 2015

Analyst(s): Susanne Matson, Maritess Sobejana

Improving customer experience in a digital economy is critical to success,

driving organizations to invest in mobile application development. Sourcing

managers must understand the complexity of the mobile testing

requirements to ensure service providers can meet business expectations.

Key Findings■ The rapid developments of mobile and other digital technologies is necessitating agile and

DevOps methodologies, requiring organizations to reskill and maintain the relevant skills in-

house. However, lack of available skills in the market is driving higher usage of external service

providers since there are multiple skills needed for mobile development and testing — plus the

requirements constantly change.

■ Mobile testing services are inherently different to traditional application testing services, as they

require added complexity, such as the testing of multiplatforms, networks, OS, as well asdifferent devices. User experience is also key in mobility as it may generate unique requirements

for the testing process as well as considerations for testing by geography and security testing.

■ Lack of forward planning and engaging sourcing as well as IT, proactively drives the business to

bypass quality test planning to meet time-to-market needs. This results in higher risk with a lack

of testing metrics in the contract in order to hold service providers accountable.

Recommendations

Sourcing managers should:■ Include experience in agile and iterative development methods as key selection criteria when

choosing mobile application testing service providers, since it requires a different skill level of

resources than traditional testing services. Due to high collaboration, a nearshore or on-site

presence is necessary to be fully effective.

■ Document the business requirements — understanding the bimodal aspect of the development

and testing required. Since mobile application testing is part of both Mode 1 and Mode 2,


2/33



addressing both the need for stability as well as agility. This may impact the selection criteria as

well as the provider experience desired.

■ Engage with the business early in order to develop an efcient testing strategy. Avoid risk of

business bypassing in accessing these applications without sufcient planning for testing

quality — in lieu of speed.

Market DenitionGartner's 2015 CIO survey revealed that currently, mobile ranked as the fth highest priority for CIO

spend. Naturally, ensuing testing demands must follow. However, there is lack of awareness by

clients on the complexity of testing required in mobile development, as well as how it differs from

traditional application testing, so it makes it difcult to select the right service provider.1

Mobile testing services are different, often requiring more complex than traditional application

testing services since they require new skill sets and considerations — such as the testing ofmultiplatforms, networks, OS as well as different devices. These services are needed to test the

function, performance, compatibility and so forth, of a mobile application — inclusive of both

manual and automated tests. Therefore, internal testing organizations need to consider mobile

application testing service providers as part of their quality strategy.

User experience is key in mobility by creating unique requirements for the testing process. With

digital business and customer experience, applications and data is increasingly available and visible

to external constituents — especially customers. Thus, quality, reliability, security and adaptability

needed for external use of systems is far beyond internal usage — and this is especially true with

mobile applications. The competitive landscape of mobile application testing service providers

comprises a variety of traditional and nontraditional players including:

■ Full service IT service providers

■ Pure-play testing or mobile testing service providers

■ Digital that offers mobile testing

■ Testing specialists who specialize in mobile testing entirely in the cloud

■ Crowdtesting providers

Buyers of mobility testing services range from business leaders, sourcing managers, application

development managers and testing managers. Services to support clients' mobile application,quality control and quality assurance to ensure the high quality demanded by consumers, clients,

stakeholders and end users are highly requested by buyers today. This research includes all test

consultancy, advisory, implementation, execution and management services related to mobile apps,

we also examine how it functions on various devices.

For the purpose of this Market Guide, we do not include crowdtesting service providers. However,

we recognize that this can be effectively used for mobile testing and these providers are covered in

Page 2 of 33 Gartner, Inc. | G00273008


3/33



"Market Guide for Crowdsourced Application Testing Services." Additionally, testing product

companies and providers are covered in "Market Guide for Mobile Application Testing."

Market Direction Application development and business managers (sales, marketing operations that include mobile

applications) are facing pressure to ensure the consistent high quality and user experience of mobile

applications. Mobile is one of the key technologies encompassed within digital business initiatives;

where demands of the business include shorter time to market, keeping costs low in often complex

testing environments along with maintaining the high levels of quality, are top of mind for business

leaders. Financial losses, legal issues and irreparable brand image would be a high price to pay if

mobile application fails to deliver the outcomes expected.

Since mobile applications drive digital business and success is determined by user experience, they

are generally led and adopted by business units rather than the IT department. IT and sourcing

therefore, needs to be fully engaged and be knowledgeable about the requirements of different

business units and their needs in order to deliver high-quality mobile apps. Thus, this entails

working closely with the business units on mobile testing services demands — which is

continuously challenging traditional IT management strategies — as it requires agile and exibleorganizations.

This market is developing rapidly. Currently, it is still fairly fragmented and unstructured, which

means more potential confusion for buyers. The need to support these applications is quite new so

the capabilities to manage the required testing are not often present within the organizations

internally. Anticipating further market maturity, there will be many tactical relationships forming to

meet intense demand.

Mobile Application Testing Tools Market Will Drive Use of External Testing Services

The mobile tools market is expected to rapidly grow with new entrants continuing to appear with

varying product offerings that will inuence the services market over the subsequent ve-year

period (see "Market Guide for Mobile Application Testing"). The reality is that before the market even

reaches maturity, the initial confusion and strong ramp-up in demand is mixed with the need for

essentially new skills and capabilities. Looking even further into the future, mobile technologies (and

how they are integrated) are likely to continue evolving rapidly, and will be inuenced by other

technologies such as the Internet of Things (IoT), beacons, wearables, smartwatches and machine

learning. These aspects will drive increased reliance on service providers. Typically, this is the case

with any emerging market.

Lack of Digital Skills Is a Major Incentive to Consider Mobile Application TestingServices

The investments IT service providers make into digital technologies and in continuously training

their staff is a key driver for buying mobile application services. Since digital skills such as mobile

application development and testing require updated products and apps, it needs ongoing

Gartner, Inc. | G00273008 Page 3 of 33


4/33



investment to keep talent and technology refreshed frequently — simply due to the sheer speed of

innovation.

Test automation skills are becoming more essential alongside digital business. The speed with

which development of digital technologies occur exposes a new reality for organizations, whereby

they will not be able to reskill or retrain fast enough — there are simply too many skills required thatgo through constant changes. Speed is also accelerating, so the time to market for a mobile project

can be a matter of days or weeks, as opposed to legacy applications in the traditional sense.

Organizations need to make a choice about whether they are willing or able to maintain all these

skills in-house. The market broadly suffers from a shortage of various specialized mobile application

testing skills, which proves a signicant challenge — even for service providers. However, because

this is their business, they make focused investments in these areas placing them in a better

position to train, maintain and acquire the right skills in comparison with internal testing

organizations.

Bimodal IT Will Drive the Need for AgilityMobile technologies will evolve rapidly and principles such as bimodal IT will help to facilitate

enterprise agility. These constant changes, both at the technology and consumer level, places

signicant demands on organizations, which make dening a mobile strategy difcult. Gartner has

covered the area of bimodal in research noting that there are two modes to consider — Mode 1 and

Mode 2 — which, when combined, serve to address both the stability and the agility requirements

within organizations.

As organizations develop bimodal IT capabilities, they will need to form application development

teams with different objectives, processes, metrics, disciplines, and consequently, varying needs in

order to support both Mode 1 and Mode 2 projects. In mobile application development, Mode 1

refers to a stable infrastructure (API, mobile app integration layer) where mobile apps can beintegrated with back-end systems; Mode 2 refers to the front-end client of the app (app UI and user

experience [UX]) that the user interacts with.

Mode 2 includes more emerging areas of interest to the organization in terms of new business

development with quicker turnaround times, agile approaches, new service methods and delivering

front-end applications. Therefore, a different approach to the various providers needed in the

sourcing mix is likely to be necessary.

We often nd providers moving into offering specialized services or modes of delivery, as well as

more exibility and ease of use. Application development and sourcing managers need to be aware

of the new opportunities and providers in this emerging market, as well as know how best to workwith these providers and also obtain services that cater for the varying needs of the organization.

Mobility itself is also constantly changing, at both the technology level and the societal level, with

the resulting demands on organizations. This uid environment makes dening a mobile strategy

complex. How can the conicting needs of enterprise stability and rapid response be resolved? The

use of a bimodal strategy — one mode addressing stability and the other agility — allows

organizations to maximize development efciency and respect information control policies, yet

addressing the urgency of volatile, competitive, seasonal and unplanned requirements.



5/33



Adoption of Agile Methodologies Require More Automation

Developing bimodal IT capabilities demand that IT organizations begin to deal with agile methods

for application development and delivery. Agile methodologies include frequent meetings between

the development team and application team, with much shorter cycles for updates and revisions.

This results in the need for increased levels of automation and tooling support, to achieve therequired speed of delivery. However, automation requires different tools and frameworks as well as a

different skill set to deliver manual execution. Moreover, initial cost for automation is high —

licensing cost, training, creation of scripts, maintenance, and so on. The automation tools

developed for traditional testing services might not work for mobile application testing, so providers

are building new or extending the tools they already have, to increase the level of automation —

leading to accelerated testing, increased efciency and reliability of testing. Automation tools are

frequently requested by clients, so creating a portfolio of strong offerings can prove to be the

differentiator for providers. Resources provided by the service provider must also be able to work in

an agile, dynamic and uid development environment. This is a completely different way of working

with service providers. Shorter sprints and shorter cycles, as well as a much more collaborative

working methodology mean different requirements of service providers. With an agile approach, it isnot best practice to use service providers for all of the work — it helps to have internal quality

sponsors that will oversee the projects. Effective engagement is dependent upon collaboration

between business leaders, application developers and testers. Working together, especially on the

same site, often allows for face-to-face collaboration that reduces misinterpretations of

requirements and enables testing during development. Having project managers on the team who

can lead and interpret results is essential.

Market Analysis

With the expansion of new technologies such as mobile being further integrated into organizationsthat are forming digital business opportunities, there are multiple challenges with organizations

needing to ensure that their mobile applications are of good quality.

Testing Mobile Applications Is Far More Complex Than Traditional ApplicationTesting

The inherent complexities in testing mobile applications are compounding the challenges to

maintain quality assurance. When testing mobile applications, the process must cover multiple

platforms, multiple networks of varying stability, different OS including frequent OS upgrades,

including the different devices the apps are running on (devices like Samsung Electronics, Nokia,

Apple and HTC). All these options and combinations require a different view. A mobile application ispart of a system, for example, mobile payment that may involve the application of a Near Field

Communication (NFC) reader and a point of sale (POS) system. Furthermore, a prominent testing

requirement in the mobile applications space is user experience. Mobile is increasingly a way for

organizations to communicate with clients, win business in new areas or reach various stakeholders

— so user experience is paramount. Therefore, usability and exibility of mobile applications is top

of the agenda. In the future, the user experience will likely become even more central, with the

devices and services less so. Further considerations such as security (like testing the robustness of



6/33



data protection solutions) are potentially even more important for mobile applications, since user-

facing components of systems will move increasingly outside of the rewall. In addition,

geographical and language aspects are elements that also require contemplation in the testing of

mobile applications.

To effectively test mobile applications, clients are using various methods, such as creatingcompetency centers for critical platforms (iOS, Android), carriers (Verizon, BT), as well as security

and digital design. Clients also use mobile testing service providers to complement internal

capabilities and/or crowdsource critical missing competencies.

Fundamental Changes in Engagement Models and Pricing

Due to the nature of mobile applications — in terms of shorter time to market, higher speed, smaller

projects and more collaborative, agile work — it can be more effective to use a combination of

services, techniques and providers. The RFI and RFP processes for traditional application testing

projects are often too long, therefore quick decisions frequently need to be made. Also, exibility in

contracting is likely more important when negotiating with mobile testing pure-play rms or digitalagencies. For guidance on some of the key criteria that can be used when selecting a mobile

application testing service provider, see "How to Choose a Mobile Application Testing Service

Provider."

An increasing number of smaller, more nimble providers are moving into this space and can often be

very effective as part of a sourcing strategy. These providers come from varying backgrounds and

can be good, exible, quick, cost-effective and skillful additions. However, note the variety of

provider and carefully consider which type of provider would make the most suitable t and what

they specialize in.

The use of on-site or onshore resources as part of the sourcing mix is also essential to ensuresuccess. Agile and DevOps methodologies (with the speed generally needed in these projects),

have a positive effect on service providers by winning deals that have sufcient resources closer to

the client. As service providers and clients become more experienced by working on agile projects,

offshore usage may be increased over time to control costs. As a result, we see service providers

creating more delivery centers closer to the client (if needed) and testing professionals on-site,

embedded within project teams with utilization of offshore resources.

Cloud is a model frequently considered for mobile application testing. Here devices can be run on

multiple systems in the cloud and tested by creating a Web-based mobile environment on a

simulator to access the mobile app. Public cloud is of interest to many clients that do not mind the

potential lack of control or privacy issues, while data-privacy sensitive clients will prefer the privatecloud. The cloud can be easily accessed; it is cost-efcient and scalable, fast and effective.

The exibility in pricing and scalability that testing-as-a-service (TaaS) represents is frequently used

for mobile application testing and as long as it is clear what the client is paying for, this can prove to

be a good alternative.



7/33


8/33


9/33



Provider/Website Category Head-

quar-

ters

Client Segments Industries (Primary Focus as Indicated by

Provider)

Partnerships

■ Media

■ Professional services

■ Retail

■ Sports and entertainment

■ Telecommunications

b-acceptance

( www.b-accept-ance.com )

Pure-play test-ing serviceprovider

France Focuses on allsegments

■ Consumer products

■ Healthcare

■ High tech

■ Media


■ Retail and wholesale distribution

■ Ranorex

■ TestLink

■ HP Quality Center

Capgemini

( www.capgemini.com )

Global full-service provid-er

France Primary focus is2,500 employeesand above

■ Banking

■ Insurance

■ Communications

■ Other nancial services

■ Public sector

■ HP

■ Perfecto Mobile

■ DeviceAnywhere

■ Appium

■ Mobile Labs

CGI

( www.cgi.com )


Canada Primary focus is2,500 employeesand above

■ Financial services

■ Communications

■ Public sector

■ HP

■ Perfecto Mobile

■ Keynote

Gartner, Inc. | G00273008 Page 9 of

http://www.cgi.com/enhttps://www.capgemini.com/http://www.b-acceptance.com/http://www.b-acceptance.com/


10/33




quar-

ters


Provider)

Partnerships

■ Healthcare

■ Energy

Cigniti Technologies

( www.cigniti.com )

Pure-play test-

ing serviceprovider

India Primary focus is

organizations be-tween 1,000 to2,500 employees

■ Banking

■ High tech

■ Insurance


■ Retail

■ Travel and transportation

■ Experitest (SeeTest)

■ Ranorex

■ TestPlant

■ Neotys

■ Perfecto Mobile

■ Mobile Labs

■ HP

■ Soasta

Cognizant

( www.cognizant.com )


U.S. Primary focus is2,500 employeesand above

■ Automotive

■ Banking


■ Healthcare

■ Higher education

■ High tech

■ Insurance

■ Life science

■ Media

■ Experitest

■ Mobile Labs

■ Perfecto Mobile

■ Applause


http://www.cognizant.com/http://www.cigniti.com/


11/33




quar-

ters


Provider)

Partnerships

■ Retail




■ Utilities

CresTech Software

Systems

( www.crestechglo-bal.com )


India NA ■ Banking

■ Insurance

■ Media and entertainment


■ Travel

■ Amazon Web Serv-

ices

■ Armorize Technolo-

gies

■ IBM

■ Orane Consulting

CSC

( www.csc.com )



■ Healthcare

■ Insurance

■ Banking

■ High tech

■ Public services

■ Manufacturing

■ Transportation

■ Retail

■ Perfecto Mobile

■ Soasta


http://www.csc.com/http://crestechglobal.com/http://crestechglobal.com/


12/33




quar-

ters


Provider)

Partnerships

Deloitte

( www.deloitte.com )




■ Public

■ Healthcare

■ Manufacturing

■ Communications

■ Mobile Labs

■ Perfecto Mobile

Epam

( www.epam.com )

Regional midt-ier full or sub-set IT serviceprovider


■ Banking


■ Healthcare

■ High tech

■ Life sciences

■ Media

■ Retail



■ Experitest

■ Xamarin

■ Perfecto Mobile

■ Selenium (open

source)

■ Appium (open source)

HP

( www.hp.com )


U.S. Focuses acrossseveral ranges

■ Healthcare

■ Public sector

■ Communications

■ Media and entertainment

Perfecto Mobile


http://www8.hp.com/us/en/home.htmlhttp://www.epam.com/http://www2.deloitte.com/global/en.html?icid=site_selector_global


13/33




quar-

ters


Provider)

Partnerships

■ Transportation

IBM

( www.ibm.com )


U.S. Primary focus isorganizations withmore than 10,000

employees

■ Banking


■ Healthcare

■ Insurance

■ Media

■ Retail




■ Perfecto Mobile

■ IBM Rational

■ DeviceAnywhere

InfoStretch

( www.infostretch.com )

Mobile serviceprovider

U.S. NA NA ■ Kony

■ Perfecto Mobile

■ Sauce Labs

■ Soasta

■ Mobile Labs

■ PhoneGap

■ Atos

■ Veracode


http://www.infostretch.com/http://www.ibm.com/


14/33




quar-

ters


Provider)

Partnerships

■ Xamarin

Infosys

( www.infosys.com )


India Primary focus is2,500 employeesand above

■ Automotive

■ Banking


■ Healthcare

■ High tech

■ Industrial machinery

■ Insurance

■ Life sciences

■ Oil and gas


■ Public sector

■ Retail



■ Utilities

■ Experitest

■ Mobile Labs

■ HP

■ Perfecto Mobile

■ Jamo Solutions

■ Sauce Labs

Kualitatem

( www.kualitatem.com )


U.S. Primary focus isless than 500 em-ployees

■ Banking

■ Healthcare

■ HP

■ IBM

■ Selenium


http://www.kualitatem.com/http://www.infosys.com/


15/33




quar-

ters


Provider)

Partnerships

■ Media

NTT Data

( www.nttdata.com )


Japan Evenly spreadfrom organizationswith 500 employ-

ees, but most or-ganizations havemore than 10,000employees

■ Healthcare


■ Insurance


■ Perfecto Mobile

■ DeviceAnywhere

■ Jamo Solutions

Olenick & Associates

( www.olenick.com )


U.S. NA ■ Banking

■ Capital markets

■ Healthcare

■ Utilities

■ HP

■ Microsoft

■ IBM

■ Soasta

■ Perfecto Mobile

■ Tahoe Partners

QASource

( www.qasource.com )

Pure-play test-

ing serviceprovider

U.S. Primary focus is

up to 2,500 em-ployees

■ Banking


■ Engineering

■ Construction and operations

■ Healthcare


■ High tech

■ Insurance

QASource is tool-agnostic, al-

though it works with multiple part-ners


https://www.qasource.com/http://www.olenick.com/http://www.nttdata.com/


16/33


17/33




quar-

ters


Provider)

Partnerships

■ Manufacturing ■ Keynote Mobile Test-

ing

RapidValue Solutions

( www.rapidvaluesolu-

tions.com )

Regional midt-ier full or sub-

set IT serviceprovider

U.S. Primary focus isorganizations with

either less than500 or between1,000 and 2,500employees

■ Banking

■

Consumer products

■ High tech

■ Insurance

■ Media

■ Oil and gas


■ Retail


■ Appium

■

Perfecto Mobile

■ SeeTest

■ BlazeMeter

■ SoapUI

SLK Software

( www.slk-group.com )

Regional midt-ier full or sub-

set IT serviceprovider

India Primary focus isorganizations with

more than 1,000employees

■ Automotive

■

Banking■ Industrial machinery

■ Insurance

■ Retail

■ Utilities

■ Wholesale distribution

■ Perfecto Mobile

■

Appium■ HP UFT


http://www.slk-group.com/http://www.rapidvaluesolutions.com/http://www.rapidvaluesolutions.com/


18/33




quar-

ters


Provider)

Partnerships

StarDust

( www.stardust-test-ing.com )


France Primary focus isorganizations withmore than 2,500employees

■ Banking

■ Healthcare

■ Insurance

■ Life sciences

■ Retail


■ Wholesale distribution

NA

SQS

( www.sqs.com )


Germa-ny

Primary focus is2,500 employeesand above

■ Banking

■ Other nancial services

■ Insurance

■ Automotive

■ High tech

■ Services

■ Other manufacturing

Perfecto Mobile

Syntel

( www.syntelinc.com )


U.S. Primary focus isorganizations withmore than 10,000employees

Banking ■ Kony

■ Microsoft

■ Salesforce

■ Experitest


http://www.syntelinc.com/http://www.sqs.com/en-group/index.phphttp://www.stardust-testing.com/http://www.stardust-testing.com/


19/33


20/33




quar-

ters


Provider)

Partnerships

Virtusa

( www.virtusa.com )


U.S. Primary focus isorganizations withover 10,000 em-ployees

■ Automotive

■ Banking

■ Healthcare

■ Insurance

■ Media


■ Experitest

■ HP Mobile Center

■ Appium

Wipro

( www.wipro.com )


India Primary focus isorganizations withmore than 10,000employees

■ Retail

■ Banking

■ Insurance

■ Healthcare

■ Mobile Labs

■ Perfecto Mobile

ZenQ

( www.zenq.com )


India Primary focus isorganizations withless than 500 em-

ployees

■ Banking


■ Healthcare

■ Insurance

■ Retail

■ Appium

■ Silk Mobile

■ pCloudy

■ Keynote

NA = not applicable

Source: Gartner (October 2015)


http://zenq.com/http://www.wipro.com/http://www.virtusa.com/


21/33



Global Full-Service Providers

Full service providers offer various IT services including testing services practice with the mobile

application testing capabilities as a subset of that practice. Mobile testing is an area of strong

demand in the market and most of the full-service providers of a certain size offer mobile testing

services. (See Table 1 for further examples of providers.) Buyers can consider using these providersfor larger, complex or global deals.

Atos

www.atos.com

Atos is a global service provider that has a testing practice with over 8,000 resources. Atos also has

a focused mobile application testing practice with just under 800 resources. Atos has most of its

regional capability based in Europe (U.K., Belgium, the Netherlands and Luxembourg [Benelux],

Spain, France), although an increasing part of its revenue stems from North America and various

other countries. More strategic focus has been placed on the Atos testing practice over the pastcouple of years, with increased investment into the practice, its rebranding into an independent unit

and a raised prole within the organization. Mobile testing is one of the areas of innovation that the

provider stated it wanted to focus on from 2015 onwards, as it leverages its global strategic

partnership with InfoStretch (a mobile application service provider that is detailed in the Mobile

Service Providers section, it delivers mobile application testing services). Atos has a mobile testing

center of excellence (COE) in India and specialized smart mobility delivery centers. Atos mobile

testing services include end-to-end automated mobile testing solutions, as well as context-driven

mobility services.

Cognizantwww.cognizant.com

Cognizant is a global service provider with a large testing practice, which includes mobile

application testing services. Cognizant has an India-centric delivery model, with most of its revenue

originating in its home market of North America but also offers services in multiple other regions and

countries.

Cognizant continues to move further into the areas of digital and mobile by leveraging its

technology labs, assets, intellectual property (IP) and partner ecosystem. Currently, the provider has

10 global labs focusing on mobile activities located in the U.S., China, India, U.K. and a few other

countries in Europe and Asia/Pacic. Cognizant offers an end-to-end mobile testing solutionleveraging internal developed IP, including automation for mobile testing via its "mint" platform and

"BlueSim," a solution for connected device virtualization using Bluetooth technology.

Examples of testing services offered include functional testing, automation testing, eld network

testing, user experience testing, nonfunctional testing, security, cross browser/platform capability

testing and application validation.


http://www.cognizant.com/http://www.cognizant.com/http://www.atos.com/


22/33



IBM

www.ibm.com

IBM is a full service global provider with various service areas, including testing and mobile

application testing services. IBM is a global provider that offers local support as well as globaldelivery centers. Mobile application testing is one of the current strategic focus areas of IBM. It uses

its portfolio of applications to support clients — from consulting and engagement through to IBM's

mobile test factory. IBM has developed mobile test facilities across India, Europe, U.S. and Asia/

Pacic, where it performs testing on multiple browsers, devices and platforms, including offerings

such as functional, performance and network testing services. IBM leverages its Combinatorial Test

Design (CTD) solution for reducing mobile test cases in the mobile test centers and it has also

worked with one of its partners, eggPlant, to develop a mobile automation solution.

An example of a recent initiative within the portfolio of mobile testing services includes its Apple

Garage, in partnership with Apple. The Apple Garage, which includes testing automation, integrates

IBM and Apple tools for continuous testing built on agile practices.

Infosys

www.infosys.com

Infosys is a full service global provider with a testing practice, as well as mobile testing capabilities.

Infosys includes just under 700 resources in its testing practice that are dedicated to mobile

application testing services. Infosys has extensive offshore centers which it leverages for clients

primarily based in North America, Australia and Europe — although it can also offer on-site or

nearshore resources where required. Within the service offerings of digital assurance, it offers

mobility quality assurance (QA) services, including mobile device testing, mobile app QA and mobile

QA consulting. These services comprise offerings such as functionality, platform integration,

performance, localization, interoperability, usability, security, UX benchmarking, network and

automation testing services or provide support around creating a mobile test strategy. Within its

mobile testing practice, Infosys has invested to accelerate and improve its effectiveness on client

engagements by creating various tools and frameworks. These include a mobile QA defect

repository tools, Mobile Scenario Testing Rapid Prototyping Engine (Mobile STRiPE) for early

validation; mobile application performance monitoring (mAPM) tool for device side performance

monitoring; network impact testing solution (iNITS) for testing mobile applications under varying

network conditions; a mobile platform specic validation framework as well as "iMobiLoad," a

performance QA tool and "CARES" a user experience test framework. Infosys also includes the

offering, Test Unit (TU)-based charging model, and another offering, integrated mobility-testing lab

on cloud, which can be accessed from customer locations worldwide.

Tata Consultancy Services

www.tcs.com

Tata Consultancy Services (TCS) is a global full service provider. It has one of the largest testing

practices in the market and its portfolio also includes mobile application testing services with

approximately 1,600 resources. TCS has recently scaled up and reinvented its digital assurance


http://www.tcs.com/Pages/default.aspxhttp://www.infosys.com/http://www.tcs.com/Pages/default.aspxhttp://www.infosys.com/http://www.ibm.com/


23/33



offerings — including mobile — by introducing specialization skills into its practice for mobile

testing, investing in a customer collaboration center at Santa Clara and a customer experience

center at Chennai Lab, and also building partnerships.

TCS incubates startups and partnerships in the mobile ecosystem and uses its dedicated mobile

testing labs in India and U.S. to train its workforce, support delivery, handle proof of concept (POC)programs and actually show customers how they test and assure in this digital space. Services are

delivered in traditional models as well as mobile-testing-as-a-service.

TCS has created mobility frameworks, other tools and IP to support delivery in this practice. Mobile

Reusable Automation Framework for Testing (mRAFT) is a framework for mobile test automation.

"iBrowse" provides automated cross-browser compatibility testing and result verication across

mobile browsers, it groups test cases based on business functions while "TCS Hy5 Test," which is

part of the TCS Hy5 Suite, enables the development, testing and deployment of applications on a

variety of mobile platforms through automation.

TCS offers assurance services on various mobile platforms, devices and applications, which includeservices such as network/environment virtualization, automation, performance, security,

accessibility, functional and nonfunctional testing on and off the cloud.

TCS primary focus is end-to-end mobility assurance engagements. Typically, TCS's engagements

span all aspects of assuring the mobile business, although it only addresses certain parts of the

mobile application testing life cycle in certain engagements.

Tech Mahindra

www.techmahindra.com

Tech Mahindra is a full service provider with a mobile application testing practice.

This company uses both on-site delivery along with its mainly Indian global delivery centers to

support its clients. Most of Tech Mahindra's revenue originates in North America but it also has a

large client base in some countries in EMEA. Its mobile application testing practice in Europe is, in

fact, its largest in terms of revenue. Mobile application testing services operate in a center of

excellence with 60 dedicated resources. The company has developed an automation framework,

"Mobile Application Testing Ecosystem" (MATE) that focuses on cloud-based automation-testing

services. There is a dedicated lab available for testing with an inventory of mobile devices. The

practice has capabilities for functional, automation, localization, user experience, compliance,

performance, security and cloud enabled testing.

Regional Midtier, Full or Subset IT Service Providers

These are smaller service providers with testing service capabilities including mobile. They offer

services in one region, one country and/or with a more specialized services portfolio. This group of

providers can be a good option both for small and midsize organizations looking for a local deal, but

some of these providers are also large enough to manage those bigger deals.


http://www.techmahindra.com/pages/default.aspx


24/33



Altimetrik

www.altimetrik.com

Altimetrik was founded in 2012 and is part of the Vattikuti Ventures. Altimetrik provides business

transformation and technology solutions for enterprises in areas of digital, connected solutions andconsumer technologies through ve capability centers in the following areas, user experience and

design, prototyping, application development and integration, information engineering and

Visualization, quality engineering and automation and DevOps. The provider employs 240 of its

engineers in its mobile application testing practice and draws most of its revenue from North

America, despite its presence in multiple other regions.

Its Quality Engineering Automation Capability Center, is where its mobile testing service resides,

using its delivery centers in India and Uruguay to deliver mobile testing solutions. Altimetrik has a

cloud-based mobile device lab called "Appcentric," a framework that connects to an array of real

mobile devices enabling remote collaboration and test automation. Altimetrik has also developed an

automation framework enabling scriptless, domain-specic test automation called "Framework forEarly Automated Testing" (FEAT). Altimetrik provides engagement models including on-premises,

cloud hosted and subscription-led offerings.

Epam

www.epam.com

Epam is an IT service provider with application development as its core competency.

The company has around 13,600 employees with 2,200 of those based in its testing practice, where

mobile application testing services is part of the portfolio. Most of its consultants are based in

Eastern Europe although it has been increasing its footprint in the North America and Asia/Pacicregions, as part of its efforts to make the company more global. Meanwhile approximately 50% of

its revenue is derived from the U.S. with the rest originating from Europe. A large portion of its

revenue is in mobile application testing, stemming from North America and Western Europe. Both

its testing practice and automation are areas of growth — and its main focus. It has offerings

including mobile test automation and mobile performance testing. Epam offers mobile testing

services as an end-to-end product, a quality engineering service that includes automation solutions,

specialized security testing, mobile performance, usability and user experience assessment.

Competency centers, established methodologies, tools and accelerators support these offerings.

Epam leverages a unied automation framework, which is reusable between Web and mobile in its

testing practice.

Olenick & Associates

www.olenick.com

Established in 1998, Olenick & Associates concentrates on offerings within software development,

including testing. The company primarily focuses on North America with delivery centers in Latin

America and Northern Ireland. Olenick & Associates has created a mobile test lab that it can use to

test applications across multiple mobile devices. It provides functional testing services to clients as


http://www.olenick.com/http://www.epam.com/http://www.olenick.com/http://www.epam.com/http://www.altimetrik.com/


25/33



well as manual testing and test automation. For test automation, it utilizes open-source functional

testing tools; and for mobile user experience testing, the provider has created an internal

framework.

RapidValue Solutionswww.rapidvaluesolutions.com

RapidValue Solutions is a provider focusing on the digital transformation for the enterprise along

three dimensions — mobile, omnichannel and cloud. Having an end-to-end approach is the primary

business, but the provider has established a dedicated sales team bringing in projects specically

for mobile application testing services. RapidValue Solutions employs around 400 resources in total,

of which, 72 are based in the mobile testing practice. The company has ofces in the U.S. and

India; it is looking to expand in the European market by opening its rst ofce in the U.K. The

company follows the scrum development model where the testing team is part of the development

team during the entire project development stages. It also has a dedicated mobile application-

testing center (DMTC) and a dedicated lab with devices, simulators and automation testing tools.The testing portfolio of RapidValue Solutions includes test automation, functional and user

acceptance, performance, usability and security testing.

SLK Software

www.slk-group.com

Part of the SLK Group, SLK Software is a service provider with around 1,500 employees. SLK

Software offers mobile application testing as part of an overall testing service, as well as a discrete

mobile application testing services practice with 50 resources. The company offers testing services

through different engagement models including test automation and performance. SLK Softwareprovides TaaS with an outcome-based pricing model, leveraging its own automation solution and its

own private mobile cloud solution.

StarDust

www.stardust-testing.com

StarDust engages with its clients throughout the process of creating and launching their digital

solutions, including mobile and websites. The mobile application testing practice comprises 60

resources and most of its revenue originates from North America with some clients in EMEA also.

The company provides testing of mobile applications — such as bug tracking or checking if themobile website is responsive to different types of media and devices. It also operates upstream of

the production, both in the design and development phases of project. StarDust works with clients

ensuring that they comply with current technical Web and mobile standards as well as meeting any

functional requirements. The company provides the following types of testing services — functional,

performance, security, location, user experience and automation.


http://www.stardust-testing.com/?lang=enhttp://www.slk-group.com/http://www.rapidvaluesolutions.com/


26/33



Syntel

www.syntelinc.com

Syntel is a full service provider including mobile application testing services in its portfolio of

services. Syntel serves clients through its mobile test center of excellence (COE) where it also offerscloud-enabled mobility testing services. Syntel has just over 24,000 employees with 1,300

resources in mobile application testing services. Syntel has developed adaptable test

methodologies for mobile applications and offers both manual and automated testing. Syntel has a

private cloud mobility-testing lab where it leverages its mobility test automation framework (MTAF).

It is built on top of open stack where vendor-specic products can create mobile test automation

suites. It also includes lab-as-a-service (LaaS) into its engagement model providing access to its

private lab. Syntel offerings include functional application testing services, performance testing, test

automation services, user experience testing and security.

Syntel's test lab has multiple devices, emulators and tools. One example is the in-house developed

cross-browser, Test Automation Framework xWAT for testing browser compatibility on both iOS and Android platforms.

Virtusa

www.virtusa.com

Virtusa is an IT services company providing IT consulting, technology and outsourcing services,

with just under 10,000 resources in total. The company was founded in 1996 with ofces and

technology centers throughout the U.S., Europe and Asia. Virtusa uses its global delivery model to

provide services to its clients, which includes testing services. In testing, the company has 148

dedicated resources focusing on mobile application testing services where it utilizes its mobility lab

and test automation frameworks to serve clients. Mobile testing services ts into its managed

service test COE; it also offers a mobile subscription service and a TaaS test factory service for

mobile automation and execution.

Virtusa leverages its test automation platform "centroid," enabling end-to-end digital test

automation.

Pure-Play Testing Service Providers

Pure-play testing service providers gain the vast majority of their revenue from testing services.

These rms can have various specializations, but all include mobile application testing within their

testing portfolio. Buyers can consider these providers for specialist skills, smaller projects, country-specic knowledge or for their exible and nimble approach.

b-acceptance

www.b-acceptance.com

Founded in 2011, b-acceptance is a pure-play testing rm with 35 resources focused on mobile

application testing services; mainly focusing on the digital world (media, m-commerce, e-


http://www.b-acceptance.com/http://www.b-acceptance.com/http://www.virtusa.com/http://www.syntelinc.com/


27/33



commerce). The company's approach concentrates on the validation of business processes.

Consequently, b-acceptance offers services executing test strategy techniques, design scenarios,

recovery of data, test execution and reporting of test campaigns. "TRA Ready Mobile" is a testing

offering that is precongured and immediately available to validate clients' mobile projects.

Cigniti Technologies

www.cigniti.com

Cigniti Technologies is a pure-play testing service provider with a mobile application testing

practice. Cigniti Technologies employs 250 resources that are specialized on mobile application

testing services. Cigniti Technologies offers mobile testing to its clients across U.S., U.K., Australia,

New Zealand, Asia and the Middle East. Lately, this company has grown rapidly, with a value

proposition primarily centered on IP tools and accelerators, but also frameworks, test COE and

investments in global enterprise labs and partnerships. Cigniti Technologies uses its mobile test

COE to combine best practices for its mobile practice and it has just invested in building a next-

generation mobility lab with cloud capabilities. The company's mobility testing services includemobile test automation, mobile performance benchmarking, customer experience and digital

testing, mobile apps testing, functional testing, compatibility testing and security testing for both

enterprises and independent software vendors (ISVs).

CresTech Software Systems

www.crestechglobal.com

CresTech Software Systems is a pure-play testing service provider with a mobile application testing

services practice. The company has over 300 consultants spread across its three delivery centers in

India and North America. CresTech Software Systems has developed a testing platform, "Mobo-Sure," covering all aspects of mobile application quality, including functional, automation,

performance and security testing across various platforms, devices and carriers. Through this

platform, the company offers mobile application testing services including device compatibility

testing, automation and performance as well as security testing services.

CresTech Software Systems has set up a mobile testing lab; enabling multidevice testing that also

includes a patented platform-agnostic test automation framework in its portfolio. CresTech Software

Systems has also developed a reusable test asset library.

Kualitatem

www.kualitatem.com

Kualitatem is an independent software testing and information systems pure-play company with

ofces in the U.S. and United Arab Emirates (UAE). It provides services for Web, mobile and

desktop applications and its mobile application testing services practice includes 35 resources in

total. Kualitatem has developed a platform for mobile application testing covering key testing types


http://www.kualitatem.com/http://www.cigniti.com/http://www.kualitatem.com/http://crestechglobal.com/http://www.cigniti.com/


28/33



such as functional, security and performance, compatibility, usability and localization testing. The

company has a lab where it provides access to customers while testing using real-time devices.

QA Mentor

www.qamentor.com

QA Mentor is a pure-play testing services company established in 2010. Mobile testing is part of its

"Lab Compatibility Services," which is also offered separately to clients. Although all 106 company

resources are able to support mobile testing projects, there are 28 dedicated resources focused on

just mobile testing. Most of the mobile practice revenue is generated in North America. Despite this,

revenue is also derived from Europe (France, Germany, Ukraine) and Russia, as well as India. QA

Mentor uses its global delivery centers to provide services to clients of varying size and scale using

its own methodology. The company offers mobile testing services along with both functional and

nonfunctional areas of testing. An e-commerce platform is what the company considers to be one

of its key "sweet spots."

QASource

www.qasource.com

QASource employs around 550 QA engineers. QASource works with clients in North America with

delivery provided by its Indian delivery center. This pure-play testing service provider offers mobile

application testing services as part of its portfolio. Depending on client requirements and growth,

the number of mobile testers uctuates slightly, but around 240 resources are dedicated to the

mobile application testing practice. QASource has some local presence in the shape of project

managers in North America but the rest of its team is based in India. QASource has a mobile device

lab of over 150 real devices utilized by its experienced test engineers.

QualiTest Group

www.qualitestgroup.com

QualiTest Group is the second largest pure-play testing service provider globally. Most of its revenue

originates from its operations in North America, Israel, U.K., the Netherlands, India and Australia.

The total application testing resources of QualiTest Group comprise 2,000 resources — and within

this practice it also offers mobile testing services. Mobile test automation is a differentiator and it

provides coverage on all major devices and platforms.

The company also uses mobile test labs where they can provide access to environments for

distributed testing. QualiTest Group aims for exibility with tools that can be generated quickly to

get projects running as well as by offering on-demand services. QualiTest Group provides testing

through the life cycle of consulting to managed services as well as crowdtesting services. Services

include tests such as test process improvement, security, accessibility and load testing.


http://www.qualitestgroup.com/https://www.qasource.com/http://www.qamentor.com/


29/33



Testing4Success

www.testing4success.com

Testing4Success is a pure-play testing service provider focusing on usability testing services. The

company ensures quality on Web, mobile or desktop applications. It has built QA test labscontaining hardware, software and tools to create multiple test environment. Testing4Success offers

mobile-application testing services, including testing on real devices. The services offered by

Testing4Success include functional testing, compatibility testing and usability testing.

ZenQ

www.zenq.com

Created in 2003, ZenQ is a pure-play testing services company with around 300 resources, with 75

residing in the mobile application testing services practice. ZenQ works with clients in the U.S.,

U.K., Canada and other parts of Europe (via its ofces in the U.S. and India). It organizes its mobile

application-testing offering along functional, performance, security and automation testing services.

It has COEs for major testing functional areas. ZenQ has developed expertise across mobile OS

platforms, offers device-agnostic testing, testing on mobile simulators and mobile platforms, and

testing automation using commercial and open-source tools.

Mobile Service Providers

Mobile service providers have mobility as their core focus. Buyers can consider these providers for

their specialist mobile skills and focus, often using modern approaches.

InfoStretchwww.infostretch.com

InfoStretch is a pure-play mobile testing provider of next-generation mobile application life cycle

services, ranging from design and development to testing and sustenance. The company employs

around 700 resources and is focused on the U.S. market, but has also expanded into the European

marketplace. Since 2004, it has been offering solutions and services to enterprises in the mobile

ecosystem. The company uses its two mobile development centers to support delivery as well as a

test lab with access to various OS and devices. It develops accelerators on top of commercial

existing tools or open-source tools and has developed automated tests for mobile handset device

testing. The range of services offered by the company includes functional, nonfunctional,

performance testing, usability testing, user experience, language testing and security testingservices.

Market RecommendationsMobile testing is no longer simply about the functionality of an application. It must cover all the

aspects of platforms, networks and devices in order to meet user expectations, which require very


http://www.infostretch.com/http://zenq.com/http://www.testing4success.com/


30/33



specic skills from providers. Evaluate mobile testing service providers based on all the aspects of

complex mobile testing requirements such as infrastructure, skills, tools, performance, usability and

end-to-end quality approach and strategy.

In addition, understand the specic demands of the organization and the objectives they are trying

to achieve, which need to account for aspects such as variations in the development methods andprocesses that must be supported (waterfall, agile, mixed), support of bimodal, partnerships and

availability of tools, accelerators and IP.



31/33



Acronym Key and Glossary Terms

API application programming interface

Benelux Belgium, the Netherlands and Luxembourg

CIO chief information ofcer

COE center of excellence

EMEA Europe, the Middle East and Africa

iOS iPhone operating system

IoT Internet of Things

IP intellectual property

ISV independent software vendor

LaaS lab-as-a-service

MTAF mobility test automation framework

NFC Near Field Communication

OS operating system

POC proof of concept

POS point of sale

QA quality assurance

RFI request for information

RFP request for proposal

TaaS testing-as-a-service

TCS Tata Consultancy Services

UAE United Arab Emirates

UI user interface

UX user experience



32/33



Gartner Recommended ReadingSome documents may not be available as part of your current Gartner subscription.

"Market Guide for Crowdsourced Application Testing Services"

"Market Guide for Mobile Application Testing"

"How to Choose a Mobile Application Testing Service Provider"

"Magic Quadrant for Application Testing Services, Worldwide"

"Essential Metrics for Outsourcing Testing Services"

Evidence

"Flipping to Digital Leadership: The 2015 CIO Agenda"

1 Gartner analysts received 47 inquiries on mobile application testing services between September

2013 through September 2014, with an increase between September 2014 through September

2015 of 64.



33/33


GARTNER HEADQUARTERS

Corporate Headquarters

56 Top Gallant RoadStamford, CT 06902-7700

USA +1 203 964 0096

Regional Headquarters

AUSTRALIA BRAZILJAPANUNITED KINGDOM

For a complete list of worldwide locations,visit http://www.gartner.com/technology/about.jsp

© 2015 Gartner, Inc. and/or its afliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. or its afliates. This

publication may not be reproduced or distributed in any form without Gartner’s prior written permission. I f you are authorized to accessthis publication, your use of it is subject to the Usage Guidelines for Gartner Services posted on gartner.com. The information contained

in this publication has been obtained from sources believed to be reliable. Gartner disclaims all warranties as to the accuracy,completeness or adequacy of such information and shall have no liability for errors, omissions or inadequacies in such information. This

publication consists of the opinions of Gartner’s research organization and should not be construed as statements of fact. The opinions

expressed herein are subject to change without notice. Although Gartner research may include a discussion of related legal issues,Gartner does not provide legal advice or services and its research should not be construed or used as such. Gartner is a public company,

and its shareholders may include rms and funds that have nancial interests in entities covered in Gartner research. Gartner’s Board of

Directors may include senior managers of these rms or funds. Gartner research is produced independently by its research organizationwithout input or inuence from these rms, funds or their managers. For further information on the independence and integrity of Gartner

research see “Guiding Principles on Independence and Objectivity ”
http://www.gartner.com/technology/about/ombudsman/omb_guide2.jsphttp://www.gartner.com/technology/about/ombudsman/omb_guide2.jsphttp://www.gartner.com/technology/about/policies/usage_guidelines.jsphttp://www.gartner.com/technology/about.jsp

Documents

Market Guide for Mobile Appl 273008