The Magazine for Professional Testers

www.agiletestingdays.com

EUROPE’S GREATEST AGILE EVENT!November 10–13, 2014 in Potsdam, Germany

“The Agile Movement Is Thriving! How Does It Affect the Future of Software Testing?”

With respect to our conference theme, we would like to receive proposals mainly for track talks of 40 minutes and workshops of 120 minutes duration. Please find our topics of interest below and use the online form to submit your proposal. The Call for Proposals will close after April 13, 2014.

Topics of Interest:Test-Driven Development (TDD)/Behavior-Driven Development (BDD)  · Acceptance Test-Driven Development (ATDD)  · Automated unit-level tests  · Automated integrati-on-level tests · Automated system-level regression tests · Automated ATDD · Exploratory testing on Agile projects · Tools “for Agile Testing”: for TDD/BDD; for ATDD; for Continuous Integration (CI), Build and Integration Management, Version Control Systems (VCS); etc. · Refactoring  · Mobile app testing  · Emerging design  · Managing technical debt in test suites · Creating and maintaining non-brittle test suites · Agile testing and compliance requirements  · Agile testing and non-functional requirements  · Requirements in Agile projects  · Continuous Testing (based on CI): e.g. Automated Regression Testing, Cont-inuous Performance Testing, Continuous Load Testing, Continuous Scalability Testing,

etc. · Testing and the fast pace of Agile development · Measuring Agility · Transition from “traditional” development methodologies to Agile teams  · Managing testers on Agile teams · The roles of an Agile tester · Testing and QA · The role of a QA specialist on an Agile team · Collaborative extension: testing challenges for distributed team · Measurable im-provements as a result of Agile practices · The problems and solutions of changing to an Agile culture as seen from a tester’s perspective · Human perspective and psychological aspects of Agile development and testing · Collaboration and building teams · Coaching and mentoring · Management and leadership in software organizations · Enterprise Agi-le  · Security testing  · Systems-thinking and patterns  · User experience and interaction design · Working with customers · Agile for embedded systems

THIS YEAR’S MOTTO

CROWD TESTING

March 2014

Mar

ch 2

014

25No.

25

34 Testing Experience – 25/2014

The information technology world is shifting from one driven by corpora-tions to one driven by consumers, who demand more flexibility as they carry their devices through a myriad of environments and ecosystems. This lays out a clear challenge to software designers and their quality assurance teams. High quality design and functionality are absolute pre-requisites for building customer loyalty. Crowd testing lets real users test the software under real-world conditions. This allows companies to gather real insight, feedback, and defects quickly and cheaply, leading to significant quality and design improvements, and improving the chances of software adoption by customers and end users. Using the crowd to test software raises the bar over other common options like outsourcing or automation when it comes to levels of quality, flexibility, speed, and cost. Today, well-designed and high quality software components are pre-requisites for building customer loyalty. Extensive testing is obliga-tory to ensure that software meets customers’ exacting design, utility, and performance expectations. Software has to perform on the various permutations of devices and usage scenarios that cannot be predicted in advance. In-house testing with limited test resources cannot meet such extensive test coverage requirements. Crowd-sourced software testing subjects the software to testing under real-world conditions by real users, allowing companies to gather real insights, feedback, and

defects quickly and cheaply, leading to significant quality improvements.

1. Introduction

Over the past few years, the crowdsourcing paradigm has evolved from its humble and isolated custom initiatives to success stories, such as Wikipedia and Amazon Mechanical Turk. In the field of information technology, crowd-sourced software testing has witnessed increased interest and adoption, especially among startups and smaller companies. The adoption has, however, been quite slow among larger businesses, mainly due the paucity of published business literature. To facilitate crowd testing by large businesses, we shall discuss some important issues, such as make-or-buy decisions, building an internal testing community or partnering with external agencies, application suitability, governance, and tester quality, and conclude by providing some guidelines for suc-cessful crowd testing engagements.

2. Today’s IT landscape

In recent years, information technology (IT) has assumed such strategic importance that many companies have already made substantial invest-ments in ERP, CRM, and other business tools, demonstrating the critical nature of enterprise IT. These companies also have consumer-facing components, such as websites and mobile applications, which should not only be well designed, but also deliver uninterrupted functionality. It is almost impossible to predict the users of such systems and their tastes a priori. Crowd-sourced software testing helps achieve a high

probability of consumer acceptance of software modules by ensuring that the software is defect-free to a very high degree.

With the growing trend in cloud computing, BYOD, etc., IT services are delivered over multiple channels and consumed by diverse stakeholders whose IT systems may come in different configurations. Truly robust software testing is time consuming, and it is nearly impossible to ensure that every possible permutation and combination of features, localiza-tions, and platforms works as intended. For example, if a web applica-tion does not render in a particular browser, or a particular software tool fails to deliver a critical functionality, business disruptions may ensue. Through crowd testing, companies could effectively diminish the likelihood that the internal software testing team might have missed critical elements during the internal testing phases. According to Paul Herzlich, a software-testing analyst at Ovum, an independent IT industry research institution,

“If you are testing software that all kinds of strang-ers are going to use, then why not use a bunch of strangers to test it.”

Thus, crowd testing offers a simple, elegant, and cost-effective solution to what would otherwise be an intractable quandary.

By Dieter Speidel, Mayank Mittal & Mithun Sridharan

Enterprise Crowd Testing

Figure 1: Crowdsourcing Trends Since 2007 (from Google Trends)

Figure 2: Regional Interest in Crowdsourcing (from Google Trends)

Testing Experience – 25/2014 35

3. Exploratory and enterprise crowd testing

Presently, much of the focus is on exploratory crowd testing services, where testers analyze software for issues, bugs, or defects based on generic guidelines and test cases. With “free hand” exploratory testing, the outcomes cannot be quantitatively predicted, as it is challenging to qualify the range, area, and number of possible defects in advance. Testers usually identify issues in the most obvious places, but some highly skilled testers may uncover defects hidden under layers of the software. Exploratory testing is best suited to testing less mission critical applications with a large number of users to ensure reproducibility of a set of issues under different configurations or to simulate a typical load on an application. Enterprise crowd testing incorporates more rigor and structure in the software testing process, while simultaneously retain-ing the spirit and agility of exploratory testing. Thus, enterprise crowd testing incorporates the best practices and concepts of both informal and formal software testing methodologies. The degree of profession-alism involved in enterprise crowd sourced software testing is higher.

In this approach, testers are scrutinized and deployed based on compre-hensive selection criteria. The software testing process involves a clearly identified scope, and is guided by extensive guidelines tailored to fit the needs of the project. The team has clearly identified roles and responsi-bilities, a single community point of contact for the client organization, formalized reports and reporting structures, extensive documentation and project tracking, etc. Furthermore, enterprise crowd testing entails formal processes built over existing software testing methodologies,

such as Agile, SCRUM, etc. As a result of such alignment, the client or-ganization experiences a smooth synchronization between its internal development and external software-testing cycles. This allows the client to reduce its project management overhead and maximize the benefits from both teams.

Díaz & Hilterscheid GmbH

www.diazhilterscheid.de

In the Mobile App Testlab your app is tested on-site on a multitude of devices. Our experienced staff members help you with optimizing your app and perfecting the user experience.

36 Testing Experience – 25/2014

4. What applications are good candidates for crowd-sourced testing?

The crowd-sourced testing methodology is the most suitable for applica-tions that are user-centric. It is frequently used in mobile applications and game development projects by releasing cheap or free versions of products in the beta stage. Large corporations could benefit from crowd-sourced testing by simulating a large user base to understand usage patterns and improve on feedback, while ensuring their applica-tions run smoothly on a number of different devices, operating systems, browsers, and language versions. In other words, applications with a high defect exposure factor, i.e. criticality as a measure of immediate customer exposure post release, are good candidates for crowd testing.

For example, Microsoft released the beta version of its Office 2010 pro-ductivity suite, which was downloaded and tested by 9 million people who provided 2 million valuable comments and insights, which resulted in substantial product improvements.

5. How does it work?

Most crowd-sourced testing companies provide the platform and the project management framework for the testing cycles, including gover-

nance and legal structures. The crowd testing company also allocates a qualified project manager who oversees the testing process.

On the consumer side of crowd-sourced software testing services, the clients specify the type and battery of tests they wish to have performed, the skill set and demographic information of testers, the types and configuration of devices the software must be tested on, etc. The cli-ent company could also consult the crowd testing company to ensure testing adequacy.

On the supplier side, the testers complete a profile, indicating their background and qualifications, the devices and platforms they have access to, and other relevant details. Testers may sometimes be required to participate in assessments or trial projects as a pre-condition for community membership. Before a project starts, testers are provided with detailed test plans, sample scenarios, tools, scripts, and instruc-tions. During execution, the testers document their observations and are rated based on the volume and quality of their reports, which links directly to their compensation and incentive structures.

The community combines aspects of collaboration and competition, as members work to find solutions to the stated problem. Forums facilitate knowledge management, networking, and discussion of bugs or relevant issues; rating systems allow for recognition of a job well done, which helps participants gain credibility and improve their careers.

Crowd testing should be considered additional complementary testing prior to production release rather than as a self-contained activity and replacement to an organization’s internal testing team. It is effective in catching any residual UI and configuration defects prior to produc-tion release.

6. Onsite, outsourced, and crowd testing

Onsite testing refers to the established practice of deploying an inter-nal team for software testing purposes. This approach has limits to its scalability and is ineffective in reproducing realistic usage situations.

In outsourced software testing, the software testing is rendered by an ex-ternal service provider, on a per-project or agreement basis. Outsourced software testing has obvious cost benefits – highly skilled resources at a fraction of costs – through labor arbitrage.

In crowd testing, the testers voluntarily test software, motivated by their inherent drive to detect software defects, establish their stand-ing in a community, gain peer recognition, etc., despite the risk of their not getting paid if no issues are detected. This self-organizing feature, shared community goals, and intrinsic motivation are strong forces for crowd testing’s success.

7. What to choose?

The onsite team, due to its proximity to customers, delivers value by focusing on technological and business challenges requiring immedi-ate attention.

EXAMPLES OF CROWDSOURCED TESTING

PREPARATION PHASEAssess client’s business needs and goals

Design the Test Plan incl. required skills and capacities

Fix price and deliverables

1

INITIATION PHASEDefine test cases and configurations

Recruit crowdtesters from a global pool of professionals

Establish secure test environment

2

EXECUTION PHASEInitiate testing across required devices and software

Manage and supervise test team

Monitor and ensure performance

3

COMPLETION PHASEPresent in-depth results to client

Refine crowdtesters’ classification ratings

Close test cycle

5

EVALUATION PHASEValidate results and conduct quality assurance

Prepare client deliverables

Process testers’ compensation

4

HOW IT WORKSHOW IT WORKS

Testing Experience – 25/2014 37

In an outsourced testing project, the testing team(s) could be either, captive or non-captive to the outsourcing company. In a captive model, the testing team is part of the company and performs software testing as an ongoing activity from which the captive team gains experience and efficiency levels through an enhanced understanding of the most common internal applications.

Non-captive teams are external to the company and work with several clients on several projects. From these engagements, non-captive teams gather experience and expertise across several areas, domains, func-tions, industries, and technologies. The client company could leverage such diversity in projects that require “thinking outside the box” and special testing efforts. A crowd-testing team’s laser focus, single-minded efforts, inherent motivation, and scalability deliver tremendous value by detecting issues before go-live. The crowd’s knowledge is diffused among its tester base, but the collective testing knowledge base is larger than that within any company.

Given the complexities of modern systems, an optimum testing organi-zation should comprise of onsite, outsourced, and crowdsourced testing teams in the right proportion. Converging on the right proportion is an element subject to experience and experimentation.

8. Build a community or establish partnerships?

While some companies, such as Netflix and Amazon, have built their own crowd capabilities internally, this is not an activity that most com-panies could achieve on their own or that makes tactical business sense. Crowdtesting is a complex activity requiring deep technical, functional, and business knowledge. So, unless the work to be done is so strategic or highly confidential to the business, a company is well advised to seek a crowdtesting company that delivers all the necessary ingredients to deploy the crowd almost immediately. These ingredients include experienced testers, devices, configurable architectures, participation mechanisms, ground rules, engagement methodologies, legal constructs, support mechanisms, etc.

9. Security and Governance

More often than not, confidential customer information is exposed to the crowd during testing cycles. Crowd-testing firms have realized the need for security and confidentiality; hence, they have implemented several layers of screening, filters, governance mechanisms, obfuscation tools, data management frameworks, etc., to assure their clients that their data and information is processed with the utmost confidence. For projects requiring high security, a pre-screened list of white hat test engineers who have long-standing credibility and professional reputation are proposed and usually accepted.

Test data management ensures security by obfuscating sensitive infor-mation or creating ‘test only’ data. This helps minimize the impact of potential security breaches during large-scale testing. Testers are also required to sign a general non-disclosure agreement (NDA) when they join the community, which forbids them from discussing their customers, products, etc., both offline and online on social networks, personal blogs, and elsewhere outside the private testing platform. Customers also have the freedom to choose their testers from the pool and furnish the chosen

ones with customized NDAs, which testers must sign before viewing the customer’s project. Furthermore, standardized communication channels, access restrictions, etc. help secure data and gain confidence in testing efforts, which results in a seamless transition across the various stages of the crowd-testing project lifecycle.

10.Ensuring the quality of testers

To ensure the quality of testers, formal and informal tests are periodically administered to individual testers based on their experience, available material, and concepts mastered. The testers and project managers are continually monitored to ensure both quality and integrity. Through constant filtering, applicants without formal training and significant professional experience are eliminated. Increasingly, crowd testing companies are also sponsoring professional certifications for experi-enced testers, so their standing and responsibilities grow in proportion to their contributions.

To avoid potential conflict of interest among testers, performance-based compensation mechanisms are deployed to ensure that testers’ compen-sation is commensurate with their level of engagement and contribution to software quality improvement. Testers who detect a larger number of less impactful defects are compensated less than those who detect a few high-impact defects. These mechanisms also help prevent any flame wars, jockeying, etc. within the community and increase testers’ focus, ownership, loyalty, and affiliation to the community, thereby creating a congenial and respectful working environment.

11. Choosing a crowd-testing company

Among the various stages of the crowd-testing process, choosing the right testing company is by and far and away the most important. It is mandatory that the client company performs adequate due diligence, extensive project evaluation, capabilities and scale of the testing com-pany, etc. before choosing a crowd testing platform.

In general, selecting a proven and experienced crowd testing platform makes the operational activities much easier, as there is usually less maintenance and governance required during the operational stages because the testing company has already addressed many of the relevant issues from past experiences with other clients.

Therefore it is important to evaluate crowd testing platforms on at least the following criteria to ensure that the platform contains at least the minimum functionality and security required:

1. It is important to ascertain the crowd testing company’s level of experience, delivery capability, skill diversity, and global outreach to accommodate the evolving requirements of enterprise custom-ers.

2. A credible pedigree demonstrated through organic growth and experience with both corporate and mid-scale companies. It is relatively easy for a small crowd-testing platform to offer mobile application and website testing services, but very few companies have the maturity to test enterprise applications and provide turnkey testing services.

38 Testing Experience – 25/2014

3. Financial stability built over a period of time is particularly impor-tant if client companies aspire to undertake long-term projects without worrying about the service provider exiting the market.

4. Many crowd-testing platforms operate merely as matchmak-ing services between clients and testers rather than providing a managed service. These platforms do not assure the number of software testers, amount of testing work done by individual testers, quality of testers or test results, nor any reliable outcomes. Such platforms are best avoided.

5. An assiduous review of the Terms & Conditions is mandatory to understand their scope of services and SLAs provided.

6. A Sourcing Hierarchy of Constraints & Criteria document should be used for evaluating platform operators for every major crowd testing project. This document should address various elements, such as:

▪ Price structures

▪ Quality, safety, security and confidentiality

▪ Professional services portfolio

▪ Capabilities and processes

▪ Delivery methodology

▪ Service infrastructure

▪ Compensation structures

▪ Convenience and scalability

▪ Functionality and quality of the platform and tools

▪ Risk management structures

Summary

Having discussed the various elements of crowd testing, it could easily be inferred that large enterprises could expect substantial cost savings and scalability by partnering with established and experienced crowd-testing companies. Crowd-testing tools, processes, frameworks, and methodologies have matured to such a high degree that crowd testing could easily be incorporated into the normal development life cycles at organizations. Crowd testing decreases the likelihood of human error by diversifying software testing activities among a much larger team. By subjecting the application to realistic conditions, actual performance data and execution paths can be traced which can be plowed back into re-engineering or improvements, leading to further quality improve-ments. Besides these direct benefits, large companies can also experience faster time to market, free up resources to focus on core operations, gain access to a large pool of talents, increase flexibility, etc.

Getting started is easy, and those organizations that dare to take the plunge are assured of the several benefits in both the short and the long term.

Bibliography

1. Brabham, Daren C. Crowdsourcing as a Model for Problem Solving. SagePub, 2008.

2. Gerrard, Paul. Crowdsourcing and Testing: Can it work? UK Tester Forums, 2009.

3. Hoßfeld, Tobias, et al. “CrowdTesting: A Novel Methodology for Subjective User Studies and QoE Evaluation.

4. Kaner, Cem. A Tutorial in Exploratory Testing. Department of Soft-ware Engineering, Florida Institute of Technology, 2008.

5. PASS Technologies AG. “Crowdtesting and the Future of Software QA.” May 2, 2013. http://www.passbrains.com/blog/crowdtesting-and-the-future-of-software-qa-2/.

6. PASS Technologies AG. “Testing Applications for the Real World.” 2012.

7. Ziegler, Wibke . Crowdtesting – Bugtracking via Crow d. Volksbank Bühl, 2013. ◼

Dieter Speidel (53) is founder and CEO of PASS Group, a leading Swiss provider of software and system test services. PASS Group is owner and home of passbrains.com, a global platform for on-demand crowd testing services, with offices in Zurich, Berlin, Boston, Belgrade, Bangalore, and Mumbai. Being active as an entrepre-neur in the IT outsourcing, software development, and testing business for more than 30 years, Dieter Speidel

has in-depth expertise in global delivery models, including near/offshoring and crowdsourcing. Since 2011, he has specialized further in the application of crowdsourcing models to IT related services and developed the passbrains platform and community for crowd testing and knowledge services.

Mayank Mittal is a strategic business leader in the ap-plication testing industry. During his career spanning over a decade, he has established large scale managed test centers and executed QA organizational transfor-mations while working with leading organizations such as Oracle, CSC, and Cognizant, delivering testing solutions to many Fortune 100 customers across the globe. Mayank works as Director Corporate Strategy

and Business Development at PASS Group AG.

Mithun Sridharan (33) is a Business Development Manager with Passbrains, based in Eschborn, Germany. He brings over ten years of international experience in business development, marketing, global delivery and consulting. He holds a Master of Business Ad-ministration (MBA) and Master of Science (M.Sc). He is a Project Management Professional (PMP) and a Certified Information Systems Auditor (CISA). He also

serves as the Communication Chair for the German Outsourcing Association.

> about the author