In tegrat ing CrowdTest ing in to

the process

Ay a l Z y l b e r m a n

QualiTest was recognized as the

only visionary pure play QA and testing Service Company in the “Gartner Magic

Quadrant”

2

Intro

3

Source: Nelson Hall Report, 2013

QualiTest is the world’s second largest testing

services specialist by revenue

Intro

Test Outsourcing Engagement Evolvement

4

Staff Augmentation

TCoE / Offshore

Managed Services

Crowd Testing

•Time & Materials

•Mainly in-house

contractors

•Cost deduction of

price per role

•Time & Materials

•Blended Rate

•Cost deduction of

average hourly

rate

•Time & Materials

and Fixed Price

•Responsibility

shitted to vendor

•Long Term

engagement

•Results Based

Testing: Service

Level Agreement /

KPIs

•Advanced pricing

models (pay per

bug/cycle)

•Test In The Wild

•Task based

•Results focus

Intro

• “Crowdsourcing is the process of obtaining needed services,ideas, or content by soliciting contributions from a large groupof people, and especially from an online community, ratherthan from traditional employees or suppliers”

• CrowdTesting is an emerging trend in software testing whichexploits the benefits, effectiveness, and efficiency ofcrowdsourcing and the Wisdom of the Crowd.

• It differs from traditional testing methods in that the testing iscarried out by a number of different testers from differentplaces, and not by hired consultants and professionals” Wikipedia

5

CrowdTesting is an approach

6

7

CrowdTesting

Access to over2.5 million testers

through over40 platforms

Top 5 reasons to use CrowdTesting

1. Access to specific talents

2. Ability to perform tasks quickly and efficiently

3. Test in the wild – different opinions, environments

4. The right incentive - reduce cost of xxx

5. It’s cool

8

Test Types1. Exploratory Testing – testing the entire functionality

2. Feature Testing – focus on specific feature/functionality

3. Localization Testing – focus on various languages

4. Compatibility – testing various Mobile devices

5. Usability – User Experience and Ease of use done by a mixture of real users and UI experts

6. Inter-operability Testing – running different applications to ensure proper performance

7. Configuration Testing – run device on different configuration settings

8. User specific testing – run testing on specific type of users (existing clients, designers etc.)

9. Test Data – create massive test data

10. Test Planning – convert design into test cases

11. Beta Testing – test the system using real users

9

Private CrowdTesting

10

CrowdTesting Management Platform

11

Business Case

12

• Testers are usually paid by the bug

• Customers usually pay by cycle or subscription

• Pricing defined based on:

– Complexity of testing

– Amount of testers

– Crowd testers qualifications required

• ROI

– Cost of CrowdTesting vs. Traditional

– Cost of finding a defect

Moment of Involvement

• Before each release (Test Environment)

• After each release (Development Environment)

• Weekly basis – in parallel to internal testing

• After each sprint

13

CrowdTesting & DevOps

• DevOps is a software development method that stresses communication, collaboration, integration, automation and measurement…

• Testers becoming closer to the clients/end users and thus required to communicate with them

• CrowdTesting helps testers to get the real end-user experience and feedback

14

Example of CrowdTesting and Devops integration

• CrowdTesting cycles is performed after initial deployment of any new feature

• Companies with very frequent releases require on-going feedback from Crowd. Using Crowd as “human monitoring system”

• Communicate and get feedback from crowd before defining sprints backlog: Surveys, UI Review, Heuristic Evaluation

• Testing as a Service (TaaS):

15

1 hour

Target

• Reinforce the test team. Beta testing for specific SIM cards and locations

Achievements

• 10 private crowd testers

• Completed full cycle in South Africa within 3 days

16

Target

• Build Test Data used for regression test automation

• Repetitive work of tagging webpages and images

Achievements

• Categorize 157.000 URLs and images

• Completion within 3 weeks (compared to 6-12 months)

• Reduce the cost by over 80%

17

Target

• Testing web game in 52 countries and 23 languages

• Testing real life conditions

Achievements

• Full coverage delivered in 30 days

• Over 800 defects detected

18

Target

• Testing 20-30 mobile apps

• Running over 200 cycles each year

• Continuous integration SDLC (release each week)

• Functional, Usability and localization

Achievements

• Reduced cost per defect from 244£ to 48£

19

Thank You !

w w w . Q u a l i Te s t G r o u p . c o m