TESTING AS A RISK CONTROL STRATEGYSergio DutraSoftware Quality Expert

FOUR STAGES OF SOFTWARE TESTING COMPETENCY

STAGE 1: JUST TEST! Take what the creators give you and test it,

using manual or automated tests. Triage issues. Update tests as necessary. Analyse and report on results.

STAGE 2: BUILD A DEEP UNDERSTANDING OF THE PRODUCT, THEN TEST IT. Drive for clear requirements and design, even if

you have to create these yourself! Build models to understand how the system

works. Look beyond problems with the product,

including how it can be misused, or how its dependencies can fail. Create tools to identify common problems, such as

misconfigurations or network issues. These can be used not just for testing but for customers and partners.

STAGE 3: UNDERSTAND HOW THE PRODUCT WILL BE USED, THEN TEST IT. Participate in user research, customer support

and other customer engagement tasks, to understand how the product will be, or is, perceived by customers.

Consider the whole product lifecycle, including acquisition, initial configuration, typical usage, maintenance, upgrade and retirement.

Drive the definition of quality standards, processes, tools and community knowledge to support the product in the ecosystem.

STAGE 4: UNDERSTAND THE BUSINESS, AND USE TESTING TO REDUCE BUSINESS RISK.

Q: WHY DO BUSINESSES INVEST IN TESTING?A: Businesses want to reduce the risk in achieving their goals. Businesses want to avoid risks like: “Deal breakers” for the consumers. Exposure to liability from consumers, regulatory agencies or other parties. Tarnishing of the company’s reputation. Exposure of the business or its assets to threats, such as theft or abuse.

RISK ANALYSIS BASICS

Identify what controls are currently in place and which ones are needed. Design and implement any new controls needed.

Identify the business assets at risk. Assess potential threats to the business and prioritize them based on likelihood and impact. Determine actions to take on each threat. Then:

Accept the risk; Eliminate the threat; Reduce the impact; or Delegate the risk to a third party.

EXAMPLES OF BUSINESS RISKS Widely visible patterns of defects across

products or services provided, or used, by the business.

Unintended exposure of business IP or other sensitive assets (e.g. accounts, passwords, etc.)

Abuse of the product, its customers or partners. Non-compliance with critical standards, or with

legal or geopolitical requirements. Aspects of the ecosystem – including partners

and competitors – that effectively nullify benefits of the product.

EXAMPLE OF ECOSYSTEM THREATS: SSL SSL is a protocol that allows secure communication between a client and a

server. Its ecosystem includes: Cryptographic and public key infrastructure (PKI) components. Entities that issue and manage certificates. Browsers, networks and all other aspects of the network stack.

Some ecosystem issues that need to be considered include: Will fixes in cryptographic dependencies require updates in the protocol? How reliable is certificate issuance and management? What issues could arise,

and how can the protocol itself mitigate these? Are there ways for consumers to improperly configure SSL? Are the users of the browsers and apps made aware – in an effective manner –

that they are communicating securely, and that it’s important to do so in the context of their use?

THOUGHT EXPERIMENT – HIGH LEVEL RISK EVALUATION OF CANDY CRUSH SAGA Key revenue earners for Candy Crush Saga

Ability to charge money for certain game features (e.g. unlock levels) Attention share of users on the game and the ability to keep growing it

Asset Importance

Main Threats

The revenue earning features of the game

Highest 1. Features don’t work and generate no revenue2. Revenue aspect can be bypassed (front or back ends)3. Revenue can be redirected to other parties4. Customers can be charged for features that don’t work

Attention share features of the game

High 1. Game can be altered to introduce unwanted aspects2. Unable to collect usage information from customers 3. Usage information can be redirected to third parties

Intellectual property pertaining to the game

Moderate Third parties can effectively copy the game and attract game users to it

Sensitive operational assets

Highest Sensitive data – such as user identities or credit card data – is stolen, damaging users hence the profitability of the game

Disclaimer:

I have only common knowledge about how Candy Crush Saga works,

and no knowledge of how the company actually evaluates risk or

tests the application.

This exercise is provided merely as an example of how to approach

testing such an app.

ELEMENTS OF A RISK-BASED TESTING STRATEGY Prioritized list of risks to the business, within the

context of the product under test. Areas of focus and how they pertain to each

business risk. Methodology applied to each area of focus.

THOUGH EXPERIMENT – RISK-BASED TEST STRATEGY FOR CANDY CRUSH SAGABusiness risk Areas of focus MethodologiesFeatures don’t work and generate no revenue

Unlock a level; Add time; Add moves; Add lives.

Functional; App sleep/resume cycle; Network reliability.

Revenue earning features can be bypassed

Persistent app state affecting levels, time, moves and lives; Service entry point and protocols; Billing service.

State-based testing; penetration testing; service DoS; false billing data entry; billing service dependencies.

Revenue can be redirected to other parties

Billing service; service protocols.

Secure design evaluation; penetration testing; false billing data entry.