The Mega Actionable Continuous Testing Workshop

Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.

The Mega Actionable Continuous Testing Workshop

Client Services Consultant – DevOps

Surya Suravarapu

2 Broadcom Proprietary and Confidential. Copyright © 2021 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries.|

• Introduction – What You’ll Learn

• The CT Mega Actionable Report

• Continuous Testing – What & Why

• Segment 1: The CT Framework Revealed– People

– Process

– Technology

• Segment 2: Making Continuous Testing Mega Real

• Conclusion, Raffle and Wrap-up

The Mega Actionable CT Workshop - Today’s Agenda


Defining the end to end Value Stream - DevOps

Ideation

Customer

Experience

Continuous Testing

Test Data /Synthetic Data

Test CaseCreation

PerformanceTesting

Mocking& Virtual Services

Continuous Integration Operations

CapacityManagement

WIP

BuildCode

Synthetics

ServiceManagement

Monitoring

Transformation Intelligence

Business

Strategy

XLHigh Level

Planning

Backlog

Features

Stories

BehaviorModeling

Test Scripting

Test Management

Continuous Delivery

Release

Orchestration

CD Automation

API Governance &Management

Quality Management

Test Planning

Tracking

OKRs

Platform / Cloud Management (Kubernetes)

Platforms


Reality: DevOps Success Is Difficult and Elusive

CEOs not happy

with the speed of

software delivery

78

%

Lost revenue and inability to

respond to market opportunities

=

Accelerate innovation

Delivery Takes Too Long

CEOs that say

software quality

needs to

improve

77

%

Bad software can cost billions

and result in irreparable

damage

+

Reduce operational risk

Quality Still Unpredictable

Very successful

at quick

development and

implementation

Neutral

Unsuccessful at

quick

development and

implementation

Only 1/10 organizations have mastered the balance between innovation and risk

Source: Freeform Dynamics 2018, How Agile and DevOps enable digital readiness and transformation

Source: Harvard Business Review Analytic Services 2019, Competitive Advantage Through DevOps: Improving Speed, Quality, and Efficiency in the Digital World


Common Challenges With DevOps Success

Massive Complexity Abundant Bottlenecks Low Visibility Enable or Protect


Key Challenges: Balancing Velocity and Risk at Scale

Traditional testing delays

More than any other part of the DevOps process

Automation needs intelligence

To provide quick feedback on new code changes

Need to proactively assess deployment risks

Lack of insight leads to compromising velocity for stability

Are we going in the right direction?

We are moving fast – but is it delivering the business outcomes we expect?

Inability to Assess Risk

Automation Can Help,

But…

55% more tests than they need

Testing Bottleneck

Align Velocity with Value

56% gaps in test

case coverage

Enablethe business

Protectthe business.


Solution: Modern, Intelligent DevOps Process and Pipeline

1

Intelligent Automation

to improve overall visibility,

velocity, and quality

Wasted

Time

AI-Powered, Intelligent Delivery Pipeline

Ris

k

Data-Driven, Predictive Insights, Risk Remediation

Testing

“Delays”

Modern Testing with Shift Left / Shift Right Practices

2

Embed quality

From requirements through

production to eliminate existing

QA roadblocks

3

Continuous

Improvement through ML

for business processes and

and site reliability


What is Continuous Testing?

The practice of testing

across every activity in the SDLC to uncover and fix

unexpected behaviors as soon as they are injected.


Delivery Velocity is key for High Performers

HIGH PERFORMERS

2018 State of DevOps Report

Deploy frequency (# of deploys per year)

LOW PERFORMERS

Leaders are deploying more frequently than laggards

2019 Continuous Testing Report

2%

11%

18%

29%

28%

11%

Several times per hour

Hourly

Daily

Weekly

Monthly

Less often than monthly

The frequency at which teams would like to deploy new builds

https://puppet.com/resources/whitepaper/state-of-devops-report

https://puppet.com/resources/whitepaper/state-of-devops-report


But QA Is THE Bottleneck.....

10

http://www.computing.co.uk/digital_

assets/634fe325-aa28-41d5-8676-

855b06567fe2/CTG-DevOps-

Review-2017.pdf

Where are the main hold-ups in

the software production

process?

32%

16%22%

63%

21% 23%30%

Plan code Pre-test Build Test/QA Release Deploy Review/Improve

“

63% agree

Traditional Testing Tools Are

Too Complex & Expensive

70%Of Testing Today Is

Still Manual*

*Bloor Research

Today’s Testing Center

Of Excellence Is A

Bottleneck To Speed(Not built for Agile approach)

http://www.computing.co.uk/digital_assets/634fe325-aa28-41d5-8676-855b06567fe2/CTG-DevOps-Review-2017.pdf


Traditional SDLC Process

Service Management

APIs

Customer

ProductionApplications

Security

Test Data/ Synthetic Data

Test CaseCreation

Functional & Performance

Testing

Virtual Service

DEVELOPMENT

Code BuildPortfolio & Project

Prioritization

Build Business

Case

Requirements Definition & Management

PLANNING

Monitoring

TESTING

OPERATIONS

Testing as a time-boxed event, in a linear

process.

Ideas


Continuous Testing Embedded Throughout SDLC

TESTING

Test CaseCreation

Test Data/ Synthetic Data

Functional & Performance

Testing

Virtual ServiceMonitoring

Continuous Feedback & Collaboration

Service Management

APIs

Customer


Security

DEVELOPMENT

Code BuildPortfolio & Project

Prioritization

Build Business

Case

Requirements Definition & Management

PLANNING

OPERATIONS

Ideas

Continuous Testing is the embedding of testing as a fundamental and ongoing aspect of every activity though the software delivery cycle


The goal: in-sprint Agile Testing

With Continuous Testing, testing keeps pace with Agile development.

Release Release Release Release Release

Traditional testing presents barriers to speed with quality.

Source: Dimensional Research, Continuous Testing – Companies Don’t Know What They Don’t Know, 2018


Continuous Testing Key Pillars

Requirements Quality

Pipeline Quality

Code and application

quality

Customer Experience

• Can we deploy & release apps across environments rapidly and correctly?

• Do we have the right tests defined and automated as part of the pipeline?

• Did we build it right?• Are we testing and validating all

along the way?• Quality?

• Are users perceiving value in the application delivered?

• Are we building the right things?• Are we testing the right things?

Testing

Lifecyle

Automation


General Approach to implementations

Think BIGStart Small

Learn fastScale thru success

Transform


Continuous Testing Maturity Model Summary

MA

TU

RIT

Y

Initial

Basic

Automated

Integrated &

Orchestrate

d

Intelligent

BUSINESS VALUE

LEVEL 0 LEVEL 1 LEVEL 2 LEVEL 3 LEVEL 4

?

Continuous

Optimization

End-to-End

Orchestration

Consistency &

Collaboration

Heroes &

Heavy Lifting


Business Impact and Metrics Summary

L0 Initial

Unpredictable quality, high costs

• Manual testing, minimal

automation

• High QA cost

• Depend on SME,

individuals to determine

quality

• Un-scientific measures

of test and data

coverage

• Poor/unpredictable

quality

L1 Basic

Reduced cost; improved code quality

• Requirements Ambiguity ↓

• Test design effort ↓

• Test maintenance effort,

change analysis effort ↓↓

• Test coverage ↑↑

• Unit test coverage ↑

• Code quality ↑↑

• Defect leakage from Dev

to Test ↓

• Test data coverage

• Test environment setup

time ↓

• Start implementing Shift

Left activities

L2 Automated

Improved dev/test agility; improved app quality

• Embrace Shift Left

activities

• Test automation level ↑↑

• Well defined

Automation

Frameworks

• Test execution time ↓↓

• Test feedback response

time to Dev ↓↓

• Defect leakage from

Integration ↓↓

• Test data provision time

↓↓

• Test environment setup

time ↓↓

L3 Orchestrated

Improved release agility

• Automate Everything

Test automation level ↑↑↑

• Test execution time ↓↓↓

• Continuous Testing a key

part of DevOps pipeline

• Defects due to

environment mismatch ↓↓

• Release quality: defect

leakage to production ↓↓↓

L4 Intelligent

Optimization and predictability

• Defect prevention

• Test scoping and

maintenance effort ↓↓↓

• Test prioritization

• Dynamic test selection

Maturity Level

Cap

ab

ilit

ies


Segment 1, Part 1: The People, Organization and Culture


The organizational shift to true Agile Continuous Testing

New org models require new collaboration skills with multiple personas

How does the COE fit into Agile

Center of Excellence or Enablement - Shift Left

+ + +

Produ

ct

Owne

r

Devel

opers

Tester

s

Produ

ct

Owne

r

Devel

opers

+

Tester

s

Agile

Team

Diverse goals Aligned goals Common goals

Aspiring Agile Testing– Product Owner +

Developers +

Independent Testers

Traditional Agile Testing– Product Owner + (Developers +

Embedded Testers)


People – Changing the Status Quo

• Change takes time, not always embraced, but it is possible

• Personalities are the biggest challenge

• Leaders & Early Adopters, buy in help drive changes

• Have to address Negatives…can’t let them grow

• Start with the WHY

• Desire to adopt a Continuous Testing mindset

• Company, Team and Individual benefits

• The HOW

• Developers and testers can work better together

• Bring both teams together on the same floor or workspace, if possible

• Regular meetups to ensure collaboration

• Define Team Goals, dependencies on each other

• The WHAT

• Deliver quality code more quickly, reduce defects

• Customer / End User benefits is the main goal


People – The Start of Your Continuous Testing Journey

• Acknowledge this is a transformation and expect it to be

uncomfortable for people, the goal is to adapt and thrive, find

the people that are up to the challenge

• Empower developers with testing capabilities

• Developers and agile teams need to understand testing is

part of their responsibility

• Improve the relationship between tester and each developer

• Use your technically aware testers to train the developers to

build relationships and to transfer good testing skills

• Keep the teams small – easier to build those relationships

• Team goals, rewards not individual

• Encourage inter-team collaboration

• Make reports easy to access and share online


Who is Software Development Engineer in Test (SDET)

• Tester who is deep skilled in software development

• Has deep development skills

• Works closely with developers/development team

• Addresses quality at the code/component level

• Primarily focuses on testing frameworks and automation

• Focus on “Build to Run/Deploy/Perform vs Test to….”


Typical adoption challenges• People / Process / Cultural challenges

• Documented value – even if compelling -- is not sufficient to drive adoption

• Lack of agreed transformation approach and roadmap

• Lack of sense of urgency

• Org and alignment issues

• Goals and metrics issues

• Skills/People issues

• Don’t have skills in process/tools/integrations, etc

• Technical/integration complexity

• Most customer already have some DevOps tools and tool-chains

• Some new tools are sometimes difficult to integrate – from process/data perspective

• Some new tools are difficult and complex to use

• Different teams want to use different tools

• Existing assets not easy to migrate

• Value not documented

• Biz value not documented despite project level success


Survey Questions - People

Question #1: Does your entire organization universally understand the definition of continuous testing? (select one)

A. Some of our team understands the definition of CT but not all of them

B. Most of our team understands the definition of CT

C. All of our team understands the definition of CT

D. There is still confusion about what CT is exactly


Your People – Survey #1



• Question #2: How well do your developers do their own testing? (select one)

• A. Our developers perform complete and thorough unit testing

• B. Our developers occasionally do unit testing of high risk code

• C. Our developers are starting to do more unit testing

• D. Our developers don’t do much testing and the burden of quality is on QA





Question #3: How technically aware are your testers? (select all that apply)

• Both our testers and developers can code an automated test

• Our testers can collaborate with our programmers

• Our testers can articulate to their managers why an approach the development team is taking is not the best option

• None of the above





Question #4: How Agile is your team? (select all that apply)

A. Your developers and QA team members work together during the current sprint to develop and test a feature

B. Your developers code a new feature in short increments and then “throw it over the wall” for testers

C. Your testers hurriedly try to keep up with each sprint usually only having days to test a new feature within a deadline

D. Our teams are practicing test driven development (TDD), writing failing unit tests before writing code





Question #5: How well do your developers and testers communicate? (select all that apply)

A. Our testers and developers use a communication tool such as Slack

B. Our testers and developers are co-located in the same area of the building

C. We incorporate group activities and team building exercises to break down communication barriers.

D. We consistently emphasize the common goal of Continuous Testing to both developers and testers

E. Our testers and developers do not routinely communicate




Segment 1, Part 2: The Process and Technique


Testing Shifting Left, Right…Up and Down

• Architecture testing

• Design and Code quality

• TDD, BDD

• Synthetic Virtual Svs and Test

data

• Unit Performance Security –

static and binary scans

• Env config validation

• Test change impact

• Hybrid VSE and Test

Data with masking

• Performance and

Security Test of every

build

• Monitoring in test

• Subset/mask test data from

prod

• Rollback testing

• Deployment validation

• Monitoring in test

• Test Performance and

Security Test based on ops

insight

• A/B tests & canaries

• Tie monitoring data to tests

and VSE

• CX Monitoring & Analytics

SHIFT LEFT SHIFT RIGHT

• Visual Model-based Requirements

• Better collaboration with biz/BAs

• CX-based requirements validation

• Test the idea

SHIFT UP

SHIFT DOWN

• Predictive Analytics – defect/failure prediction

• Adaptive testing – dynamic test optimization

• Intelligent Automation


Understand Current State - Value Stream Mapping Real vs Perceived

Assessment of the Development, Testing & Release Processes

Development

Testing

Release/

Operations

▪ Goals & Initiatives▪ Organization, Team Structure▪ Processes▪ Development Tools▪ Code Quality ▪ Testing Expectations

▪ Testing Goals & Initiatives▪ Testing Organization & Team Structure▪ Processes & Maturity▪ Testing Tools ▪ Test Creation & Maintenance▪ Test Artifacts, Data & Automation ▪ Testing Environments & Constraints▪ API, UI, Functional, Performance,

Regression

▪ Release Train management▪ Operations team structure review▪ AS-IS Release Process review

▪ Key System Integrations ▪ Approval & Code Promotion Review▪ Current Toolset ▪ Challenges & KPIs

Organizational▪ Main Driver for change - $, Efficiency, Speed, Quality,

Risk ▪ Enterprise IT Organization Structure Review▪ Application Architecture Review – Integrations▪ Ownership & Team Responsibilities

Planning

▪ Agile Process Maturity▪ Challenges▪ Traceability▪ Key integrations with Development,

Testing & Release processes

Outcomes▪ Current State & Desired State ▪ Mutual Activity Plan▪ Prioritized Actions – High Value ideas, Low Hanging Fruit▪ Value Metric Measurement Process


Example DevOps Value Stream Assessment– Typical High-Level

Elapsed time:

Process Time:

Wait Time:

Effort:

WIP/Size:

Quality:

Elapsed time:

Process Time: Wait Time: Effort:

WIP: Quality:

Elapsed time:


WIP: Quality:

Elapsed time:


WIP: Quality:

Elapsed time:


WIP: Quality:

Requirements Development & Int Test Staging/Pre-prod Release to Prod

DevOps Process Efficiency = Σ(Process Time)/Σ(Lead Time)

Effective Flow Rate: WIP Delivered/Σ(Lead Time)

Process Flow Rate: WIP Delivered/Σ(Process Time)

Impediment Rate: Σ(Impediment Time)/WIP delivered

Feedback Process Time: Wait Time: WIP: Quality:

Deployment Process Time: Wait Time: RW Time:WIP: Quality:


Deployment Process Time: Wait Time: RW Time: WIP: Quality:


Deployment Process Time: Wait Time: ReWork Time:WIP: Quality:


Deployment Process Time: Wait Time: RW TimeWIP: Quality:

Process Time

Wait Time

Process Time

Wait Time

Process Time

Wait Time

Process Time

Wait Time

Process Time

Wait Time

Process Time

Wait Time

Process Time

Wait Time

Process Time

Wait Time

Process Time

Wait Time

Process Time

Wait Time

Process Time

Wait Time

Process Time

Wait Time

Process Time

Wait Time

Process Time

Wait Time

Process Time

Incident Remediation Feedback Loop


Improving the Continuous Testing Process

• Make automation a priority

• Do not remove all manual testing tasks, but adopt an

“automated first” mindset

• Automate everything when possible

• Code scanning upon check-in, build creation and

deployments

• Remove “repetitive work” -- Focus that on areas that will be

run repeatedly

• Set up the plumbing first so that you can let the water flow

later without any worry or concern about leaks

• Map out your SDLC and identify automation opportunities

• Where is your team? Are they ready for change?

Implement a “ You build it you own it” mentality, testing becomes a critical aspect of development


Our PoV on Continuous Testing Lifecycle Automation

Test creation and execution

automation

Test process automation

Test environment automation

CI/CD integration

Testing

Lifecyle

Automation

Automate test generation and maintenance

� Automated model-based test generation and

optimization

� Automated change impact analysis

� Automated automation

� Automated test execution

� API + GUI: Functional + perf, security

Automate the testing process

� Automated test process orchestration

� Automated analytics and dashboards

� Defect management automation

� Traceability automation

� Automated analytics and test insights

Automate test environments

� Test environment setup

� Env provisioning

� App/data deployment

� Continuous test data

� Continuous virtualization

Automate integration with CI/CD

� TDD/BDD automation

� Automated selection of tests

� Automated trigger of tests from CI/CD engines

� In-sprint automation


The Benefits of Automated Testing

• Find issues earlier in the SDLC because developers are helping with testing

• Automated testing saves a huge amount of time and allows for faster release cycles

• Code quality is improved. You can test everything on every build

• Save precious time

• Less risk because you know your code is good with every build


General Approach to solutions implementations

Think BIGStart Small

Learn fastScale thru success

Transform


Process Maturity -- Summary View

Mostly manual testing process

• Manual requirements definition and tracking

• Manual test design and execution

• Manual test data management

• SCM, manual builds, no CI

• Hand-coded virtual services

• Manual test env provisioning and deployments

• Manual release mgmt.

• Classic QA/manual tester skillset

Focus on test optimization

• Requirements modeled in MBT tool

• Automated test design

• Automated unit tests

• Some TDD/BDD

• Some GUI/API test automation (F/P/S) ~20%

• Scripted test data; synthetic test data

• Code scans

• Algorithmic performance analysis (profiling)

• Recorded virtual services

• Scripted env provisioning and deployments (incl. test data and VSEs)

• Basic CI – automated builds

• Model-based tester(MBT) and some QE skills

Focus on CI integration

• Automated TDD (BDD, ATDD)

• Automated test automation from models – GUI/APIs (F/P/S) ~40%

• Automated testing tied to CI cycles.

• Focus on In-sprint automated testing (i.e. unit, integration, app-level system tests)

• Automated test data generation – sub-setting and masking

• Data rich VSEs

• Modeled test env. definitions; automated provisioning & deployments tied to CI

• Automated release promotion within release train

• QE/SDET skills

Focus on CD Integration

• Greater level of test automation ~60%

• Full automated traceability --Requirements models tied to releases, code, test data, virtual services and performance scripts

• Full test process/lifecycle automation and integration with CD processes

• Focus on e2e automated release testing (i.e. inclusive of x-app system testing, canary and A/B testing)

• Model-driven test environment management and deployment of assets

• Deployment and release coordination across complex release trains

• SDET/SRE skills

Focus on predictive capabilities and optimization

• Full testing/CD lifecycle data collection, integration and x-lifecycle data correlation in data marts/warehouses

• Use of advanced analytics, data correlation and ML

• Adaptive test (and automation) optimization using ML on lifecycle data

• Self-healing tests assets (e.g. automation)

• Predictive testing capabilities, e.g. defect prediction, adaptive test optimization

• Self-identifying A/B scenarios

• DET, Test Data Science/Analytics skills

Maturity Level

Cap

ab

ilit

ies

L0 Initial

L1 Basic

L2 Automated

L3 Orchestrated

L4 Intelligent


Survey Questions - Process

Question #1: Have you mapped out your SDLC to identify automation opportunities? (select one)

A. Yes, we have mapped out our SDLC to identify areas of automation

B. No, we have not mapped out our SDLC to identify areas of automation

C. We started the process but are having trouble identifying which areas of automation to tackle first

D. We started the process and know exactly what areas we should automate


Your Process – Survey #2



Question #2: How mature is your test automation? (select one)

A. Very high degree of in-sprint automation (> 80%)

B. Medium degree of in-sprint automation (< or = 50%)

C. Automation lags development sprints

D. Limited automation (< 30%)





Question #3: How mature is your unit testing? (select one)

A. Very mature with TDD and BDD

B. Mature with automated functional, performance and security testing

C. Somewhat automated with code coverage around 50%

D. Limited unit testing with no clear measure of code coverage





• Question #4: How would you describe your test environment management? (select one)

A. We have virtual environments and test data on demand

B. Some level of automation in test environment setup

C. Manually created environments and test data

D. Ad hoc test environment management (i.e. dependent on ISP)





Question #5: How would you rate your current Continuous Testing Process progress? (select one)

A. We have no Continuous Testing Process in place

B. We are interested in adopting a Continuous Testing Process in the next 6 months

C. We are currently implementing a Continuous Testing Process

D. We have adopted a Continuous Testing Process and believe we are on the right track




Segment 1, Part 3: The Technology


Key CT Technology Components

CT Platform

Requirements Modeling

Functional, Perf and Sec Automation –

API/GUI

Requirements & Test

Management

Service Virtualization

Test Data Management

Test Asset Management

Test and Release

Orchestration

Other CI/CD tools

(integrations)

Typical Components of CT Platform


Technology

• The right technology is key to achieving Continuous Testing

• Manual testing approaches must give way to newer, faster technology-based solutions

• Automate Everything possible

• Companies often struggle with which tools to implement

• No one tool suite will work–Focus on Integration Platform

–Open Source and Vendor tools


Some Guidelines for Technology

• Provide testing solutions that developers will use

• Look for tools that will help you develop, test and analyze continuously

• Pick best of breed tools that work together that works in your environment

• Review available tools that support your process and prioritize implementation (open source & vendor solutions should co-exist)

• Make the best decision for the team and commit to it…

• Let’s review some tools available


Transforming the traditional agile backlog grooming & Test Case creation/maintenance with model-based Requirements

Product Owners

Developers

Testers

Product Owners

Product Owners

Happy Path:

Create User ID

Product Owners

Agile Requirements

Designer


Model Based Testing with ARDShift-left testing efforts into building a model

1. Improve Collaboration - build better requirements,

enhance communication & requirements validation

2. Automate Test Creation and Optimization – dynamic

generation based on testing goals/scenario (functional,

regression, smoke, etc.)

3. Analyze Test Coverage & Risk programmatically

4. Automatically handle the impact of requirement

changes – automated test suite maintenance

5. Identify & define data requirements via powerful data

expressions to support both positive and negative testing

6. Generate automation assets needed to execute the

generated tests in any language for any framework

7. Integration with Agile tools – Rally, Jira, TFS, ALM…


How ARD Fits into the Continuous Testing Ecosystem

REQUIREMENTS & TEST CASE

MANAGEMENT TOOLS

AUTOMATED MODELS

CA ARD leverages selenium recorder to record tests

and generate models based on those recordings

Creating

Unambiguous

Models

TEST AUTOMATION

(NOT LIMITED TO BELOW)

Create & Maintain

Automated Test

Scripts

Integrates with…

ENVIRONMENT AUTOMATION

Service

Virtualization

Load/Append into

Existing/Running

Virtual Service

DATA AUTOMATION

Test Data

Manager

Define Test Data

Attributes & Find

Existing Data

Create NEW Data /

Synthetic Data

Dynamic Test Case

Creation & Maintenance

always Up-To-Date

Provide Traceability

Coverage statistics

Agile

Requirements

Designer


Model-based Collaborative Test Design

OVERLAYING ALL LAYERS ROLES AND BENEFITS

Removes all ambiguities

MODELER (PO/BA/TESTER)

Draws and maintains business process flows

Baselines understanding across the team

TEST DATA ENGINEER

Match test data to each test case

Automatically generate synthetic data

Find and reserve data automatically

SERVICES ENGINEER

No dependency on external interfaces

Accelerate Code Development

TEST AUTOMATION ENGINEER

Automatic creation of scripts

Automatic impact analysis

Business Flows based on User Stories

Alternate Scenarios

Negative Scenarios

MODEL

Data Requirements

Data Parameters

Test Datamart connection

TEST DATA

External Interfaces Requirements

API Definitions


SERVICES

Framework linkage

Keyword linkage

Actions on the app

TESTAUTOMATION

SDET


Dependency Challenges in Distributed Systems Development

“I can’t do anything until I have everything…

and I never have everything!”

ESB

!

!

! ! !

System Unavailable

x x x

x

Invalid data Access Fees

Incomplete

Development


Addressing Constraints with Virtual Services

“I have full system access when I need it”

ESB

Agile-

Parallel

Developme

nt

Service

Virtualization


the Wind tunnel of Software


Hybrid Test Data Management

Enterprise Wide Test Data Management

Dev/Test

Controlled

(Consumers)

Model-

based

specFill in the

gaps

(Synthetic &

Cloning)

Profile,

Subset,

Mask

Allocation

and

Reservation

Provision

on Demand

Ops Controlled

(Providers)

Model, synthesize,

virtualize

Profile, subset,

mask

Test Data

Management


Differentiating between CI and CD

CI != CDCode & unit

test

Commit build, int.

test

Backlog

Design

System Test

• Deploy, test, promote

Pre-prod

• Deploy, test, promote

Prod

• Deploy, go-live, monitor

PackageCI Process

CD Process

Artifact

repo/

Containers

Artifact

repo/

Containers


Enterprise CD requirements are different than that for CI

Pipeline/App Env Config

Tool Chain Integration

Process Orchestration

Complex Deployments

Systems/user testing

Rollbacks Approval

ManagementRelease

ManagementVisibility

Governance, Compliance, Audits


Challenges doing enterprise CD with CI tools

• Everything is scripted or a plug-in

• Dependency on buggy open-source codeScripting

• Difficult to perform complex pipeline tasks like rollbacks, b/g deployments

• Difficult to manage pipeline inter-dependencies in complex release trains

Difficult to support ARA

• Debugging across 100s of scripts -- complex pipelines often brittle

• Requires Dev skills vs functional SME skills Maintainability

• App meta data (dependencies, config, properties, etc.) has to be scripted

• Pipeline does not auto-adapt based on changes. E.g.: no testing intelligence

Lack of app meta-data & Reporting

• Lack of integrated dashboards, no integrated data lake

• Difficulty in supporting compliance, identity management, audits

Lack of visibility and governance


Continuous Deployment and Release Management

QA/TEST PRE-PROD PRODUCTIONDEV

AUTOMATED

DEPLOYMENT

Continuous Delivery DirectorOPTIMIZE

PIPELINE

Mo

del-

based

Rele

ases:

CA

CD

D

PLAN & MANAGE

RELEASES

Mo

del b

ased

D

ep

loym

en

ts:

CA

AR

A


CI & CD solutions can co-exist

CI: As-code

• Developer friendly

• Less complexity

• Lower controls and governance

CD: Low-code

• Functional specialist/management friendly

• High complexity

• Higher controls and governance

DevOps Value Stream Management Platform (CDD)

Continuous Testing


Continuous Testing Platform considerations▪ Functional Testing

▪ API, UI, Security, Regression ….

▪ Scriptless Automation enable everyone to automate

▪ Record & generate script

▪ Performance & Load testing basic levels and extreme levels

▪ Test data available for execution

▪ Virtual Services/Mock Services for constraints as needed

▪ API Monitoring for modern architectures

▪ Shared Platform for execution

▪ Reporting & History

▪ Compare executions

▪ Drive Load internal & external (cloud) & Cloud Migrations


Continuous Testing Platform

Quality Analytics & Insights

• Shift-Left, Intelligent Test Automation

360 Degree API Test,

Reuse Scripts for

Monitoring

Eliminate Test Env Delays

with Integrated Test Data

and Virtual Services

Scriptless

Automation

Generate Tests

& Automation

Functional UI,

API, Load

Testing

Built-On Open Source

• Evolution of BlazeMeter, Industry’s #1 Shift-Left Performance Testing SaaS Solution

• UI/Functional Testing, Service Mocking and Virtualization, API & GUI Testing, Monitoring

• Available as SaaS & Private Cloud

• Multi-Cloud Support (AWS, Azure, GCP)

• Integrated Service Virtualization & Mock Services

• Script Conversion – shiftleft.blazemeter.com


Defining the end to end Value Stream - DevOps

Ideation

Customer

Experience

Continuous Testing

Test Data /Synthetic Data

Test CaseCreation

PerformanceTesting

Mocking& Virtual Services

Continuous Integration Operations

CapacityManagement

WIP

BuildCode

Synthetics

ServiceManagement

Monitoring

Transformation Intelligence

Business

Strategy

XLHigh Level

Planning

Backlog

Features

Stories

BehaviorModeling

Test Scripting

Test Management

Continuous Delivery

Release

Orchestration

CD Automation

API Governance &Management

Quality Management

Test Planning

Tracking

OKRs

Platform / Cloud Management (Kubernetes)

Platforms


DevOps & Continuous Testing | Reference Architecturefit within a broader DevOps ecosystem

Continuous Feedback & Collaboration

Continuous Integration

Monitoring

Portfolio & Project

Prioritization Service Management

Security

Build Business

Case Requirements

Modeling & Management

Ideas

UI

Performance Testing

User Acceptance

Testing

Test Data

API

APIs

Customer


Continuous Deployment

Test CaseCreation

CodeBuild

Security

Virtual Service

Continuous Delivery

Planning & Requirements

OperationsCollaboration Continuous

DeploymentContinuous Integration

Continuous Testing Development

API Management & Testing

Source Code Repository

Artifact Repository

Project & Portfolio Planning

Requirements Management

Deployment Tools

Requirements Modeling

Testing Efficiency Tools

Container Orchestration

Environment Configuration

Clarity

PPM

Agile

Requirements

Designer

Continuous Delivery Director

Service

Virtualization

DXI

APMTest Data

Management

Planning

Operations

Development

Continuous Testing

Performance & Functional Testing

Endevor

Automic

Automation

Shift LeftContinuous Testing

Runscope &

API Management

DevOps IntelligenceAI-Powered Predictive Insights & Feedback


Survey Questions - Technology

Question #1: Which tools are you currently using to help you achieve continuous testing? (select all that apply)

A. Requirements definition and modelling (CA, Tricentis, etc.)

B. Service Virtualization (CA, Smartbear, Tricentis, etc.)

C. Test data management (CA, Delphix, Tricentis, IBM, Informatica, etc.)

D. Pipeline orchestration tools (CA, Xebia, etc.)


Your Technology – Survey #3



Question #2: What tools do you feel are absolutely vital to achieving continuous testing at your company? (select all that apply)

A. Requirements planning and modelling

B. Test data management

C. Service Virtualization

D. Pipeline orchestration

E. Performance testing

F. UI testing

G. API testing

H. Other (please specify)





Question #3: Which continuous testing tools are you currently evaluating? (select all that apply)

A. Requirements modelling

B. Service Virtualization

C. Test data management

D. Pipeline orchestration

E. Performance testing

F. API testing

G. Other (please specify)





Question #4: What do you feel are the biggest obstacles for your team to adopt new continuous testing tools? (select all that apply)

A. Budget constraints

B. Cultural challenges that don’t permit us to change

C. Lack of desire by senior management

D. Can’t teach an old dog new tricks (developers don’t want to test…)





Question #5: What do you consider the best way for your team to learn about continuous testing tools? (select all that apply)

A. Webinars and webcasts

B. White papers and data sheets

C. Live product demonstrations

D. Industry analysts

E. Other (please specify)




Technology Maturity Summary View

Ad-hoc tools with some integration

• Requirements definition tools (e.g. CA-Agile Central, Jira)

• Spreadsheets/docs for test design, test data and test env specification

• Mocking/stubbing tools

• IDE, SCM tools and build tools

• Spreadsheet-based release tracking

Integrated Test Platform

• Requirements modeled in CA-ARD, tied to AC/Jira

• Unit testing tools such as: Junit

• Other test automation tools such as Selenium, UFT

• Synthetic test data using CA-TDM and CA-ARD

• Basic virtual services with CA-SV, CA-SVCode

• API testing CA-DevTest, SOAP-UI etc.

• Perf testing using jMeter, CA-BZM, etc

• Test Environment manifests defined in config tools (Puppet etc) tied to deployment tools (CA-CDA)

• CI tools like Jenkins

• App release tracking in CA-CDD

Test Platform Integrated with CI platform

• TDD tools such as: Gherkin, Cucumber

• Tie CA-ARD models to automation tools (e.g. Selenium)

• Integration w/ APIM (such as CA-APIM) tools; API monitoring tools such as CA-Runscope

• Sub-setting and masking of test data using CA-TDM

• Data rich virtual services (CA-SV) with dynamic data handling (CA-TDM)

• All test tools integrated into CI platform

• CA-CDA deployments of dependencies within release train

• CA-CDD release tracking across apps in release train

Test Platform Integrated with CD platform

• Tie CA-ARD with SCM systems like Git, CA-SV, CA-TDM, CA-BZM

• A/B testing tools like Optimizely

• X-release train artifact tracking using CDD and CDA

• All test tools integrated into CD platform

Test Platform Integrated with Analytics/ML platform

• Use CDD/CDI for data aggregation across lifecycle, use of intelligent analytics

• Predictive analytics and ML tools and frameworks

• Machine learning for test optimization (ARD) based on lifecycle data (CDD)

• All test tools integrated into analytics/ML platform with automated centralized data collection

Maturity Level

Cap

ab

ilit

ies

L0 InitialL1 Basic

L2 Automated

L3 Orchestrated

L4 Intelligent


CT Capability Detail by Maturity LevelKey Capability Level 0 Initial Level 1 Basic Level 2 Automated Level 3 Orchestrated Level 4 Intelligent

Requirements

management

Manual requirements

definition and tracking

Model-based requirements

definition; manual traceability

mapping to stories/features, tests,

defects

Automated traceability management to

test assets; model versioning

Automated traceability management to

CD lifecycle assets (e.g. code, builds,

environments, releases, etc)

Automation requirements verification

based on CX analytics

Test design and

management

Manual test design;

manual tracking in

spreadsheets

Automated test design and

optimization from model

Some automated tracking; change-

impact testing; some risk-based

prioritization

Fully automated test tracking tied to CI

processes. Test model versioning;

automated change impact testing tied to

code changes

In-sprint testing

Full model-based integration of all test

assets.

Fully automated test tracking tied to

CD processes.

Integration of tests with monitoring

Adaptive test optimization based on ML

on past data

Test automation Mostly manual test

execution

Automated unit tests

Basic GUI/API test automation (<

20%)

API structural testing

Automated test automation (~40%);

execution integration with CI processes;

in-sprint test automation; automated API

monitoring

Full test process Lifecyle

automation(~60%) integrated with CD

processes; release lifecycle test

automation (incl.x-app system tests,

canary and A/B tests)

Self-healing automation using ML,

Test data

management

Manual test data

definition

Scripted test data generation;

synthetic test data

Automated test data with sub-setting,

masking, reservations; integrated with CI

processes; virtual test data

Automated test data provisioning

integrated with CD processes and tools

Predictive test data computation using

analytics

Virtual Services Basic mocking/

stubbing

Recorded virtual services

Synthetic virtual services created

with synthetic data

Data rich virtual services

integration w/ API Registry

Automated VSE provisioning integrated

with CD processes and tools;; VSE

updates based on production usage

Self-healing VSEs

Code quality Manual unit tests; code

reviews;

Automated basic unit tests, code

quality scans, SAST/DAST; code

profiling

Automated TDD/BDD/ATDD tied to CI

processes; automated unit perf tests.

Model versions tied to code versions

Full support for dark deployments Code promotion based on predictive

analytics; predict failures based on code

quality analytics

CI/CD/Release

Management

Integration

No CI. Use of SCM

and manual builds

Basic CI integration, e.g. Automated

builds tied to automated unit tests

Full CI lifecycle integration within release

train. In-sprint tests triggered from

successful builds. Monitoring during perf

tests

Full integration into CD lifecycle

processes with orchestration across

release trains; full monitoring in all test

environments

Automated release risk/quality prediction

Test

environment

and deployment

management

Basic – statically

allocated

environments; manual

deployments

Scripted provisioning/updates to

env. Scripted deployments

Model-driven test environments (with test

data, VSEs, test suites) and deployment

rules; integrated with CI process

automation

Integrated with CD automation; support

for dark deployments; A/B and canary

tests

Self-healing/sizing test environments

Test analytics

and intelligence

Basic metrics tracking;

trend analysis

Automated descriptive analytics of

metrics in silos

Some diagnostic analytics using manual

correlation of data

Automated diagnostics analytics using

automated correlation

Full testing/CD lifecycle data

warehousing; prescriptive analytics and

ML support


Segment 2: MegaCorp Case Study


MegaAction Corp – A Case Study

• Where MegaAction Corp is at in their journeyIdentifying

• Goals, gaps, bottlenecks across the end to end process/organizations

Value Stream Mapping exercise

• For MegaAction testers and developersBuilding a game plan

• Of the MegaAction Corp CT implementationMeasuring success

• With MegaAction Corp senior managementSharing results

• From MegaAction Corp’s journeyKey learnings


MegaCorp Case Study

Assess people, process,

technology

Develop mission/vision/goals

statement

Build a COE for a solution

framework – P/P/T

Start with pipeline automation,

then quality improvement

Continuous “Shift Left” Testing

Continuous Improvement

SOLUTION APPROACH

1-2 week release cycle plus

on-demand releases

85% test coverage

75% reduction in regression

test effort

Cost savings >.5M in 1 year

RESULTS

• Manual testing (4 hrs for

regression 30K+ hrs overall

testing)

• Limited test coverage

(<44%)

• Limited non-functional testing

• Low release velocity

(monthly)

• Unstable environments

• Manual release process

• Limited collaboration

• One Click to Production Goal

CHALLENGES


MegaCorp: Looking at People, Process, Technology


MegaCorp – Bringing People Together


MegaAction Corp – Focus Areas

• Automate the Deployment & Release pipeline

• Automate Environment Builds

• Enable Model Based Testing

• Automate Test scenarios

• Automate Test Data provisioning for builds

• Integrate Test execution into Pipeline


MegaCorp – The Process and Technology


MegaCorp State 1 year later


MegaCorp Results


MegaCorp Results


Wrap-up


Thank You for Joining us Today

Documents

The Mega Actionable Continuous Testing Workshop