Vladimir Primakov - Qa management in big agile teams

QA Management in BIG Agile teams

Volodymyr Prymakov

Speaker info

Volodymyr Prymakov, UkraineSenior QA Manager at Ciklum, TCoE

Head of Performance QA Unit at TCoE

•14 years in QA

• 45 projects experience

• Certified: ISTQB Advanced Test Analyst &

Manager, ICAgile CP, SAFE

/in/vladimirprimakov/

Agenda

1. Big Team, What is it?

2. Process (Development and QA)

3. Infrastructure

4. Collaboration

5. Transparency

6. QA Team organization

7. Automation

8. Questions

What is a big team?

What is a big team?

Team

What is a big team?

Team Big team? - Expected!

What is a big team?

Team Big team? in Reality….

What is a big team?

Most of the problems in a big team

relates to

cross-team work and

common dependencies

Big agile team is a team of >50 people

1 PO

1 Tech Lead

5 Developers

2 QAs

1 PO

1 Tech Lead

5 Developers

2 QAs

1 PO

1 Tech Lead

5 Developers

2 QAs

1 PO

1 Tech Lead

5 Developers

2 QAs

1 PO

1 Tech Lead

5 Developers

2 QAs

1 PO

1 Tech Lead

5 Developers

2 QAs

PO Manager

Dev Manager

Architectors

QA Manager, Leads

SMSMSM

Program Manager

Release Manager

Dev. Teams

Leadership Team Release train

Products

1. Big (A lot of functionality or sub-products)

1. Complex (Architecture and infrastructure)

1. (Quite often) Monolete architecture or

interdependable components

1. A lot internal and external integrations.

2. A lot of functionality interdependencies.

High Regression risk!

3. Many end-users (sometimes, in many

countries)

PROCESS

Process - Overlapped Releases

Supported by the same teams

MASTER BRANCH

STABLE BRANCH

Process - Overlapped Releases

Supported by the same teams

STABLE BRANCH

MASTER BRANCH

Process - Overlapped Releases

Supported by the same teams

MASTER BRANCH

Bug Hotfixes

STABLE BRANCH

Process - Overlapped Releases

Supported by the same teams

MASTER BRANCH

Bug Hotfixes

STABLE BRANCHRegression Problems

Process - Overlapped Releases

!!!Delay because of merging,

regression and other problems

Supported by the same teams

Process - Overlapped Releases

!!!Delay because of merging,

regression and other problems

Supported by the same teams

Overlappin

g a

ctivitie

s

Process - Overlapped Releases

!!!More Delay

Supported by the same teams

Overlappin

g a

ctivitie

s

Overlappin

g a

ctivitie

s

Process - Overlapped Releases

!!!More Delay

SHIFTED RELEASES

Supported by the same teams

Overlappin

g a

ctivitie

s

Overlappin

g a

ctivitie

s

Process - Overlapped Releases

Supported by the same teams

PROBLEMS:

● No Buffer for human factor and

unpredictable problems

Process - Overlapped Releases

Supported by the same teams

PROBLEMS:

● No Buffer for human factor and

unpredictable problems

● Overlapping activities

● Big context switching

Process - Overlapped Releases

Supported by the same teams

PROBLEMS:

● No Buffer for human factor and

unpredictable problems

● Overlapping activities

● Big context switching

● Complicated branching

strategy and risky hot-fixing

process

● Not compatible with

Continuous Delivery

Alternative Process - Overlapped Releases

Regression and release support by Another team

PROBLEMS:

● Lack of new functionality

(implementation) knowledge

● Knowledge Transfer required

● Not enough expertise for bug

fixing

● Extra Collaboration needed

● Not Enough Capacity

● Motivation problem

System Dev & QA Team

STRAITFORWARD Release Pipeline

MASTER BRANCH

NOW

STRAITFORWARD Release Pipeline

MASTER BRANCH

FUTURE

STRAITFORWARD Release Pipeline

BENEFITS:

● SIMPLE and STRAIGHTFORWARD

● NO WASTE of TIME FOR

KNOWLEDGE TRANSFER

● NO CONTEXT SWITCHING

● RELEASES EVERY 2 WEEKS

● COMPATIBLE with CONTINUOUS

DELIVERY

Process Effect on QualityNEW PROCESS

Bug Rate

Story

Points

Delivered

OLD PROCESS

STILL GOOD VELOCITY

30-40% LESS BUGS

STABILIZATION Period

● No Feature Merging in

master or release

branch

● Only Blocker and

Critical bugs fixing

● Regression testing

● Bug Retest.

Release management FOR OTHER PRODUCTS

Monitor Release

Weight Impact

Prepare in advance

Test and Retest it!

Releases for Integrated Systems (internal and 3-rd parties)

Late Code Merging

Late code merging leads to bugs, no time for testing and release delays!

Late Code Merging

To Avoid Merging Problems:

● Size stories optimally

Late Code Merging

To Avoid Merging Problems:

● Size stories optimally

● Merge as often as possible

Late Code Merging

To Avoid Merging Problems:

● Size stories optimally

● Merge as often as possible

● Early Deliver and Test (Automatically)

Late Code Merging

To Avoid Merging Problems:

● Size stories optimally

● Merge as often as possible

● Early Deliver and Test (Automatically)

● Deadline for merging + buffer for testing

Late Code Merging

To Avoid Merging Problems:

● Size stories optimally

● Merge as often as possible

● Early Deliver and Test (Automatically)

● Deadline for merging + buffer for testing

● Communicate, manage, and revert risky merges.

Regression testing approach (e.g.)

Country 2

Country 1

Country 3

Country 4

Markets

Regression testing approach (e.g.)

Fast

Feedback about product

quality state.

Scope:

P1 Test cases

Country 2

Country 1

Country 3

Country 4

Markets Regression Cycle 1

Mostly p1, p2

bugs

Regression testing approach (e.g.)

Fast

Feedback about product

quality state.

Scope:

P1 Test cases

Country 2

Country 1

Country 3

Country 4

Markets Regression Cycle 1 Regression Cycle 2

Cross-Platform

Compatibility

testing

Scope:

Exploratory

testing

Mostly p1, p2

bugs

Mostly p3 or lower

priority bugs

Regression testing approach (e.g.)

Fast

Feedback about product

quality state.

Scope:

P1 Test cases

Country 2

Country 1

Country 3

Country 4

Markets Regression Cycle 1 Regression Cycle 2

Cross-Platform

Compatibility

testing

Scope:

Exploratory

testing

Regression Cycle 3

Validate

Quality after

Bug-fixesverifications

Scope:

P1, P2 test

cases

Mostly p1, p2

bugs

Mostly p3 or lower

priority bugs

p1, p2, p3 or lower

priority bugs

Regression testing approach (e.g.)

Country 2

Country 1

Country 3

Country 4

Markets Regression Cycle 1 Regression Cycle 2

All other

platforms

Regression Cycle 3

Platform 2

(Chrome)

Platform 1 (IE)

Platform 1 (IE)

Platform 2

(Chrome)

Platform 1 (IE)

Platform 2

(Chrome)

Platform 2

(Chrome)

Platform 2 (IE)

Cross Platform Testing approach - Sprint N

Regression testing approach (e.g.)

Country 2

Country 1

Country 3

Country 4

Markets Regression Cycle 1 Regression Cycle 2

All other

platforms

Regression Cycle 3

Platform 1 (IE)

Platform 2

(Chrome)

Platform 2

(Chrome)

Platform 1 (IE)

Platform 2

(Chrome)

Platform 1 (IE)

Platform 1 (IE)

Platform 2

(Chrome)

Cross Platform Testing approach - Sprint N + 1

Regression testing approach (e.g.)

Type of Test Scope Outcome

Smoke End-to-end flow (most

popular booking)

Blocker bugs

Acceptance Alternative End-to-End

flows

P1 and P2 bugs

UAT User Acceptance Test

scenarios

P1, P2, P3

Some bugs may be missed

Regression P1 P1 test cases P1 and P2 bugs

Regression P2 P2 test cases P3 bugs

Exploratory Exploratory test cases P1, P2, P3

Testing Order!

System Intergration Testing

● Obligatory test integrations with real life internal and 3d party services

● Do confirmation testing for the rest of functionality if needed (Exploratory, Cross-platforms, or more wider testing)

Cross team Bug Escalation

Priority

p3, p4, less

QA Coordinator /

Release Coordinator

Product

Backlog

Cross team Bug Escalation

Priority

p3, p4, less

Priority

p1, p2

QA Coordinator /

Release Coordinator

Product

Backlog

Assign to current FixVersionTESTER

Cross team Bug Escalation

Priority

p3, p4, less

Priority

p1, p2

TEAM

QA Coordinator /

Release Coordinator

Product

Backlog

Assign to current FixVersionTESTER

ASSIGN to A CERTAIN TEAMAssign to Developer and Resolve Quickly!Is this bug produced by the team?

Does the team has expertise to fast fix it?

Or does the team has capacity for bug fixing?

Cross team Bug Escalation

Priority

p3, p4, less

Priority

p1, p2

TEAM

QA Coordinator /

Release Coordinator

ASSIGN to A CERTAIN TEAMAssign to Developer and Resolve Quickly!

Product

Backlog

Assign to current FixVersionTESTER

Cross team Bug Escalation

Priority

p3, p4, less

Priority

p1, p2

TEAM

TECH LEADS

QA Coordinator /

Release Coordinator

ASSIGN to A CERTAIN TEAMAssign to Developer and Resolve Quickly!

Product

Backlog

Assign to current FixVersion

Which team produced the bug?

Which team has better expertise to fix it?

Which team has more capacity for bug fixing?

TESTER

Cross team Bug Escalation

Priority

p3, p4, less

Priority

p1, p2

TEAM

TECH LEADS

QA Coordinator /

Release Coordinator

ASSIGN to A CERTAIN TEAMAssign to Developer and Resolve Quickly!

Product

Backlog

Assign to current FixVersionTESTER

Cross team Bug Escalation

Priority

p3, p4, less

Priority

p1, p2

TEAM

TECH LEADS

QA Coordinator /

Release Coordinator

ASSIGN to A CERTAIN TEAMAssign to Developer and Resolve Quickly!

Product

Backlog

Assign to current FixVersionTESTER

SCRUM MASTERS

Which team has more capacity for bug

fixing?

Cross team Bug Escalation

Priority

p3, p4, less

Priority

p1, p2

TEAM

TECH LEADS

QA Coordinator /

Release Coordinator

ASSIGN to A CERTAIN TEAMAssign to Developer and Resolve Quickly!

Product

Backlog

Assign to current FixVersionTESTER

SCRUM MASTERS

Cross team Bug Escalation

Priority

p3, p4, less

Priority

p1, p2

TEAM

TECH LEADS

QA Coordinator /

Release Coordinator

ASSIGN to A CERTAIN TEAMAssign to Developer and Resolve Quickly!

Product

Backlog

Assign to current FixVersionTESTER

SCRUM MASTERS

POs

Which team has lower priority functionality

which can wait for the sake of bug fixing?

Cross team Bug Escalation

Priority

p3, p4, less

Priority

p1, p2

TEAM

TECH LEADS

QA Coordinator /

Release Coordinator

ASSIGN to A CERTAIN TEAMAssign to Developer and Resolve Quickly!

Product

Backlog

Assign to current FixVersionTESTER

SCRUM MASTERS

POs

Cross team Bug Escalation

Priority

p3, p4, less

Priority

p1, p2

TEAM

TECH LEADS

QA Coordinator /

Release Coordinator

ASSIGN to A CERTAIN TEAMAssign to Developer and Resolve Quickly!

Product

Backlog

Assign to current FixVersionTESTER

SCRUM MASTERS

POs

FACILITATE

Align QA approaches across teams

Align QA approaches across teams

• DoD for test cases

• DoD for Automation scripts

• Test suite Structure

• Test plan and run structure in test management

system

• Regression approach

• Bug Reporting

• Release readiness reporting

• and many others

Agreeing project process/approach

Agree cross-team approaches/processes on all levels

Agreeing project process/approach

Agree cross-team approaches/processes on all levels

Involve all concerned parties in the discussions

Agreeing project process/approach

Agree cross-team approaches/processes on all levels

Involve all concerned parties in the discussions

Align cross-team procedures

Agreeing project process/approach

Agree cross-team approaches/processes on all levels

Be tolerant and persistent

Involve all concerned parties in the discussions

Align cross-team procedures

Agreeing project process/approach

Agree cross-team approaches/processes on all levels

Be tolerant and persistent

Document agreements

Involve all concerned parties in the discussions

Align cross-team procedures

Infrastructure

Infrastructure problems

2h of infrastructure

downtime for the team in

size of 50 Devs and QAs

may cost 12.5 man days…

(several thousands USD)

Test Environments Requirements

SIMILAR to PRODUCTION

AUTOMATED and CONTROLLED

FAST and POWERFUL

LESS INTERDEPENDENT

ROBUST and STABLE

Typical Infrastructure/environment problems

Problem

Essential difference between environments (configuration, integrations, data) leading to leaks of bugs to production

Uncontrolled changes and manual interventions

Configuration problems.

Slow and overloaded environment. Usage conflicts.

Integration issues (external and internal).

Services and DB compatibility.

Unreliable or absent test data.

Unpredictable downtimes. Connectivity and access issues.

Solution

Minimize the difference. At lease 1 test environment should be 99% identical to production.

Automate Everything. Apply change control procedure. Limit unauthorized access. Notify about changes in advance!

Increase Capacity. Build several test environments.

Don’t save on the environment. Use more powerful network.

Controlled 3d party releases. Component decoupling. Mocks usage. Backward compatibility.

Mocking Data, Standard data sets usage, Agreeing test data provision from 3-d parties, etc.

Plan B: Backups, alternative suppliers etc.

Environments (e.g.)

Testing Owners

Integrations

Hardware capacity

DB

Devops should be part of teams

● ~1 Devop for 2 teams

● Open communication. Participation at

daily scrums and other sync-ups

● Proactive resolution of ongoing

infrastructure and environment

problems

● Devops backlog prioritization for sake

of teams’ needs

Cross team Collaboration

Strategic Planning

Plans are hidden from teams

Strategic Planning

Strategic Planning

Strategic planning helps to build the right product

Product increment and other initiatives for a Quarter/Half-a-year

Sprint Planning (Devs)

Teams

Grooming

Sprint Planning (Devs)

Teams Cross-Team

GroomingTech-leads

Preplanning

Sprint Planning (Devs)

Teams Cross-Team

Grooming

Common Risks

and

Dependencies

Identified

Tech-leads

Preplanning

Sprint Planning (Devs)

Teams Cross-Team Teams

GroomingSprint

Planning

(adjusted)

Common Risks

and

Dependencies

Identified

Tech-leads

Preplanning

Sprint Planning (Devs)

Teams Cross-Team Teams

GroomingSprint

Planning

(adjusted)

Common Risks

and

Dependencies

Identified

Sprint

Start

Tech-leads

Preplanning

Sprint Planning (QAs)

Cross-Team

QA Leads

Preplanning

Sprint Planning (QAs)

Sprint Planning (QAs)

Cross-Team

Additional sprint

testing scope

identified

QA Leads

Preplanning

Sprint Planning (QAs)

Cross-Team Teams

Sprint

Planning

(adjusted)

Additional sprint

testing scope

identified

QA Leads

Preplanning

Sprint Planning (QAs)

Cross-Team Teams

Sprint

Planning

(adjusted)

Additional sprint

testing scope

identified

Sprint

Start

QA Leads

Preplanning

Sprint Planning (QAs)

Cross-Team Teams

QA Leads

Preplanning

Sprint

Planning

(adjusted)

Additional sprint

testing scope

identified

Sprint

StartCross-Team

QA Sprint

Scope

Review

Next Slide =>

QA Sprint Scope Review & Planning

Participants

QA Manager,

QA Leads,

1 QA from

every team

QA Sprint Scope Review & Planning

Participants Goals

1. Review Sprint scope (Features)

2. Define Dependencies, risks, and impact

3. Adjust scope and type of testing

4. Define additional testing activities

5. Identify impact on automation and

performance scripts

QA Manager,

QA Leads,

1 QA from

every team

QA Sprint Scope Review & Planning

Participants Goals

1. Review Sprint scope (Features)

2. Define Dependencies, risks, and impact

3. Adjust scope and type of testing

4. Define additional testing activities

5. Identify impact on automation and

performance scripts

-------------------------------------------------

1. Discuss QA Automation coverage

increment and other achievements

2. QA automation scope adjusting

QA Manager,

QA Leads,

1 QA from

every team

QA Sprint Scope Review - Outcomes

Regular Team & Scrums

Teams Cross-Team

ScrumScrum of

Scrums

● Cross-team

Dependencies,

impediments, and help

needed

● Environmental Issues

● Overall Sprint Scope

covering (once)

● Are we ready for Code

Freeze?

● Discussing release

stoppers: Blocker and

critical bugs

Auditory:

Release Manager,

Scrum Masters,

Tech Leads,

QA Leads,

Devops Lead

Regular Cross Team Syncups

Dev Manager,

QA Manager,

Tech Leads,

DevOps Leads,

Release Coordinator

● Strategic Plans,

● Common

Approaches,

● Important Ongoing

Activities,

● Other Common

Questions and

Problems

Readiness Checkpoints

Ready for Regression?

Ready for Stage Testing?

Ready for UAT by Business?

Ready For Production?

Regularly Sync-ups

on Product Readiness!

Release Manager,

Scum Masters,

PO Manager,

QA Leads/QA Coordinator,

(POs),

(Tech Leads)

Lessons Learned

Teams

Team

Retro

Sprint/Release

Finishes

Lessons Learned

Teams Cross-Team

Team

RetroQAs

Retro

Sprint/Release

Finishes

Lessons Learned

Teams Cross-Team

Team

RetroQAs

Retro

Release

Retro

Sprint/Release

Finishes

Cross-Team

Knowledge sharing

Spread Knowledge among the project!

!!!Record videos and/or document the stuff

● Business Domain

● Product Functionality

● Overall Architecture

● Component Technology

● 3-d party components

● Common approaches, etc.

Minimize frequency of sync-ups

Meet as often

as it benefits,

but do not

overdo

Minimize frequency of sync-ups

Meet as often

as it benefits,

but do not

overdo

!!!Avoid long

meetings at

essential

sprint/release

phases

Transparency

Scope understanding

Initiative/Epic level

Know and

align with

the

project

roadmap

Scope understanding

Sprint/Release level

Use Fix Version for

features and bugs

in advance to

track overall

release scope

(cross-team)

Realtime quality boards

Monitor Quality on daily bases or more often

1. Dynamic Jira Dashboards 2. Kanban Boards for Blocker and Critical bugs

Testing reporting

1. Report/Escalate on Blocker and

Critical bugs by a necessity.

2. Report regression testing results

on daily bases.

3. Involve all concerned parties in

the report.

4. Build quality

awareness/transparency on the

project!

Quality trends

Monitor and analyze Quality trends.

Apply Corrective actions if needed.

QA Team

Typ

ica

l QA

Te

am

Str

uc

ture

QA Manager

QA Manual Lead

Team 1

Senior/Middle QA

Middle/Junior QA

Team 2

Senior/Middle QA

Middle/Junior QA

Team 3

Senior/Middle QA

Middle/Junior QA

QA Manual Lead

Team 4

Senior/Middle QA

Middle QA/Junior QA

Team 5

Senior/Middle QA

Middle/Junior QA

Team 6

Senior/Middle QA

Middle/Junior QA

QA Manual Lead

Team 7

Senior/Middle QA

Middle/Junior QA

Team 8

Senior/Middle QA

Middle/Junior QA

Team 9

Senior/Middle QA

Middle/Junior QA

QA Automation Lead

Core Automation Team

Senior/Middle Auto QA

Middle Auto QA

Middle/Junior Auto QA

QA Team Typical Roles

Defines and aligns QA Approaches on the project,

Strategic Planning and Reporting,

Monitor and adjust ongoing QA activities,

Resource planning, recruitment, competences, etc.

Escalation questions

QA Manager

QA Approaches implementation,

Cross team ongoing QA work planning and coordination in a sprint (release),

Regular QA reporting,

People management in their teams (Mood, PDPs), etc. QA Lead

Building and aligning QA automation approaches and expertize on the project, Organizing the corresponding Knowledge Sharing,

Leading automation Framework development and support,

Automation team management.

QA Automation

Lead

QA Component Lead role

1. Main knowledge holder / expert in the product area.

2. Responsible for the quality of the product area. Regularly

monitor its quality. Escalate problems if needed.

3. Define dependencies and quality risks in the area. Inform

others about them.

1. Defines manual and automation testing scope for the product

area.

2. Monitor and analyze automation testing results for the

product area. Execute automation tests if needed.

1. Plans and organizes regression and other kind of testing for

the product area.

QA Coordinator Role

1. Organize QA syncups and planning meetings.

2. Plans and organizes cross team testing activities

a. regression testing.

b. testing at production.

c. etc.

3. Monitor overall product quality state.

4. Escalate Blocker and Critical bugs if needed.

5. Prepare and send quality reports.

6. Play a representative role in release readiness

meetings.

Coordinate cross-team testing activities in a release.

Cycle the role

between QA leads.

Delegation on All levels

Other roles:

● QA Product Market Lead

● Platform statistics

management

● Devices Management

● A/B Testing management

● Payment Card and refund

Management

● Etc.

Automation

Why Automation?

Cross team

work

Regression Risk Automationprevents the risk

Requirements to QA automation

All Levels Correct order Cross Platform

Requirements to QA automation

Sufficient

Coverage

Stable

and

Trustful

High Speed

Requirements to QA automation

FrequentExecuted on all

environments

Aligned with Manual QA

Manual QAAutomation QA

Aligned with Manual QA

Integration with Test Management system

Non-Functional Automation

Parting Words

Questions