Embed Size (px)
Citation preview
Seven Best Practices for
Accelerating Continuous Testing
Marc Hornbeek
Senior Solutions Architect
2 Spirent Communications
“There is no continuous delivery
without testing”
Dev Ops
3 Spirent Communications
Fast CT is required to race
through the many test steps in
the pipeline from Pre-Flight
through to Delivery
4 Spirent Communications
CT infrastructure must
be stable and accurate
because test failures at
any point in the pipeline
cause delays or roll-
backs
5 Spirent Communications
Consequences of
poor CT practices
80% failure rate when
using current tools
IDC Fortune 1000 survey,
December 2014
6 Spirent Communications
Accelerated Continuous Testing
Best Practices Principles
Comprehensive
&
Results
Test fast
&
Don’t Stop
Powerful
Relevant
7 Spirent Communications
CT best practices are across the pipeline
“Shift Left”
8 Spirent Communications
9 Spirent Communications
Focus on customer relevant defects
not quantity of defects
“Quality is everyone’s Responsibility.”
Dr. W. Edwards Deming
10 Spirent Communications
Pre-Flight Team-work Design for CT
CT End-to-End Culture
11 Spirent Communications
CT Process design
Fault tolerant
Obfuscation
Intelligent dashboards
CT-Ready Infrastructure
12 Spirent Communications
CT-Ready Tools
Restful APIs
Cache and pipeline
Virtual and physical
Program agnostic
Large scale
13 Spirent Communications
CT Tools Integration
Pre-Flight
CI Tests
Integration
Regression
System
Deployment
14 Spirent Communications
Orchestrate Test Topologies
Physical and Virtual
Utilization
measurement
Topologies
Test Automation
15 Spirent Communications
Accelerate Test Execution
Powerful servers
Test design
Pipeline
Scale horizontally
Thresholds
Dynamic test schedules
16 Spirent Communications
Fast Relevant CT Analytics
Continuous Monitoring (CM)
Relevance
Dashboards, Alerts, logs,
thresholds
Aggregation
Snapshots and trends
Telescope for diagnostics
17 Spirent Communications
Case Study
360 developers, six sites
36 MLOC+, 100+ features
86 build targets
complex interdependencies
200+ topologies
12K test cases
80+ deployment packages
(physical and virtual) 1,280,000,000
test possibilities
18 Spirent Communications
The primary contributing
factor was CT and CM !
CT Case Results
Metric
Major release
(#months)
Minor release
(#weeks)
# Features
Defects
Integrations / day
Tests / day
Automated tests
After
3
2
150
10
100
10
85%
Before
6
4
113
1260
0.5
0.3
5%
19 Spirent Communications
Best practices tool Best Practices Assessment Tool
simplifies the assessment process.
20 Spirent Communications
Summary
7 CT best practices emphasize
speed, don’t stop,
comprehensive and relevant.
Benchmark against Best
Practices Blueprint and
DevOps Assessment.
Don't fall into the “do-CT-
yourself / NIH” trap!
“There is no continuous delivery
without testing”
21 Spirent Communications
www.spirent.com/clear
1) CT Blueprint white papers
2) CT eBook
3) CT Assessment Tool
Take-aways
http://www.spirent.com/go/AMR
22 Spirent Communications
How can Spirent help?:
Test tools
CT Orchestration
Lab Management
Professional Services
23 Spirent Communications
© Spirent Communications, Inc. All of the company names and/or brand names and/or product names and/or logos referred to in this document, in particular the name
“Spirent” and its logo device, are either registered trademarks or trademarks pending registration in accordance with relevant national laws. All rights reserved.
Specifications subject to change without notice.
spirent.com
Thank you
Seven Best Practices for Accelerating Continuous Testing
Marc Hornbeek
Senior Solutions Architect
Twitter: mhexcalibur
