Case Study - Implementing DevOps for a complex hardware/software-based network product

Case Study - DevOps Cost-Effectively Secured Quality and

Time-to-Market for a Complex

Hardware/Software Network Product

Marc Hornbeek

Senior Solutions Architect

2 Spirent Communications

Before and After

BEFORE DevOps AFTER DevOps


Takeaways

5 Key Best Practices

(Continuous Testing)

Best Practices Assessment Tool


Network Test System

Four-layered product:

Use Cases

Application

Firmware

Hardware

36 million lines of code

43 build targets

93 deployment packages


Business Pains

< 2 releases/year… LATE

Rising escalations & defects

Features deferred

75% corrective work

CapEx & Opex budget

Lack of process visibility

Intellectual property security


Technical Pains

5 centers w/ separate branches / labs

22 hour build + 24 hours test

2 GB images 9 hours to transmit

7 day deployment

<50% tests automated

Inconsistent metrics

Culture Dev/QA/Delivery silos

Credentials management issues


Business Goals

Predictable monthly releases

50% less corrective work

25% more new features / year

Stabilize spending

Secure intellectual property


Vision and Architecture

Plan

DevOps specialist team and Modular packages

Security layers

Target Metrics

95% automated build, test and release process

One hour build/test (98% reduction from 46 hours)

Nightly 8 hour regression (90% reduction from 2

weeks)

24 hour release test (85% reduction from 4

weeks)

One day deployment (85% reduction from 7 days)


Solution Approach

3. Accelerate

1. Infrastructure

2. Scale-Up


Integrate Processes

Common infrastructure

Process design

Controls


Stabilize and Secure

Process re-engineering

Tools

Fault tolerance

Intellectual Property admin


Metrics & Telemetry

Process times

Environment reliability

Intelligent dashboards

Security metrics


Remove Bottlenecks!

Modular-ize

Incremental-ize

Remove dependencies

Pipeline workflows

Discount false failures


Scale It Up !

Virtualize build and test

Elastic build and test resources

Multiple DevOps setups

Workflow optimizations


Organize for Speed

DevOps team expansion

QA folded into development

Customer-focused development

teams

Fewer management layers


Optimize

Distributed version management

with secure workgroups

Target Pre-Flight for each

development team

Development owns QA/testing

Development owns training


Business Results

Phase 1

• Secure stable process

• Predictable dates

Phase 2

• Quality improvement

• Reduced corrective work

Phase 3

• CapEx stabilized

• OpEx reduced


Technical Results

Stability: 99%

Speed:

One hour build / test

8 hour package

24 hour release test

Security admin simplified


Lessons Learned

Unexpected OpEx reductions

Platforms stabilized faster

Manage culture proactively

Continuous testing

Best practices expertise


Continued Improvements

1) Pre-Flight to reduce reverts

2) Hybrid cloud bursting

3) Smarter results analytics

4) Continuous culture

5) Automate security audits


Spirent CLEAR DevOps Solution Blueprint

Continuous

Deployment

(CD)

Continuous

Integration

(CI)

Plu

g-i

ns

Plu

g-i

ns

SUT Tools

Lab Management

Physical, Virtual, Hybrid Lab

Analytics ALM

6. Expertise and professional services

1. Test orchestration & lab management

2. Comprehensive suite of test tools

3. Physical, virtual and mixed hybrid labs

4. CI/CT/CD/CCM tools integration (EVCI)

5. CT analytics, ALM integration

Orchestration

Continuous Test (CT)

CCM


CT Best Practice Assessment

Best Practice Description

(P)

Practice Level

Score To

what extent does the organization

practice this? 0=not sure, 1=Rarely, if

ever; 2= Sometimes; 3=Most of the

time; 4=Always; 5=We are really good

at this. Unsure enter NA.

(I)

Importance

Score How

important is this practice to the

organization? 0=not relevant,

1=not important, 2=nice to have,

3=important, 4=very important,

5=critical

(G)

GAP Priority

=f[(P),(I)] Computed result is 1-15. A score

7.5 or higher indicates an

important GAP

CI builds are tested in a clone of the production environment. (Note:

"production environment" means "variations of customer configurations of a

product".)

2 4 8

Tests are selected automatically according to the specific software changes. CT is

orchestrated dynamically whereby the execution of portions of the CT test

suites may be accelerated or skipped entirely depending on how complex or

risky the software changes are.

2 4 8

Test resouces are scaled automatically according to the resource requirements of

specific tests selected and the available time for testing. 2 4 8

Release regression tests are automated. At least 85% of the tests are fully

automated and the remaining are auto-assisted if portions must be performed

manually.

2 4 8

The entire testing lifecycle including CI, nightly regression, weekend regression

and release acceptance phases are automatically orchestrated. The test suites

for each phase include a pre-defined set of tests that may be selected

automatically according predefined criterion.

2 4 8

Continuous Testing practices (CT) 2.0 4.0 8.0

Practice Level Average Importance Average Average GAP


Practice Area

(P)

Practice Level

Score To what extent does the

organization practice this? 0=not

sure, 1=Rarely, if ever; 2=

Sometimes; 3=Most of the time;

4=Always; 5=We are really good

that this. Unsure enter NA.

(I)

Importance

Score How important is this practice to

the organization? 0=not

relevant, 1=not important,

2=nice to have, 3=important,

4=very important, 5=critical

(G)

GAP Priority

=f[(P),(I)] Computed result is 1-15. A score

7.5 or higher indicates an

important GAP

Pre-Flight Practices (Pre-Flight) 2.3 3.9 7.7

Continuous Integration practices (CI) 3.1 3.5 3.3

Continous Testing practices (CT) 2.0 4.0 8.0

Continuous Delivery practices (CD) 2.7 3.0 4.0

Continous Change Management practices (CCM) 2.3 3.0 5.3

DevOps System practices (System) 2.5 3.2 4.8

DevOps Team practices (Team) 3.0 3.8 3.8

Overall Assessment 2.5 3.5 5.3

Practice Level Average Importance Average Average GAP Level

DevOps Best Practice Assessment Offer

Email: [email protected]


Summary

DevOps works for platform

waterfall and Agile software

processes

Best practices are critical

Don't “do-CT-yourself”


spirent.com/devops

http://www.spirent.com/Solutions/DevOps

http://www.spirent.com/Solutions/DevOps


© Spirent Communications, Inc. All of the company names and/or brand names and/or product names and/or logos referred to in this document, in particular the name

“Spirent” and its logo device, are either registered trademarks or trademarks pending registration in accordance with relevant national laws. All rights reserved.

Specifications subject to change without notice.

spirent.com

Thank you

Case Study -

DevOps Cost-Effectively Secured Quality and Time-to-Market for a Complex

Hardware/Software Network Product

Marc Hornbeek

Senior Solutions Architect

[email protected]

Twitter: mhexcalibur

mailto:[email protected]



Technology

Case Study - Implementing DevOps for a complex hardware/software-based network product