Test-Driven-Development for Networking: Making CI Work for You by Colin McNamara

www.Nexusis.com 877.286.39871

Continuous Integration and Test Driven Dev for Network Engineers

Colin McNamara CCIE #18233

@colinmcnamara



Agenda

What drives my perspective

Why TDD / CI for Network Engineers

CI / TDD overview

Network Change Management Use Case


Who am I?

Chief Cloud Architect::Nexus

OpenStack ATC/Ambassador

OpenStack Core Reviewer

Manage DevOps Team

Manage Agile BPO Team

Beard Growth Specialist

CCIE 18233, RHCE, ETC, ETC

@colinmcnamara

www.colinmcnamara.com

http://www.colinmcnamara.com

What do

we do?

50% upstream

For Core Dev’s

Software Development, Systems Integration, Network Design

DenicaCloud

Rosetta

Aviator

More

Toolchain for Deploying CI systems and the associated tooling into and along with various cloud platforms

Project Denica

Core Contributions

and solutions in

OpenStack

OpenDaylight

Open vSwitch

Puppet

Agile / Lean Software

Development Practices

• SDN Development for Multi Vendor Integrations

• Agile Process Consulting

• Cloud Platform Development

• Systems Integration

• Upstream contributions

• DevOps Consulting

Extensive experience with the folliwng:


That all boils down to.. We make unicorn farts


Why TDD / CI for Networking?


Our industry is changing


Four closely related items


Agile infrastructure evolution

Classic Engineering

12 + month dev times

QA pushed till last

Large amounts of bugs

Maps easily to manual silo’d process’s

ITIL Aligned

AGILE

2 Week Dev times

Integrated QA

Small bug counts

Does not map into silo’dprocess’s

Cloud Aligned


Speed Racer (Agile Infrastructure Developer)

• Integrated tools

• Integrated test QA

• Software as a manufacturing line

• Code rolls from Step to Step

• Need infrastructure as code…


DevOps is the “New” Network Ops


Core SDN truth

ONE change

In one FILE

Can CREATE a system

Or DESTROY a system


What if you could actually KNOW it would work

Test EVERY network change made

Map each application to the network component underneath

Automatically run tests with reports of results



Agile Development Continuous Integration for Network EngineersTechnical items you need to know


DevOps Workflows


Complex View


Simple CI System Walk ThroughSVC Only

SVC + Network

“Virtualized”

Production

System


Obligatory Tools Discussion


Git / Gerrit



Integration Opportunity for CI Pipelines

VIRL(CMS) / JunoSphere

Functional Test

Network OS level testing

MATE / WAE

Stubs / Modeling

Integration Test through API’s

Tail-F, Schprokits, Puppet/Chef, Ansible

Config Distribution

Unified Data Models


Use CaseChange and Release Management


Change and Release Management


Change and Release Management

Jenkins Test Harnesses

Templates and EPG’s

Governance Tooling

GitGerrit

GitGerrit

Gerrit

Gerrit

Jenkins Test HarnessesAntVagrant

PuppetJenkins Test Harnesses

VagrantPuppet

GitGerrit

PuppetOther

GitGerrit

Test HarnessesPuppetOther

Jenkins Test Harnesses

PuppetOtherJenkins

Test HarnessesVagrantPuppet

GerritGovernance Tooling


Maturity Levels – how to move forward

Level0 Level1 Level2 Level3 Level4• Self Service Builds

• Nightly Builds

• Build Artifacts are stored

• Self Services deploy to test

• Auto deploy of builds to 1st env

• Mostly standard deploys

• Significant Test execution at build time

• Visibility: Team

• Latest reports always accessible

• Build on commit

• Dependency Repo

• Secured Configs

• Self-Service deploy to test and prod

• Standard Process across all environments

• Some Static analysis

• Automated Functtests run nightly

• Visibility: Cross siloe’d team (DevOps)

• Historical reports available

• Triggered Builds

• Build Cluster

• Test Gated Automated Promotions

• Database Deployments

• Multi-tier SOA

• High Code Coverage• Security Scans• Risk based manual• testing

• Report trending

• Build from Snapshots

• Gated Commit

• Continuous Deployment to production

• 100% Coverage

• Gross Silo Analysis

Develo

pm

en

tO

pe

rati

on

sQ

AM

an

ag

em

en

t

• Build Value Stream Maps

• Build Scripts• Source Control

• Deployment Scripts• Source Control

• Build Test Automation

• Source Control

• Visibility: SiloedIndividuals

• Tool Generating Reporting


How to get there – Level 0

•Create standards for service chains (cfg,controler)

•Configuration Templates , puppet modules

•Use source control - Git

•Build VSM of specific IT process

Development

•Script Deployments – Expect, Simple Puppet

•Use source control – Git

Operations

•Build new services in lab / cloud / virt environment – VIRL / GNS3 / VMware / OpenStack

•Simple test harness built (ping)

QA

• Individual Kanban Boards / Ticket tracking

Management





• Source Control



Level1 Level2• Self Service

Builds• Nightly Builds• Build Artifacts

are stored





• Visibility: Team• Latest reports

always accessible

• Build on commit• Dependency

Repo• Secured Configs




• Automated Funct tests run nightly



Develo

pm

en

tO

pe

rati

on

sQ

AM

an

ag

em

en

t

Level0



• Ci System Implemented

• Scripts build virtual env on demand

• Results from built system get stored

Development

• Deployment via Config Mgmt tooling

• Governance / CI system deployed

• Ops review of config diffs

Operations

• CI System Implemented

• Common functional tests created

• Common port / path tests created

QA

• Visualize flow – Kanban Board for team

• Initial ScrumBan Process’s Mapped

Management





• Source Control



Level1

Level2• Self Service Builds• Nightly Builds• Build Artifacts are

stored





• Visibility: Team• Latest reports always

accessible

• Build on commit• Dependency

Repo• Secured Configs




• Automated Funct tests run nightly



Develo

pm

en

tO

pe

rati

on

sQ

AM

an

ag

em

en

t

Level0



• Ci Systems integrated with Governance + Src Control

• Dendancy Mapping (Mvn, Ant, EPG’s)

• Keys, Passwords as variables, stored securely

Development

• CI Chain Integrated w/ release

• Governance tool used for Change Control Board

• Puppet + Vagrant + Scripts abstracted from Dev, test prod

Operations

• CI System Implemented

• CI system running Spec tests / Flake8 etc

• System building environment and running tests emulating applications

QA

• Roll up reports of multiple team boards

• Burndown charts stored

• Card movements logged

• DevOps changes mapped to existing process’s

Management





• Source Control



Level1 Level2

• Self Service Builds• Nightly Builds• Build Artifacts are

stored





• Visibility: Team• Latest reports always

accessible

• Build on commit• Dependency Repo• Secured Configs



• Some Static analysis• Automated Funct

tests run nightly



Develo

pm

en

tO

pe

rati

on

sQ

AM

an

ag

em

en

t

Level0


Take Aways


Key Points

TDD increases quality / decreases risk / increases velocity

Network Operations and Design benefit from Agile Methodologies

Defining and executing on a maturity model necessary to succeed.



@colinmcnamarawww.colinmcnamara.com

Technology

Test-Driven-Development for Networking: Making CI Work for You by Colin McNamara