OPNFV CI and ChallengesHow we solved them – if we solved them at all!

Fatih DegirmenciPrincipal Developer, Ericsson

Who am I?

• Principal Developer, Ericsson• Member of the Technical Steering Committee, OPNFV• Project Team Lead, Releng, OPNFV• Committer to several Infrastructure Projects, OPNFV• Active Technical Contributor, OpenStack

What is OPNFV?

Open Platform for Network Functions Virtualization

“OPNFV is a carrier-grade, integrated, open source platform to accelerate the introduction of new NFV products

and services.”

OPNFV is a midstream project

Enhancements

Additions

Realizing ETSI NFV Architecture

To transform telco

Service Providers

Telco/Network Vendors

IT Vendors

Linux Distro Vendors

Education/ResearchInstitutions

Altogether

By doing

DEPLOYDEVS BUILD &INTEGRATE

TEST USERS

We call it Platform CI

Target POD(SUT)

Jumphost

InitiateRun for certain

Scenario

DownloadISO

Deploy FunctestFunctional Testing

YardstickPlatform Benchmarking

Pull DownFunctest Image

Pull DownYardstick Image

Labs connected 24x7using JNLP

Google CloudStorage

DockerHub

DockerHub

Using the tools

On distributed OPNFV Infrastructure

Which is, rather complex…

Lab/POD Ericsson Linux Foundation Orange Intel Huawei

Host OS Ubuntu CentOS

Installer / Cloud Execution Apex Compass Fuel Joid

SDN OpenDaylight ONOS OpenContrail

Testing Functional Testing (Functest) Platform Benchmarking (Yardstick)

Addons / Features SFC BGPVPN KVM for NFV FD.IO

But we achieved a lot!

What Release1Arno

Release2 Brahmaputra

Release3Colorado

Scenarios supported by CI1 2 20+ 37+

No of Pharos labs2 3 12 14

CPU Architecture3 1 1 2

No of nodes utilized by OPNFV Jenkins 13 90+ 130+

Time to run release verification ~30 hours 400+ hours 600+ hours

Total no of OPNFV deployments in CI4 8000+

Despite the Challenges

• Being a midstream project• Availability of stable versions from upstream

• OPNFV Development Workflow• Slow feedback

• Diverse infrastructure and lack of uniformity• Diversity is good but we need some homogeneity

• Community CI/CD Awareness and resistance to change• Contributors with different backgrounds, mindset &

culture

Being a midstream project• OPNFV can go as fast as its slowest upstream…• In order for OPNFV to do CI, upstream must do CD!

OPNFV Development WorkflowProject Team OPNFV gerrit OPNFV CI/Test OPNFV Release OpenStack gerrit OpenStack CI/Test OpenStack Release

Requirement

PatchTest Release

Downstream

Release

Test

Long development cycleDownstream will delay to next releaseSlow feedback, > 5 monthsOPNFV specific issues cannot be tested/detected in time

Document

OPNFV Workflow with 3rd Party CIProject Team OPNFV gerrit OPNFV CI/Test OPNFV Release OpenStack gerrit OpenStack CI/Test OpenStack Release

Requirement

PatchTest Release

Downstream

Release

Test

TestFast feedback

Fast development cycleFast feedback, < 1 dayOPNFV specific issues can be tested/detected in time; Correction in same release

Document

OPNFV Workflow

3rd Party CI

contribute contribute contribute

pull from masterpost feedback

Infrastructure Diversity

LaaS

LaaS

LaaS

Community CI/CD Awareness

Arno ( 2015-06) Brahmaputra (2016-02) Colorado (2016/09) Now (2016/11)0

50

100

150

200

250

300

40

168(+320%)

251(+49%)272(+8%)

13 36(+176%)75(+108%) 84(+12%)

Total Contributors Contributors to CI

In the pipeline

• Re-thinking our CI/CD and Infrastructure• CI and Infrastructure Evolution

• Further increase the community awareness in CI/CD• “Recruit” more core people

• Close collaboration with upstream projects on CI/CD• Push them to CD

Questions