From ci to cd

From Continous Integration to Continuous Delivery Henri Gomez INSA Lyon 11 June 2012

Aim of this document

•  Today Software Challenges •  Expose what are CI and CDs •  Continuous Delivery •  Continuous Deployement •  Contents Repository •  Native Packaging

Business Interaction Networks

Axway is the Business Interaction Networks company. We are the only provider in the market today to manage, run, secure, and monitor all your business interactions — emails, files, messages, services, events, and processes.

About @hgomez

•  CI Architect @ Axway

•  Former Senior Ops Director •  Former Developer, Team Leader and Architect

•  JPackage Project Founder •  OpenJDK 7&8 packages for OSX •  ASF Member (Tomcat & XML-RPC)

Software Challenges

Software Challenges Today

•  Software projects are larger and complex •  More and more component interoperability •  Many teams and often remote teams •  Tight schedules •  Frequent needs and requirements changes

Software Challenges Early ages

•  Build / Release on a “per request” basis after –  Project reaches a particular step (milestone/release candidate)

•  Mandatory pre-requesites –  Project in proper state -> yes it build –  Well documented per standards -> doc is not an option –  Properly reviewed -> someone tested product

•  Late discovery –  Some pre-requesites where just not there ;(

Software Challenges Then came Agile

Software Challenges Mandatory tooling for Agile

•  Standardize Build, Test and Deployment

•  Provide Continuous Builds, Test and Deployment

•  Ensures solid, reproducible and analyzable steps

•  Visible to all actors (Dev, QA, Product Owner)

Software Challenges Ideal life-cycle for Agile

Continuous Integration

Continuous Integration Definition

•  Continuous Integration is the practice of integrating early and often, so as to avoid the pitfalls of "integration hell".!

•  The ultimate goal is to reduce timely rework and thus reduce cost and time. When done well, continuous integration has been shown to achieve these goals."

Continuous Integration Workflow

•  Team members work on code –  Code and Unit tests –  Push on SCM after local tests passed

•  CI Server periodically polls SCM and if changes –  Check out all code –  Build all code Run all unit tests –  Run acceptance tests –  Deploy snapshots to artifacts repository

In case of error in any step, CI server inform all parties

Continuous Integration Building

•  Fetch code –  From SCM

•  Build code –  Test code in its dependency land

•  Run tests –  Unit tests –  Acceptance tests

•  Deploy Dev artifacts –  Developers artifacts (ie: Maven jar/war) –  Precisely identified (name and version)

Continuous Integration QA = Quality Assurance

•  Functional Testing –  Match expected behavior –  Test regressions fixes

•  Performance Testing –  Smoke Tests –  Stress Tests –  Response Speed –  Throughput –  Application behavior metering –  Systems hosting application behavior

Continuous Integration Build in Action #1

Continuous Integration – QA Focus

Continuous Integration – QA Unit Testing

•  Unit Tests test the code •  Ensures the code is doing what we intend it to do •  Many extremely quick and small tests

Unit Testing ensure we make the software right

Continuous Integration – QA Functional Testing

•  Check code behavior •  Automated or manual •  Tests performed in real world, ie a Web UI

Functional Testing ensure we make the right software

Continuous Integration – QA Performance Testing

•  Check system and application behavior •  Stress tests •  System and application metrics collected and analyzed

Performance Testing help figure hotspots and runtime problems quickly

Continuous Integration – QA Capacity Planning

•  Performance testing ++ •  Stress and tortures tests •  Define Limits (ie: x transaction/s for a fixed system)

Capacity Planning bring abacus to support and sales teams to answer customers needs

Continuous Integration – Tools

Continuous Integration - Tools A rich ecosystem

•  Sources Repositories"•  Build Engines"•  Test Engines"•  Integration Engines"•  Binaries Repositories!•  Deployment Tooling"

Continuous Integration – Tools Some names and logos

Continuous Integration – What’s next ?

Continuous Integration – Next What’s next ?

•  Continuous Deployment is not Continuous Integration

Continuous Integration – Next Continuous Deployment in action

Continuous Integration – Next Assembly is key !

•  Fetch Dev Artifacts –  Mandate component approach

•  Assemble Artifacts –  Components Assembly guidelines –  Special care of Components configurations

•  Deploy Usable Artifacts –  Ready to use by upstream teams like QA-Ops –  Should be configurable from the outside

Continuous Integration – Next Delivery and Deployment

•  Delivery "–  Providing product to end user"–  User could then install or update it on his own systems!

•  Deployement "–  Providing operational solution to end user"–  User could directly use it on his own or leased systems"

Continuous Delivery

Continuous Delivery Definition

•  Continuous Delivery could be seen as the process who bring products to internal and external customers

•  Internal customers –  Developers –  QA –  Sales

•  Various medias –  CD/DVD –  AMI/VMDK/VDI –  Native packages

Continous Delivery Requirements

•  Contents Repository –  Precisely identified components –  Components fetchable via APIs

•  Scriptable operations –  Components Fetch –  Components Assembly –  Media generation (packages or burn)

•  It will turn into a Media Factory –  Predicable process –  Automated process

Continuous Deployment

Continuous Deployment Definition

•  Continuous Deployment is the process who bring products running on systems to internal and external customers

•  3 main requesters –  Dev Teams –  QA Teams –  Ops Teams (internal or customers)

Continuous Deployment For Dev Teams

•  Code Testing from Trunk

•  Deployment decision by Dev Teams

•  Operated on Dev Teams Servers

Continuous Deployment For QA Teams

•  Product Testing on Internal Releases

•  Deployment decisions on demand by QA Teams

•  Operated on QA Teams Servers

Continuous Deployment For OPS Teams

•  Product install or update from Releases

•  Deployment decision by OPS/Customers –  To inform users about planned operation –  To follow their customers SLA

•  Operated on Production Servers –  Under control of customers Ops –  With customers agreements (Cloud/Saas)

Continuous Deployment Requirements

•  Products Repository –  Precisely identified products –  Products fetchable via APIs

•  Automated operations –  Products Fetch –  Products Deploy (install/update)

Continuous Deployment Factory

•  Deployment Factory –  Production of Native Packages –  Production of OS Images

(VMWare, Amazon Web Service)

•  Images and Packages –  Images for boot strap

operations (Virtualized environments)

–  Packages to install and update systems (Virtualized and Physical)

Contents Repository

Contents Repository Definition

•  Shared referential –  For Development Teams –  For QA Team –  For Assembly Team

•  APIs powered –  Upload / Download operations by APIs

•  Security in mind –  By Repository sections –  By Users –  Peering and sites replication

Contents Repository Benefits

•  Common referential –  Artifacts provided by developers in development phase –  Artifacts tested by QA in validation phase –  Artifacts assembled in continuous delivery and deployment

chains

•  Automation ready –  Unique identifier for artifacts (id, group, version, kind) –  Programatic access to artifacts easily via APIs

•  Web based –  Security via redundancy (on/off sites) –  Peering via replication (on remote sites)

CI Today and beyond

Today CI

•  From Source to Artifacts –  Developper commit –  CI build project(s) related to commit –  CI publish artifacts project(s) to Content Repository

Today Delivery

•  From Artifacts to Delivery –  Assembler collect artifacts, build and deploy from its workstation –  Assembler push usable delivery to WebLiv –  Customers and Axway team get delivery from WebLiv

Today Deployment

•  From Artifacts to Deployment –  Assembler collect artifacts and launch appliances build –  Appliances deployed on WebLiv and Amazon S3 (Cloud) –  Customers and Axway team get delivery from WebLiv or S3

Tomorrow Continuous Delivery

•  On demand delivery via QA/Project Manager

Tomorrow Continuous Packaging

•  Native packages continuous production –  For developpers usage –  For QA usage –  For Cloud Factory usage –  For Ops usage

Tomorrow On Demand Deployment

•  On Demand Deployment (ie: QA on Internal Releases)

Tomorrow Applicances Builder Feeding

•  Appliance Builder feeded with native packages –  Turn Appliances builder in Appliances Factory –  From source to VMWare appliances –  From source to EC2 AMIs

Native Packaging

Native Packaging Situation

•  Widely used in Unix world •  Componentisation approach •  Runtime dependencies aware •  Powerfull life-cycle support •  Web based packages repositories for

modern delivery

Native Packaging Bonus

•  Quicker bootstrap for customers on their own systems

•  Customers Ops Teams may use them as regular packages

•  Help create VM images or appliances with up to date products

•  Help update products in existing VM images/appliances

•  Faster install/update process for customers •  Perfectly suited for Cloud Operations

From ci to cd

Technology

QCon'17 talk: CI/CD at scale - lessons from LinkedIn and Mockito

How Docker simplifies CI/CD

Java PaaS – Enabling CI, CD, and DevOps - Huihoodocs.huihoo.com/javaone/...PaaS-Truly-Enabling-CI-CD-and-DevOps.pdfJava PaaS – Enabling CI, CD, and DevOps. AuthX Overview Who We

Automate CI/CD with Rancher

The missing CI/CD Kubernetes component: Helm ......The missing CI/CD Kubernetes component: Helm packagemanager The past two months I have been actively migrating all my services from

CI-CD and DevOps with Ruby

CI / CD with fabric8

Exploring Docker in CI/CD

In US since 2015 CI/CD at scale Lessons from LinkedIn and ... · @LinkedInEng | @mockitoguy CI/CD at scale Lessons from LinkedIn and Mockito QCon, San Francisco, 11-2017 Szczepan

6 AWS Services / Serverless CI CD

Achieving CI/CD with Kubernetes

TMF2014 CI-CD Workshop Michael Palotas

CI/CD Pipeline with Docker

Web Performance testing with CI/CD

CI/CD with Docker on AWS

GitHub & GitLab CI/CD

CI/CD Best Practices

CI/CD for mobile at HERE

From ci to cd - LavaJug 2012

GitLab CI/CD - An Overview · DevOps Porto - Filipa Lacerda @filipalacerda gitlab.com/filipa GitLab CI/CD - An Overview