Automating Software Development Life Cycle - A DevOps Approach

Software Development Life Cycle

A DevOps Approach

Akshaya Mahapatra

Akshaya Mahapatra Software Engineer @ VCE (Virtual

Computing Environment) Experienced in

◦ Java/J2EE◦ VMware products

vSphere, vCloud◦ Puppet

Developing solutions for application deployment on cloud

About me

3

Why DevOps? Why now? Stages in SDLC

◦Check-in◦Build and Test◦Infrastructure setup◦Release◦Monitoring and scaling

A typical use case Q & A

Agenda

Dev and Ops have different goals.◦Features vs SLA

Skills are different.◦Dev team lacks infrastructure knowledge◦Ops team unaware of complexity of

development. Slow feedback loop

◦Business -> Ops ->Dev

Dev Ops - Two Teams

An idea, practice or culture that brings Dev, Ops and Business together.

How?◦Repeatable processes◦End-to-end automation.◦Continuous monitoring

What is DevOps?

Virtualization and Cloud Computing◦Programmable Infrastructure◦Just-in-time Provisioning Network Compute Storage

Agile software development◦Speed is critical.

DevOps – Why now?

Application Needs◦Nodes can fail◦No “manual” configuration◦Heterogeneous Execution environments

Advancement of technology◦Puppet eco-system

DevOps – Why now?

Software Development Life Cycle

Continuous Integration

Continuous Deploymen

t

Check-in

Continuous Monitoring

Scale Out

Stages in SDLC - Packaging

Check-in

Version Control Continuous

Integration System

Build Test

Packaged Application

warearAppliancevApp

Stages in SDLC – Prepare Infra

Packaged Application

Infra Setup

LB APP

DB

vSphere

VM Templates

IP Assignment

APP

Stages in SDLC - Deployment

Configuration Management

MonitoringTools

Scale Out

MonitoringLB APP

DB

vSphere

Config

APP

Apache License Initial Release – October 2000Central Repository.

◦Better access controlFast branching and tagging.

Version Control - Subversion

Version Control - GIT

De-centralized. Inherent redundancy.

Local repository. Efficient use of space.Offline Use

SVN vs GIT

SVN GIT

svn checkout.svn addsvn commit.svn commitsvn diffsvn updateonline

git clonegit addgit commitgit pushgit diffgit pullonline/offline

Version Control – Best Practice

All “stable” code goes to trunk.Major development in branches.

◦Merge to trunk when stable.Snapshots for release are tagged.

Simple, Java basedXML based configurationDependency Management

◦Project ->TargetEase of logging and debugging. Integration with major IDEs.

Build – Apache Ant

Beyond “typical” build◦Reporting◦Collaboration.◦Project life cycle.

Automated dependency management

Sensible default configurationBetter suited for complex projects

Build - Maven

settings.xml◦ ${user.home}/.m2/settings.xml

<proxies> <proxy> <host>172.30.100.25</host> <port>8080</port> </proxy> </proxies><repositories> <repository> <id>central</id> </repository></repositories>

Maven

Dependency in pom.xml<dependencies> <dependency>

<groupId>log4j</groupId><artifactId>log4j</artifactId>

<version>${log4j.version}</version> </dependency></dependencies>

Maven

Distribution Management in pom.xml<distributionManagement><repository> <id>devops-snapshot</id> <name>DevOps Repository</name> <url>http://codecamp.com/repo/devops</url></repository> </distributionManagement>

Maven

Open source Continuous Integration server

http://jenkins-ci.org/◦Written in Java◦Performs jobs◦Numerous plugins◦Wide support◦Post build actions

Jenkins

Continuous Integration -Jenkins

Code Repository(GIT/SVN)

MavenAnt

JunitTestNG

Selenium

Check-in Workspace

Target Server

Jenkins(Hudson) Build

Update

Test3

1

2

4 Packaging

PackagedApplication

Infrastructure Setup

Clone VM

ConfigureVM

Blue PrintsTemplates

Infra Setup

3

1

2

Power On

BuildSuccessful

4

VM

VM

VM

VM

vSphere

Cloning Specification:◦Networking Info◦DNS◦Domain

Host/Cluster Information OS Customization

◦Sysprep in windows◦IP configuration in Linux

Infrastructure Setup

Supported by VMware and up-to-date Clone VM:

◦ Create Clone Spec: Set host name, network info, data store, host resources, folder etc.

◦ Task task = vm.cloneVM_Task(vcFolder, vmName, cloneSpec);

◦ if (task.waitForTask() == Task.SUCCESS) return true; else return false;

VMware API - VIJava

Supported by VMware and up-to-date Clone VM:

◦ Add-PSSnapin VMware.VimAutomation.Core◦ Connect-VIServer -Server $VC◦ New-VM -Name $cloneName -VM $sourceVM -

ResourcePool $respool -Datastore $datastore

VMware API - PowerCLI

PowershellNew-NetIPAddress –InterfaceAlias “Local Area Connection” –IPv4Address “192.168.1.100” –PrefixLength 24 -DefaultGateway 192.168.1.1

Set-DnsClientServerAddress -InterfaceAlias “Local Area Connection” -ServerAddresses 192.168.1.254, 192.168.1.253

IP Address- Windows

/etc/sysconfig/networkNETWORKING=yesHOSTNAME=codecamp.devops.comGATEWAY=192.168.1.1

/etc/sysconfig/network-scripts/ifcfg-eth0DEVICE=eth0BOOTPROTO=staticBROADCAST=192.168.1.255IPADDR=192.168.1.10

/etc/resolv.confsearch devops.com nameserver 192.168.1.2

IP Address- Redhat

Install Applications on Templates◦ Red hat with Apache◦ Windows with SQL Server◦ Cent OS with Tomcat◦ Red hat with ActiveMQ

Create Virtual Machines from templates.◦ Applications already installed.

Configure virtual machines.

VM Templates – Option 1

Templates with Base OS◦ Red hat◦ Windows◦ Cent OS

Create Virtual Machines from templates. Install required components on virtual

machines on demand using Puppet.

Better alternative..

Open source Configuration Management

http://puppetlabs.com/◦Written in Ruby◦Infrastructure as code.◦Client Server system.◦Gaining lot of traction◦Simple domain specific language (DSL)

Puppet

Puppet

Catalog

PuppetMaster

Node(Puppet Agent)

Node(Puppet Agent)

Facts

Facts

CatalogConfigRepository

Client Server Architecture

Puppet – DSL - Package

package { "sudo": ensure => "installed"

}

package { 'mysql': ensure => installed, source => ‘C:/mysql-winx64.msi',

}

Puppet – DSL - User

User { “codecamp":ensure => "present",uid => "1001",gid => "1001",comment => “Code Camp Developer",home => "/home/codecamp ",shell => "/bin/bash"

}

Puppet – DSL - File

file { "C:/apache": ensure => directory, recurse => true, source =>puppet:///modules/apache",}

Puppet – DSL - Exec

exec { "Install Apache" : command => 'install_apache.bat', require => File["C:/apache"'], path => "C:/apache/bin", }

Puppet – DSL - Service

service { "Apache2.4": ensure => running, require => Exec['Install Apache'], enable => true, }

Puppet – DSL - Class

Class apache { File { "C:/apache": }

exec { "Install Apache" : }

service { "Apache2.4": }}

Puppet – DSL - nodes.pp

node apache1 { include apache }

node tomcat1{ include tomcat}

Get information on all the nodes: Host Name, domain, IP Address facter -p

Puppet manifests can access them as global variable

Ex: “$::hostname” Customizable.

Facter – A complement

Puppet – DSL

class ntp { case $::operatingsystem {

centos, redhat: { $service_name = 'ntpd'}

debian, ubuntu: { $service_name = 'ntp' }

}…………}

Most of application deployment need Orchestration support◦Orchestration is “cumbersome” with

base puppet. Changes need to wait until next “agent” run..

Puppet – Orchestration

Open source Asynchronous Orchestration

◦Can manage puppet runs. Scalable

◦Parallel execution

Marionete Collective(mCollective)

Puppet with mCollective

PuppetMaster

Node(Puppet

and mCollective

Agent)

Catalog

mCollective Client

Node(Puppet and mCollective

Agent)

ActiveMQCommands Commands

Catalog

Find out reachable nodes◦mco ping

Find service status◦mco rpc service start service=httpd

Run puppet agent once on a node◦mco puppet runonce –W

hostname=apache1

mCollective Commands

Environments◦Development◦Test◦Production

Defined as a config parameter.Puppet Master can handle multiple environments.

Puppet Environment

Puppet Environment

PuppetMaster

Test

Development

Production

svn/dev

svn/test

/svn/prod

Managing what nodes get what classes..◦Nodes.pp grows out of control when you have thousands of nodes.

Solution: ◦External Node Classifier

Puppet – Large Deployments

Key/Value pair lookup toolProvides environment hierarchy

◦Default◦Node Name◦Environment◦Domain

Hiera (External Node Classifier)

Puppet and Hiera

PuppetMaster

Node(Puppet Agent)

Hiera

External Node Classifier

FactsCatalog(Classes)

Facts

Catalog(Classes)

Continuous Monitoring

VM

VM

VM

Hyperic

Infra Setup

VM

VMVM

PuppetMaster

Check Threshold

VMware Products◦Hyperic – OS/Infrastructure◦App Insight – Inside Application

Java Tools and APIs◦Sigar◦Jconsole

Monitoring

Use Case

A simple Web App

Web Server(Apache)

App Server

(Tomcat)

App Server

(Tomcat)

App Server

(Tomcat)

<application> <name>Puppet Demo</name>

<nodes><name>Tomcat Server</name><template>CentOS56 64bit</template><key>war_file</key><instances>3</instances>

</nodes><nodes>

<name>Apache Load Balancer</name><template>CentOS56 64bit</template>

</nodes></application>

App Blue Print

Clone templates to create virtual machines.◦One VM for apache Load Balancer◦Three VMs for Tomcat and App .war file.

Create Puppet Manifest or Configure ENC.◦Associate Puppet/Classes with node

names. Run Puppet Agent on Virtual Machines

◦In correct sequence.

Steps

Install Apache◦mco puppet -W hostname=apache1

runonce Install Tomcat

◦ mco puppet -W role=apserver runonce Configure Apache

◦mco puppet -W hostname=apache1 runonce

Steps

Puppet and Puppet Forge◦ puppetlabs.com◦ forge.puppetlabs.com/

VMware VIJava API◦ vijava.sourceforge.net/

Provisioning◦ github.com/puppetlabs/razor

More Info:

?akshaya.mahapatra@gmail.com