Www.electric-cloud.com Software Production Management Agile/Continuous Integration Backend infrastructure to realize Agile software development methodologies

www.electric-cloud.com

Software Production Management

Agile/Continuous IntegrationBackend infrastructure to realize Agile software development methodologies

November 18, 2008

Jeff PriceAccount Manager

Dave FinneganSystems Engineer

April 21, 2023 Slide 2

What am I going to tell you today?What am I going to tell you today?

Four practical steps to support Agile / Continuous Integration

Impacts on the Build/Integration team

Best practices you can adopt right now

Commercially-available tools to overcome these impacts


Practical steps for Agile / CIPractical steps for Agile / CI

One : Enable developers to build small pieces of functionality

Two: Automate the full integration build

Three: Perform an integration build for each SCM commit

Four: Minimize broken builds and maximize integrated testing of new features


Impacts on Build/IntegrationImpacts on Build/Integration


Problem : Developers work on isolated components; Integration becomes a trouble spot

Two: Automate the full integration build







Two: Automate the full integration buildProblem : DIY build scripts are fragile and expensive









Problem : Number of builds and total build time explodes










Problem : keeping the main line buildable and clean


Overcoming ImpactsOvercoming Impacts


Problem : DIY build scripts are fragile and expensive




Give the developers access to the build!Give the developers access to the build!

Put full builds in the hands of developers

Set up an internal web page that allows a developer to fire off a build

The developer should be able to select:Which branch to build…… or the location of his personal source treeWhether to run unit tests or notSelect platforms

The build runs and the engineer gets an email with the result


The Problems for the Build team…The Problems for the Build team…






Script your build the way you write code!Script your build the way you write code!

Design generic, modular build elements which can be reused

Make use of parameterization

Develop or apply Continuous Integration systems (start with cron)

Standardize on a single set of build toolsPick a scripting language and stick with it

Enforce SCM for build scripts

Code review for build scripting

Know when to stop fiddling








Make the build fast, automatic, and visibleMake the build fast, automatic, and visible

Achieve fast buildsThat means better build structureFast build hardware, special software for parallel buildsgmake -j may get you 2-3x improvements

Schedule builds through automationcron for CI buildsWeb front-ends and request systems for developer builds

Investigate better build reportingCollect and publish build result information

Build statusCollect metrics: errors, warnings, tests, etcInclude changelogs

Build tracking repository








Guard the mainline and create policiesGuard the mainline and create policies

Private BranchesGive developers private branches, or multi-stage CI streams

Integration BuildsRequire developers to run integration builds before merging

SCM TriggersUse SCM hooks to fire an integration build for every checkin

Automate Checkin RevertsIntegrate SCM and CI to revert checkins which break the build

Communicate StatusUse notification, dashboards, or lava lamps to communicate mainline status


Going Beyond DIYGoing Beyond DIY


The Build ManagerThe Build Manager


Improving on in-houseImproving on in-house

I need fast buildsThat means better build structureFast build machines, special software for parallel buildsgmake -j doesn’t give the speedup up I need

I’m doing 200 builds a day, I need better toolscron just won’t cut itI need resource management toolsI need developers to build in the production environment

I need reporting toolsEveryone wants live build feedbackPerl + Excel isn’t enough

I need this because builds are now the heartbeat of software engineering


Electric Cloud SPM SolutionsElectric Cloud SPM Solutions

ElectricCommanderAutomate Production Processes

Build Package Test Deploy

ElectricInsightVisualize Builds

ElectricAccelerator10-20x Faster Builds

Build ToolsMake, Ant,Maven, etc.

Package ToolsInstallers, etc.

Test ToolsUnit Test Tools,Code Coverage

Deploy ToolsMisc.

SCM ToolsPerforce, Subversion, ClearCase,AccuRev, etc




I’m doing 200 builds a day, I need better toolscron just won’t cut itI need real management toolsI need developers to build in the production environment




ElectricAcceleratorElectricAccelerator

Problem it solves: Build cycle is too longVery long builds

Complex matrix of builds and targets

Benefits:10-20x faster builds

How it works:Run builds in parallel using a cluster of standard hardware

Automatically discover and track dependencies

Plug-compatible replacement for GNU make, Microsoft nmake, Visual Studio, Ant

No process changes required

Works with ElectricCommander or stand-alone

Based on proven, patented dependency management technology


How Do You Get There?How Do You Get There?

Requirements for real build acceleration:

Fast and accurate builds via parallelism• Requires understanding of dependencies

Robust and scalable• Support high build volume and avoid resource failure

Work within existing environment• Must not require costly re-write

Visibility and reporting• Provide status reports and tools for diagnostics


Managing Dependencies for Parallel BuildsManaging Dependencies for Parallel Builds

Builds have inherent parallelism

Should be able run pieces concurrently

Large SMP Machines (gmake –j)

Distributed builds (distcc)

Other attempts yield small results due to dependencies:

Incomplete or unknown

Can’t be expressed between Makefiles

Result: broken builds

01010101010101010101

01010101010101010101

Source Code

ObjectFiles

Executables

Release

01010101010101010101

01010101010101010101

01010101010101010101

01010101010101010101

Libraries

01010101010101010101

01010101010101010101

01010101010101010101

01010101010101010101

01010101010101010101

01010101010101010101


Solution for Fast, Accurate BuildsSolution for Fast, Accurate Builds

Deduce dependencies on-the-fly:Watch all file accesses: these indicate dependenciesAutomatically detect out-of-order steps

Desired Actual

10101010101010101010101010101010101

Compilesource

class.cpp class.obj10101010101010101010101010101010101

app.exe

read write writeread

Linkapplication

Compileand linkstepsrun inparallel

old!

10101010101010101010101010101010101

class.cpp class.obj

read write

10101010101010101010101010101010101

class.obj10101010101010101010101010101010101

app.exe

writeread


Solution for Fast, Accurate BuildsSolution for Fast, Accurate Builds

Deduce dependencies on-the-fly:Watch all file accesses: these indicate dependenciesAutomatically detect and correct out-of-order stepsSave discovered dependencies for future buildsResult: high concurrency is safe

Desired Actual

10101010101010101010101010101010101

Compilesource


app.exe


Linkapplication

old!

10101010101010101010101010101010101


app.exe


Rerun link

10101010101010101010101010101010101

class.obj10101010101010101010101010101010101

app.exe

writeread

Discard


ElectricAccelerator ArchitectureElectricAccelerator Architecture

Build Cluster(10-1000 CPUs)

ClusterManager

Developer Machines

Production Machines

Network

Agent Agent …

Electric File System

Node

KernelUser-level

Build with ElectricMake

instead of GNU Make,

Microsoft NMAKE,

Visual Studio


File Caching for Network OptimizationFile Caching for Network Optimization

File caching on nodes cuts 90-95% of network traffic

Just-in-time compression cuts traffic 2.5-3x

P2P file transfers offload 20-25% of outbound traffic

Electric Make

Make Machine

Network bandwidth concentrates at make machine

Peer-to-peer file transfer

Node


AgentFile

Cache

Network

Node


AgentFile

Cache

Node


AgentFile

Cache

Node


AgentFile

Cache

April 21, 2023 Slide 28Slide 28Slide 28

Linear ScalabilityLinear Scalability

Local 20 CPUs SpeedupSamba 16m 58s 16.4xMySQL 23m 124s 11.3xGtk 15m 95s 9.4x

0

5

10

15

20

0 5 10 15 20

#CPUs in cluster

Sp

eed

up

Samba

MySQL

Gtk

0

5

10

15

20

0 5 10 15 20

#CPUs in cluster

Sp

eed

up

Samba

MySQL

Gtk

16.4x

11.3x 9.4x


Works Seamlessly in EnvironmentWorks Seamlessly in Environment

Works seamlessly with existing infrastructureGNU make, Nmake, Visual Studio

No learning a new interface/tool

Use existing Makefiles and scripts

Identical log files

Works with grid systemUse dedicated cluster of servers or central data center

Fully integrated with the Platform LSF grid computing solution

Provide build resources just when needed

Minimal changes required for end users








ElectricCommanderElectricCommanderProblem it solves: Manual, inflexible build and test infrastructure

Web-based platform for managing distributed processesFaster, more efficient build and release processes

More efficient resource utilizationBetter visibility and predictabilityCan standardize throughout an organization

Schedule/Trigger

Linux serverWindows

serverLinux server

Test harness

Resources

Pool

Linux server

Procedure

Step

Step

Step

Step

StepReport

Unit Test

Compile

Snapshot

Setup

Job

Step

Step

Step

Step

Step

Job

Step

Step

Step

Step

Step

Job

Step

Step

Step

Step

Step

Job

Step

Step

Step

Step

Step

April 21, 2023 Slide 32Slide 32

Managing Global TeamsManaging Global Teams

Specific Access and Permissions Based on RoleAnywhere in the World

SW DEVELOPERSMunich

ENGINEERING MGRBoston

BUILD TEAMSan Francisco

OUTSOURCEDQA

Bangalore

SW DEVELOPERSSan Francisco


Managing ResourcesManaging Resources

Test ServersBuild Servers Production Servers Virtual Servers

SW DEVELOPERS

ENGINEERING MGR

BUILD TEAM

OUTSOURCEDQA

SW DEVELOPERS

VMware Lab Manager

VMware Lab Manager

VMware Lab Manager

Dynamic Provisioning


Unit Tests

Deployment ToolsSCM Tools

Build Tools


SW DEVELOPERS ENGINEERING MGR BUILD TEAM OUTSOURCEDQA

SW DEVELOPERS Build ToolsSCM ToolsUnit TestsDeployment Tools

Integrate Tools and ProcessesIntegrate Tools and Processes


Tie it all TogetherTie it all Together

SCM Tools

Unit Tests

Deployment Tools

Build Tools


SW DEVELOPERS ENGINEERING MGR BUILD TEAM OUTSOURCEDQA

SW DEVELOPERS


ElectricCommander BasicsElectricCommander Basics

Schedule/Trigger

When to run a procedure

Resources

Pool

Process to run automatically

Records results of running a procedureLinux serverLinux server Linux server

Test harness

Windows server

Job

Step

Step

Step

Step

Step

Job

Step

Step

Step

Step

Step

Job

Step

Step

Step

Step

Step

Job

Step

Step

Step

Step

Step

Procedure

Step

Step

Step

Step

Step

Setup

Snapshot

Compile

Unit Test

Report

Container for processes and

results

Project

Command or script


Continuous IntegrationContinuous Integration

Integration Build

Build Success

QADeveloper runs local build and smoke tests

Developer checks tested code into SCM system

Integration build at frequent intervals or upon check-in

SCMSCMSystemSystem

Check-In

electriccommander

Production Servers


Frequent Problem: Continuously Broken Frequent Problem: Continuously Broken BuildsBuilds

Build Failure

Developer builds/tests on local system, checks in code

Integration build started, breaks (“it worked on my machine”)

Team impacted while check-in is backed out or build fixed

SCMSCMSystemSystem

Check-In

XWait

XWait

Back Out

Integration Build

electriccommander

Production Servers


Solution: Pre-Flight Builds and TestsSolution: Pre-Flight Builds and Tests

Developers

SCMSCMSystemSystem

electriccommander

Build or Test Failure

Success

Test Servers Production Servers

Developers build and test across all targets/platforms

Ensures successful integration build

Developers can check in changes with confidence

Broken builds less likely to affect the entire team

Success

Preflight Workflow

ElectricCommander

IDE

1. Developer invokes preflight through Visual Studio, Eclipse or command line

Developer changes

Source

snapshot

if {x<y}

…

func {x};

func {y}; 1. Developer changes and source snapshot uploaded to ElectricCommander

2. ElectricCommander overlays the changes on top of the snapshot

3. ElectricCommander runs a build as if the developer had checked-in the code

if {x<y}

…

func {x};

func {y};

+

SCM

Preflight Result

1. Preflight result displayed in IDE

2. Autocommit checks in successful changes to SCM (optional)

if {x<y}

…

IDE








Pinpoint Data for ReportingPinpoint Data for Reporting

Extract data from log files, create reports

Problem:• Large, complex log files• Only a small amount of data

matters • Log files get deleted

ElectricCommander solution:• Extract key data from log:

• Tests executed• Errors• Diagnostic messages

• Save in database

Properties

cl.*warning

xxxx*error*

xyzzy

Patterns

tests: 217 errors: 2

Job Step PostprocessorLog

File


Visibility: Management ReportsVisibility: Management Reports








Builds are too expensive to ignore!Builds are too expensive to ignore!

Multi-Platform Gaming Company

Problem:

Challenging, complicated build-test matrix (Xbox, PS3, PC)

Slow builds kept QA waiting and Developers avoiding p.m. checkins

Initial Continuous Integration approach led to 300 broken builds per month

Solution: ElectricAccelerator + ElectricCommander

Build acceleration led to more frequent builds and testsBuild time from 2.5 hours to 12 minutes

Developer “preflight” builds reduced broken builds by 90%Preflight build and test run on 3 platforms prior to check-in10% due to developers not doing preflight


Customer ExamplesCustomer Examples

Networking Semiconductor Cellular ISV Other


What we have covered todayWhat we have covered today

Four practical steps which are required to support Agile

Impacts of these steps on the Build/Integration team

Best practices you can adopt right now to handle these impacts

Commercially-available tools which overcome these impacts


Thanks for attendingThanks for attending

For more information:Visit our website: www.electric-cloud.com

E-mail: [email protected]

Phone: 408.419.4300


Questions and Answers

Backend infrastructure to realize Agile software development methodologies


Supplemental SlidesBuild Your Own


Build Your Own?Build Your Own?

Features

Inve

stm

en

t

Trigger builds after check-ins

Automated build and test for one build tool, one

machine

Scheduled jobs

Seems pretty straightforward…



Features

Inve

stm

en

t


Database of build results


machine

Notifications via e-mail/RSS

Scheduled jobs

Web-based access to results

Just a few finishing touches



Features

Inve

stm

en

t


machine

Scheduled jobs


Web-based access to

results



Single job can use multiple resources

Run job steps in parallel

Multiple jobs run on one resource simultaneously

Run multiple builds

simultaneously

Resource pooling, load

balancing

Multiple servers, remote invocation

Resource selection criteria

Multiple servers, concurrency


Features

Build Your Own?Build Your Own?In

ves

tme

nt


machine

Scheduled jobs


Web-based access to

results






Run multiple builds

simultaneously


balancing



Cancel jobs

View partial results of builds in progress

Monitor system status

On-demand job invocation via

Web

Better monitoring and control


Features


ves

tme

nt


machine

Scheduled jobs


Web-based access to

results






Run multiple builds

simultaneously


balancing



Cancel jobs




Web

Tools for extracting

data from logs

Trend reports

Cross-product reports

Resource utilization

reports

Annotate builds after completion

Customizable reports

Better reporting


Features


ves

tme

nt


machine

Scheduled jobs


Web-based access to

results






Run multiple builds

simultaneously


balancing



Cancel jobs




Web


data from logs

Trend reports



reports



Logging and error reporting

Retry steps after errors/ failures

Timeouts for runaway job

steps

Detect resource failures

Better error handling


Features


ves

tme

nt


machine

Scheduled jobs


Web-based access to

results






Run multiple builds

simultaneously


balancing



Cancel jobs




Web


data from logs

Trend reports



reports






steps


Developer access to production

builds

LDAP authentication

Role-based access control

Shared access by multiple teams

User impersonation,

password management

Priorities

Handle multiple independent groups


Features


ves

tme

nt


machine

Scheduled jobs


Web-based access to

results






Run multiple builds

simultaneously


balancing



Cancel jobs




Web


data from logs

Trend reports



reports






steps



builds

LDAP authentication



User impersonation,

password management

Priorities

Conditional steps

Modular, composable

process stepsExtensibility

Parameters Control environment

for jobs

More programming features


Features

Build Your Own? It Never EndsBuild Your Own? It Never EndsIn

ves

tme

nt


machine

Scheduled jobs


Web-based access to

results






Run multiple builds

simultaneously


balancing



Cancel jobs




Web


data from logs

Trend reports



reports






steps



builds

LDAP authentication



User impersonation,

password management

Priorities

Conditional steps

Modular, composable

process stepsExtensibility

Parameters Control environment

for jobs

Portable across hardware/OS

platforms

SCM integration, bill of materials

Support multiple languages and

build tools

Web interface for editing processes

Command-line interface


Questions and Answers

Documents

Www.electric-cloud.com Software Production Management Agile/Continuous Integration Backend infrastructure to realize Agile software development methodologies