Automating Software Production Processes

Kevin Jameson, Founder

History of Multics, Unix, Emacs, Codefast

• 1965-1969 Multics– MIT Project MAC, GE, Bell Labs, Naming culture of the time

• 1971 Unix– Was another joke name (“GNU is Not Unix”, “EINE Is Not Emacs”)

– Unix = Eunuchs = “Multics without balls”

• 1976 TECO Emacs (Tape/Text Editor COrrector)– [1 J^P$L$$

J <.-Z; .,(S,$ -D .)FX1 @F^B $K :L I $ G1 L>$$

– Richard Stallman, “Editor Macros” package -> Emacs

History of Multics, Unix, Emacs, Codefast

• 1965-1969 Multics– MIT Project MAC, GE, Bell Labs, Naming culture of the time

• 1971 Unix– Was another joke name (“GNU is Not Unix”, “EINE Is Not Emacs”)

– Unix = Eunuchs = “Multics without balls”

• 1976 TECO Emacs (Reconceptualized what an editor was)– Richard Stallman, “Editor Macros” package -> Emacs

• 1978 Multics Emacs – Bernie Greenberg, Multics Emacs in MacLisp

History of Multics, Unix, Emacs, Codefast

• Multics Emacs -> Codefast– 1978 Bernie Greenberg, Multics Emacs in MacLisp

– 1979 U of Calgary buys a Honeywell Multics System

– 1982 Jameson, Codefast v0001 (Multics Emacs Lisp), U of Calgary

• Multics Emacs -> Unix Emacs -> GNU Emacs– 1978 Bernie Greenberg, Multics Emacs in MacLisp

– 1979 U of Calgary buys a Honeywell Multics System

– 1981 Gosling, Gosling Emacs (Unix C, MockLisp), U of Calgary

– 1985 Stallman, GNU Emacs (Unix C, GNU Lisp), MIT

– 2006 GNU Linux, Emacs, Java, Modern IDE Editors (Eclipse, etc)

Initial Codefast Concept

• Main Concept – A Program to Write a Program – Possibility of perfect quality from generated code – no bugs!– Possibility of perfect speed – computer speeds, not human speeds

• Main Blocking Point – Software Build Processes– Could generate code, but could not build it automatically– Software builds, especially multiplatform builds, were big blocks

• 1994 O’Reilly Book – Multiplatform Code Management– Showed how to automatically generate build scripts, makefiles

• 2004 Software Lifecycle Automation Book– Applied principles to software lifecycle processes, not just builds

The Innovation—Production Cycle

Death by Variations

Codefast Focus

Typical Production Pains

• Constant Changes of All Kinds

• Break Dependencies

• Increase Iterations

• Use Up Human Labor

• Slow Down Cycle Times

What Codefast Does

• Puts the Production Cycle in a “Black Box”

• Automates the Internals

• Gives It a User Interface

• Removes Human Labor

• Increases Quality of Cycle Processes

Codefast User Interface – 2 Modes

1. Give Orders

1. Get Files

2. Modify Files

3. Put Files

Codefast Solution Structure

So What Happened?

?

Production Cycle + Codefast = What Happened?

Pause to Reflect On The Possible Outcomes…

• Automation chaos worse than existing human chaos?• Limited scope of application?

• Only good for build function, but useless for test function?• Only good for Function X? Language X? Platform X?• Only good for Situation X? Variation X?

• It works, but has low quality success rates? 50%? 75%?• It works, but is too slow? No speedup? No punch?

• So many ways to fail, or to be limited in speed or quality

• A tough game, right?

How Do You Measure Quality Success?

σ

Perfect Quality – Our Best Moment

5 Months, 1 MLOC, 7688 Checkins, Zero Build Failures

Our Sigma Graph Over Time (~2 Years)

Codefast Perpetrator Report

• Reason Count (22 failures in about 1 year)

• Compile-dep 7 (one piece depends on another)

• Compile 4 (compile failure on other platform)

• Sys Admin 3 (bad shutting down machines)

• Bug-rollover 2 (special case defect in system)

• Unknown 2 (network - existing file “not there”)

• Missing-coll 1 (forgot to checkin a collection)

• Compile-set 1 (forgot a piece of the change set)

• Config files 1 (bad knowledge base change)

• Hardware 1 (machine/disk/network crash)

Our Track Record

1.75 Years, 1+ MLOC, 31541 Checkins, Never Below 99.99%

Technical Issues and Solutions

• “BIG” in Software Development Is Hard to Do– BIG doesn’t work well --- not BIG SIZE, not BIG PROCESS– BIG doesn’t scale well for speed, manageability, or quality– BIG is too slow, too complex, too many dependencies, too hard

• Learn to Work with Pieces – of Structure, Process, Responsibility– Must break down big code tree structures into smaller parts– Must break big recursive make processes into smaller parts– Must break human responsibilities into smaller parts– Then process parts in automated, parallel, distributed, incremental ways

• Structure, Process, and Responsibility Pieces– All three areas must be broken into pieces and combined together

Structure Pieces

• The Structural Granularity Problem– Big granules (such as big code trees) don’t work very well– Too big, too complex, too slow, too many dependencies– Too much surface area and exposure to failures– Can’t do parallelism or incrementalism easily on one big piece– Each granule is custom – it is irregular in size, language, processes

• A Granularity Solution – Collections– A generic “container” for code, files, knowledge of all types– Collections are like code objects, but are stored on a hard disk– They have a data type that determines permissible operations– Collections can do per-instance overrides of various kinds too– Internal collection structure is flexible– Multi-language, Multi-platform, Multi-Policy, Multi-Site, Multi-everything

Structure Pieces and Typical Examples

• Structure Pieces – Examples – Split builds by platform? No, that is just 1 big granule per platform – Directories in recursive make structures – ok, but weak pieces– Multiple IDE project files – ok, but weak pieces– Codefast collections map on to make directories or IDE trees – good

• Good Indicators of Structure Pieces– Pieces have individual names (identity by location vs identity by name)– Pieces have individual properties and custom overrides– Pieces can be relocated without breaking processes (abs pathnames)– Pieces can be shared and aggregated into different projects– Good pieces are like good objects, inherit, share, override, etc

Process Pieces

• The Process Pattern Problem– Scope of pattern is usually local in nature, not-sharable, not scalable

• Pattern usually belongs to a single program (make, ant, IDE)– Quality of overall process is limited by the person that writes the code

• Complexity – Every instance or combo of patterns is different– Responsibility for process pattern application is limited by human abilities

• Humans usually code every line in every make / ant tree structure

• A Process Pattern Solution – Knowledge Base + Generators– Scope -- Globally shared outside all project makefiles or IDE boundaries– Quality -- Automated application of global patterns to structural pieces– Responsibility – Human labor vs perfect parallel, incremental processes

Process Pieces and Typical Examples

• Process Pieces - Examples– Local make/ant rule templates – ok, but weak

• local, often language specific, usually tied to a particular program– Table-of-process-steps GUI tools – ok, but weak

• apply patterns to micro steps in a serial build– Best indicator is global process patterns that are automatically applied

• across N processes, N platforms, N machines, N configurations

• Good Indicators of Process Pieces– Pieces have individual names (identity by location vs identity by name)– Pieces have individual properties and custom behaviors– Pieces can be relocated without breaking processes (abs pathnames)– Pieces can be shared among multiple aggregated processes– Good pieces are like good objects, inherit, share, override, etc

Dependencies Among Pieces

• The Three Big Dependency Problems1. Dependencies among structure pieces (structure structure)

• Code A depends on code B

2. Dependencies among process pieces (process process)• Process A puts a file in location B, process C picks file up from B

3. Dependencies among structure & process pieces (structure process)• Change a structure piece, break the build process• Change a process piece, and it won’t fit the old structure

• A Dependency Solution – Extraction and Generation– Automatically extract dependencies among sets of structure pieces– Automatically generate custom processes to fit the current structure pieces– This way, processes always fit structures, regardless of ongoing change

Build Speed Solution #1

• The Speed Problem1. Must have structure pieces

2. Must have process pieces

3. Must calculate dependencies among pieces

4. Must coordinate multiple computers to work on pieces

• Speed Solution #1 – PerfectBuild Express – (CF-1)– Modify or extend existing code tree to make better structure pieces– Use existing processes (makefiles, antfiles, IDE project files)– Automatically extract dependencies from pieces– Automatically calculate parallel / incremental build order among pieces– Automatically calculate, schedule, and execute parallel builds– Benefit is faster time to value (parallel / incremental speed)– Less migration work required – Migration is simpler and faster

Build Speed Solution #2

• The Speed Problem1. Must have structure pieces

2. Must have process pieces

3. Must calculate dependencies among pieces

4. Must coordinate multiple computers to work on pieces

• Speed Solution #2 – PerfectBuild Enterprise – (CF-2)– Modify or extend existing code tree to make better structure pieces– Automatically calculate custom process pieces– Automatically extract dependencies from pieces– Automatically calculate parallel / incremental build order among pieces– Automatically calculate, schedule, and execute parallel builds– Benefit is maximum value (parallel, incr speed, + Six Sigma quality)– More migration work is required – but more value is gained

Software Build Approaches

1. No structure pieces, no process pieces, serially on N machines • More throughput, same speed, same quality

2. No structure pieces, no process pieces, parallel on N machines• More throughput, more speed, same quality• Only available for particular platform, language, tooling scenarios

3. Yes structure pieces, Your process pieces on N machines• More throughput, more speed, same quality• General solution, platform, language and tooling independent• Parallel (10x) and incremental (100x+) builds

4. Yes structure pieces, Yes process pieces on N machines• Maximum throughput, speed (parallel, incremental)• Maximum quality (Six Sigma)

Vendor Matrix

Structure Process Dep Speed SpeedSigma

Vendor Pieces Pieces CPUs Tput Mgmt Para IncrQuality

Baseline - - 1 - - - - -

Vendor-1 - - N - - - -

Vendor-2 - - N - -

CF-1 Yours N -

CF-2 N

1x 10x 100x

Build Speed Measurements

• Difficult to Measure in a Standard Way– Under What Conditions?– Specific to Particular Customer Trees, Tools, Situations, Policies

• CF-1 PerfectBuild Express– Use Codefast structure pieces, Use Customer Process Pieces– Best incremental speed measured at customer site– About 75x (7h 40min -> 6 minutes)

• CF-2 PerfectBuild Enterprise– Use Codefast structure pieces, Codefast process pieces– Best incremental speed measured at Codefast site– About 486x

Preferred First Implementation

Structure Process Dep Speed SpeedSigma

Vendor Pieces Pieces CPUs Tput Mgmt Para IncrQuality

Baseline - - 1 - - - - -

Vendor-1 - - N - - - -

Vendor-2 - - N - -

CF-1 Yours N -

CF-2 N

1x 10x 100x

Starting with CF-1Is the Easiest Way

Some Closing Thoughts

• Learning to work with pieces, parallelism, incrementalism– Small, Lightweight, Extremely Fast, but requires coordination

• Learning to work with Six Sigma failure rates (~ zero)– Must learn to trust the system more

– Like retirement—you must learn to do other things with your time

• 31,541 checkins over 1.75 years = ~20,000 checkins per year– 20,000 checkins means 20,000 builds / test runs per year

– If you can save 1 or 2 hours on each one, with 100x speedup,

– That’s 20,000 hours saved out of your production cycle

– With 99.99% perfect results

You Can Do This Too

1.75 Years, 1+ MLOC, 31541 Checkins, Never Below 99.99%

Our Product Name