Page 1October 31, 2000

An Introduction to Large-Scale Software

Development

Steve VarnauCore HP-UX Operation

October 31, 2000

October 31, 2000 Page 2

Large-ScaleSW

Devel. Agenda

• Software Change

• Software Configuration Management (SCM)

• Change Control

• Large-Scale Software Development

• Example: HP-UX Operating System

October 31, 2000 Page 3

Large-ScaleSW

Devel.

The Need for Managing Software Change

• Environment Changes

• Computer Hardware

• Competition

• Business Processes

• External Requirements (Y2k, Euro,…)

• New Standards (XML, MP3, e-Speak,…)

• All Software Demands Change

• Incremental Development

• Maintenance (Corrective, Perfective, Adaptive)

Software Evolution

October 31, 2000 Page 4

Large-ScaleSW

Devel. Benefits of Source Control / SCM

• Manage Constant Software Change

• Improve Debugging Productivity

• Trace Defect Reports to Correct Versions of Source Code

• Understand Change History

• Reproducibility

• Build (Compile) Software Accurately, Repeatably

• Grow Project Size

• Avoid Over-writing and Losing Changes

• Concurrent Development (Improve TTM)

• Coordinate Work with Development Policies

SCM

October 31, 2000 Page 5

Large-ScaleSW

Devel. Terminology - File Versioning

• Revision / Version - Single copy of a source file as it appeared at given point in its change history (checkpoint)

• File / Element - Complete change history; all versions

• Branch - Set of successive versions; parallel to other branches

• Delta - Change from one file version to the next

SCM

October 31, 2000 Page 6

Large-ScaleSW

Devel. Terminology - Version Tree

0

1

2

3 1

0

Version

Branch

Filefubar.c

Delta

SCM

October 31, 2000 Page 7

Large-ScaleSW

Devel.

Terminology - Multiple File Configuration Management

• Variant - File based on another, co-exists with original

• Derived Object - File automatically generated from source files

• Change Set - Set of versions across multiple files making up a single logical change (e.g., feature or fix)

• Baseline - Consistent set of versions across all files

SCM

October 31, 2000 Page 8

Large-ScaleSW

Devel.

fubar.c foo.c bar.c

Terminology - Baseline

SCM

October 31, 2000 Page 9

Large-ScaleSW

Devel. Coordinating Work

Problem: Multiple people working on same file at the same time can result in lost work.

• Solution 1: Serialize work with file locking.

• Solution 2: Work in isolation (branch) and merge.

Branching and merging scales much better.

Merging can be semi-automated using 3-way merge algorithms.

SCM

October 31, 2000 Page 10

Large-ScaleSW

Devel.

0

1

2

1

0

CommonAncestor

ConflictingChanges

3

MergeResult

Merging - Version Tree

SCM

October 31, 2000 Page 11

Large-ScaleSW

Devel. Formal definition (IEEE)

SCM is the process of identifying and defining the items in the system, controlling the change of these items throughout their life cycle, recording and reporting the status of items and change requests and verifying the completeness and correctness of the items.

Related Terms: Source Control, Version Control

SCM

October 31, 2000 Page 12

Large-ScaleSW

Devel. Managing Change

Change Control - Formal processes for tracking and approval of software modifications.

Managed with SCM and defect tracking tools.

Usual Policy Areas:• Documentation - Change request / defect report

filed.• Approval - Change request reviewed by release

management or technical board.• Quality Requirements - Code reviewed, regression

tests passed.

Policies often change by release stage and in response to problems.

Change Control

October 31, 2000 Page 13

Large-ScaleSW

Devel. Development Process

Key Components of Development Process:

• Software Configuration Management

• Change Control

• Quality Control

Change Control

October 31, 2000 Page 14

Large-ScaleSW

Devel.

Principles of Large-Scale Software Development

• Reproducibility

• Policy Enforcement versus Policy Auditing

• Process, Process, Process

A process must be defined to be improved

• Automation

A process must be automated to be repeatable

• Continuous Functional Integration and Test

Scaling

October 31, 2000 Page 15

Large-ScaleSW

Devel. Software Integration

Integration: Combining multiple features or components developed in isolation.

Large projects are subject to constant change, so isolation on a stable base is necessary.

Continuous Functional Integration and Test (CFIT)

• Changes integrated throughout development.

• Changes not allowed to regress functionality or quality.

Scaling

October 31, 2000 Page 16

Large-ScaleSW

Devel. HP-UX Development

• 220,000+ Source Files

• ~2.5 Million Versions

• 52,000+ Derived Files (Single Nightly Build)

• 900+ User Accounts

• More Than 1 Terabyte of Disk Space

HP-UX

October 31, 2000 Page 17

Large-ScaleSW

Devel. HP-UX Concurrent Development

• Normally 2-3 Releases Under Development

• Several Releases Under Maintenance

• Heavy Use of Branching and Merging

HP-UX

October 31, 2000 Page 18

Large-ScaleSW

Devel. HP-UX Multisite Development

• Four Primary Development Sites

• Three in USA (Three Time Zones)

• One in India

• One USA Site Hosts Three Logical Sites

• Partner Companies

• Several Secondary Sites

HP-UX

October 31, 2000 Page 19

Large-ScaleSW

Devel.

Tiered Software Integration Model

1. Developer Task - Edit, Compile, Test

2. Submit - Batch Changes, Acceptance Test

3. Baseline - Nightly Build, Regression Test

4. Integration - Monthly Packaging, Partner Test

5. Release - Semi-annual Delivery, Production Use

HP-UX

October 31, 2000 Page 20

Large-ScaleSW

Devel. Integration Levels

Build

Submit

Baseline

Integration Cycle

Release

Modifications

HP-UX

October 31, 2000 Page 21

Large-ScaleSW

Devel. Process Evolution

• Continuous Process Improvement

• Root Cause Analysis (RCA)

• Capability Maturity Model (CMM)

1. Initial

2. Repeatable

3. Defined

4. Managed

5. Optimizing