Understand Release EngineeringQuality Software Release

Liang Gao

We’ll cover

• What is release engineering.• Why do we need release engineering• Release version planning• Branch management• Build process• Release process• Source code control• Release engineering best practice

What is Release Engineering

Deliver

Deliver

Why do We need Release Engineering

• Do you know how many releases Cisco IOS has?

Why do We need Release Engineering

• Do you know how many platforms Cisco IOS runs?1000 Series 1600 Series 1700 Series 2500 Series MC 3810 4500 Series

7000 Series 7400 Series 7500 & 7000 RSP Series 7600 OSR Series 10000 Series

  12000 GSR Series

  Catalyst 4000

  Catalyst 5000

  Catalyst 6000 & 6500

What is Release Engineering

• build a good quality binary • Source + tools = product

What is Release Engineering

• Compile • Verifying that the software on the

(uncharacterized) end user system continues to execute properly

• Source code quality control• branching and merging separate but parallel

codelines of development • Make it a predictable, quality release

Overall PictureCustomer

Marketing/Product

Release Planning

Release Engineering

QC

DE

Customer

Types of Releases• Major: Introduce major functionalities, major infrastructure changes

unaware by users• Minor: Introduce minor enhancements and bug fixes• Maintenance/service: Bug fixes, mostly critical bugs found by customers• Patch: Bug fixes targeted toward on customer• Platform: Release only one selected platforms, mostly introduce new

hardware platforms (ASIC, new hardware revision, component changes

Sample Release Process IntroductionQC lead identifies an official build

as release candidate

Build master rebuilds the candidate with a release name using build tool

Build master puts the signed image to release directory

Auto Regression

QC Lead to announce Exit QC

Technical Writer and QC provide Release Note

Store image, Go Live, Sales notify customer

Release Manager's question

• Which release contains this defect repair? • Which releases have this defect fixed? • Which codelines contain this defect repair? • What files have changed between these two

releases? • What defects have been added/dropped

between this release and that release? • What changes have been added/dropped

between this release and that release?

Introducing Branch and its Management

Branch Management

Release Management

• Customers may use different versions• How many branch you need to support• XP and Vista will be co-exist.

Release Management1.1 1.2 1.3 1.4 2.0 2.1

1.1r1 1.1r2 1.1r3 1.1r4 1.1r51.1Throttle

1.2FCS 1.2r1 1.2r2 1.2r3

2.0

3.0 3.0FCS

1.2Beta

Bug Fix

Bug Fix

Bug Fix

2 month

Maintain Multiple Releases

1.1

1.2

1.1r1 1.1r2

1.2r1 1.2r2

One Testing Group

Maintain Multiple Releases

P4 – Beta31 week 2008-09-17 2008-09-23

P4 - Golden Week 10 days  2008-09-24  2008-10-9

Extended time period 4 days 2008-10-10 2008-10-15

P4 - FCS - Internal 5 days  2008-10-16  2008-10-22

Sanity on CTU0z0r1, all New Features need to be run.

10/16-10/17

FCS regression Suites : Security ScanflowSanityRelease jobAll Featuress sanity (Besides private script, please also do manual sanity on al lFeaturess)System level testing automation jobsUpgrade testingChecksum testing

10/17-10/23Official images

Confirm files are on staging srv 10/23 FCS

Merge Process

Merge Process

• Get the Diff• Use CVS or Subversion to find code conflict• Get developers to result• Compile binary• Auto regression• Code commits

Branch Process

• Copy over

Bug Fix Checkin Process

• A field in bug report: To be fixed in• It is filled by DE manager case by case• Forward Fix

– older version found bug, that will need to be fixed in the later version

• Backward fix– Current version’s old code found bug, need to be

fixed in the previous version as well

• Both forward and backward fix• A tool can be used to auto propagate the fix

CM Tools For Source Code Control

• CVS• Subversion• Rational Clearcase

Stable Branch

• After a full regression, no P1 bugs, high percentage of the regression test cases passed. No many regression bugs.

• Stable branch take a great effort to get. – Regression and bug filling– Bugs fix– More regression– ….

Stable Branch Strategy

• Main branch is stable– Release branch is for new code development– Strictly code merge criteria

• Release branch is stable– New branch contain most of the new code

(every one can check in– Extensive testing on the release branch

Branch Management in Real Life

Some of The Best Practice in Process

• New feature Code Checkin Process• Bug Fix code checkin process• Branch open and close time process• QC on Branch Control• Code Freeze• Nightly build and smoke test• Customer Patch• Testing Coverage on different release• Release Criteria Sample

New code check in process

• Developer checkout the private branch and develop the code. • Developer drop the private branch build image for tester to test• Test file bugs and developer fix bugs on the private branch• Code check in criteria

– No P1 bugs– P2 bugs no more than 2– Code coverage 75%– Automation script is developed. – Tester has authority to veto the commit– `

Bug fix process.

• Developer checkout a private branch for bug fix• Developer done code review• Developer provide code diff in the bug report• Developer provide the simple test to show bug is

fixed• Developer run small scale regression on the core

feature that might be affected by this bug fix• Code check in.

Branch Open and Close Windows Timing

• For new features, open the branch (that code can be checked in) on a specific time. Accept commit only it meets the commit criteria.

• If the feature missed the branch open time, it has to wait for the next round (performance impact on the developer)

• Customer found bug, regression bug code fix can be checked in through out the cycle. Use nightly build and daily smoke test

• Get on the regression train whenever possible. • More regression more stable of your branch

Branch Open and Close Windows Timing

Close CloseCloseOpen Open

QC on Branch Control

• The firmware is always built by QC to ensure the source code is in repository and the quality of the firmware. Every firmware built including the source code is kept and backup for reference.

• QC will not accept firmware from developer for testing. If developer request to test his/her firmware for bug fixes, QC will re-test again using QC built firmware.

• The firmware build can be done by demand or kick off by cron job. Every internal build from QC is a complete checkout from the latest source tree.

Code Freeze

• Feature Freeze• Code Freeze

– When the code is ready for beta and freezed, all source code checked in require bug id, no new feature check in will be allowed and source tree is locked. The code will be reviewed by code reviewer. Once it is reviewed, the QC project lead will open the permission for the developer to check in.

– Any last minutes bug fixes require reviewed by two chief architects. QC Project lead will determine the number of line changes and the risk involved to allow the bug check in.

Development and Testing At a Glance

MRD/PRDFunctionalSpecification

CodeDevelopment

Test Plan Test Case

CodeReleaseto Test

Code toTest

AlphaStart

Scripting

Dev

Test

CodeControl Opened Bug ID

Feature Freeze

Code FreezeBetaStart FCS

Various Testing

CodeReview

Bug Fixes

Form

GoldenWeek

Daily/Frequent Code Drops

Nightly build and smoke test• The Nightly Build and Smoke Test is a process in which a software product

is completely built every night and then put through a series of tests to verify its basic operations. This process is a coding-stage process, and it gets initiated at the start of the coding phase of the project. The process produces its savings by reducing the likelihood of several common, time-consuming risks – unsuccessful integration, low quality, and poor progress visibility.

• The Nightly Build and Smoke Test can be used effectively on projects of virtually any size and complexity.

• It is the responsibility of the Build Group to create the required environment for running the nightly builds and monitoring it on a daily basis.

Customer Patch

• For customer patch release, a bug id is required. • Test Engineer will try to reproduce the same

problem in test environment. • Developer will determine the cause and fix

accordingly. • Since this is patch release, we will normally make a

copy of release source directory, changed the version to identify the customer, check in only the fixes, and build.

• Test Engineer will test again to verify the problem is no longer there before release to customers.

Types of TestingTest Type Description

New Feature Test Perform new feature tests according to MRD, RFE and functional specification. The new feature test cases will become part of regression test once the product is released.

Regression Test Perform feature tests that is already available in the past releases, this is to ensure the existing working features do not break in the new release.

System Test Ensure device/software works as a whole by exercising all major components in hardware and software.

Interoperability Test Software interoperability ensures the implementation works with our products and other security products. Hardware interoperability ensures our hardware devices work with other hardware vendors to test such as interface link, NSRP, etc.

Performance Test For NetScreen devices, measure the throughput of firewall and IPSec and ramp up rate. For Global PRO, we measure the number of devices Global PRO can support.

Capacity Test Measure the maximum supported capacity listed in product specifications such as MRD/PRD/RFE or datasheet.

Security Test Run through list of well-known security scanners and attack software to assure our devices are not vulnerable to these well-known attack software.

Automation Test Automation does not provide new test cases, we develop automation software to facilitate manual regression test, we have more than 30% of ScreenOS regression test cases run in automation environment.

Types of Testing PerformedMajor

ReleaseMinor

ReleasePatch

ReleasePlatform Release

SFR CSP

New Feature Test Full Full TBD Full

Regression Test Full Partial Partial Partial Partial

System Test Full Full Full Full

Interoperability Test Full TBD TBD TBD

Performance Test Full TBD Full TBD

Capacity Test Full Full TBD

Security Test Full Full Full Full Full

Automation Test(partial regression)

Full Full Full Full Full Full

SFR – Special Feature Release CSP – Customer Specific Patch

Sample Release Criteria

• Long run tests should be executed for 5 days without QC showstopper being found.

• There are no open critical bugs characterized by data corruption or card or machine freeze.

• There are no open ship stopper bugs characterized by bad customer experience such as failure to install and configure.

• Less than 1 incoming critical bugs per week Incoming bug trend should be checked for decline over time.

• No more than 10 open bugs at GA end.• Metrics equal to or better than last product release.• 5 days of long run tests with no failures