OSDC 2006 Presentaton: Building with a Version Control Audit Trail

Building with a Version Control Audit Trail

Alec ClewsVoga Consulting

http://voga.com.au/http://alecthegeek.wordpress.com/

http://voga.com.au/

http://alecthegeek.wordpress.com/


Summary

● Building software in an audited and managed environment as part of our Configuration Management (CM) process

● Using Open Source Tools (GCC, GNU Make and Subversion) as a framework for a controlled build environment


Agenda

● The different types of build and the place of the system build

● Auditing and logging requirements for system build

● Implementing a CM System Build Framework– Version Control Set-up– The Makefile– The Driver script– The output


FLOSS and a culture of formal change management

● This paper takes a “commercial” view on the process of software change management

● However most aspects are still valid for an Open Source environment– Desire to provide a quality use experience– Need to communicate effectively between team

members– Reduce the amount of unsatisfying project work

● rework, report writing,etc...


The Change Lifecyle

● Document and agree the change● Implement the change (hack the code and

desktop build/testing)● Integration build and unit test● System Build in a controlled environment● System Testing (or UAT, or beta, or whatever)● Release for distribution or production


Continuous Integration vs. System Builds

● Continuous Integration(see Martin Fowler)– Builds rapidly for and frequently, often on each

commit– Runs unit tests– Will need to set-up for unit testing framework– Happens automagically

● System Builds– Build for production, or nearly production

● Compiler optimisation, different configurations etc.– Runs on a different schedule


How might the system build be done?

● Fetch the sources and build them● Ask the developer for already built software!● Issues with these approaches. How can we

answer such questions as?– Which source versions were used– Under what configuration was this built?

● Compiler switches, which libraries and tools (the build configuration)

– Can we “re-create the build”?


The “re-create the build” myth

● When a problem happens we often want to re-create the environment for further testing

● But if we re-run the build process that does not assure that we will get the same result– Dates and time change, this trivial difference masks

possibly other major changes in our files– We usually don't have a record of the previous build

configuration and can't re-create it anyway● Net result is that we can't reliably get the same

result twice


Consistent Build Process

● It is possible to define a consistent build process by using tools such as Make, Ant or CONS and some additional scaffolding

● Must be done in a controlled environment● Must be done with pre-defined scripts and documented process

● Scripts should be under version control● A record should be keep of all system builds


Further Improvements

● Use the consistent process● After the system build save the build directory

tree (source code, makefiles, build log, built files) into an archive (consider burning a CD)– This is our built baseline. We can always take this

off the shelf and see what software we used to “re-create the build”.

● However it's often hard to identify source versions and what build environment was used


Build Environments

● Consider building on virtual hosts● The host image can be archived as changes

are made● Becoming more popular as VMWare, Xen etc.

etc. become available for zero cost


There is a better way

● As we build new files identify– The source files and their version number– The tools and the settings used– The environment used (platforms, packages etc.)

● At the completion of a successful build we can preserve the outputs in Version Control– Including a complete audit trail of the environment,

the build log and the input files


The result

● In our version control system we have a complete audit trail of each system build

● We can extract our built files at any time – now we can “re-create the build” just by fetching files from version control

● If our change ticket system is integrated into version control we can identify which changes when into a build

● We can do impact analysis


Why do we want to do this

● We want to close the change management loop– A “Line of site” from change request to deployment

● Keeping the auditors happy– N.B. May be a legal requirement to keep the

external auditors happy. In some environments auditors can shut down the business if they are unhappy with our software process


Example Implementation

● There are a variety of ways to this● Following example uses

– A simple C project (5 source files)– GNU Make– Subversion

● Some commercial products also available.– e.g. Electric Cloud


Subversion (SVN)

● Now a major player in Version Control market● Each commit generates a whole new version of

the repository● Baselines are created using tags. A tag is a

new leg in the repository directory tree– SVN users use naming conventions for tags,

vendor tags, trunk and branches● SVN supports properties: name/value pairs that

store arbitrary information against file revisions


Some concepts and words

● Build Tags. Tags created for each build we perform. Each build creates a new SVN directory tree in our build project c.f. Build Baseline

● A Build Project. The logical grouping for the work performed by running a makefile. Each project has it's own tag directory in SVN

● A Build Configuration. Documentation of the environment in which the build is being done (e.g. Platform, compiler, library versions...)


Example SVN Layout● Source code● Marks the build tags

tree● A build Project● Build


The Makefile

● To create the audit trail we must know the complete list of dependencies– i.e. All the source files used to build a target file

● Peter Miller already did the hard work for me– His paper explains an effective way to implement

makefiles with a complete dynamic dependencies– A side affect is that your makefile is more efficient!


Dependencies in C

● A list of input files that an output file depends on. e.g.– a.o: a.c.a.h b.h– main: main.o a.o b.o

● Dependencies can change any time we edit the source code, including editing any header files

● We must re-calculate the dependencies at make time if the source files or header files change


Dynamic dependency

# Generate a dependency file$(DEP_DIR)/%.d: $(SRC_DIR)/%.c

$(CC) $(CFLAGS) MM MG MT $(OBJ_DIR)/$*.o MT \

$(DEP_DIR)/$*.d MF $@ $<

Creates a file that contains o/a.o d/a.d: src/a.c h/a.h

Include the dependency file in the makfile# Find the name of all the Dynamic dependency file and include

them

include $(patsubst $(OBJ_DIR)/%, $(DEP_DIR)/%, $(patsubst %.o,%.d, $(OBJ)))


Built From List

● When we preserve a new target we want to record

1.Unique ID of each source file used

2.Revision number of each last source file change● This is the Built From List

– The list of files the target is built from● SVN properties can be used to store the Built

From List (BFL)– Uses a CSV format for simplicity


Calculating BFL

● When a file is build from complete dependency list we can also discover the CM data for the dependencies our target is built from

# Generate a Built From List, where the object file came from a C file and some header files

svn info $^ | sed e '/^Path:/,/^$$/ {\

:ack N;\

/\n$$/! b ack\

s/..*\nURL: $[^\n]*$\n..*\nnnnnnLast Changed Rev: $[^\n]*$\n..*$$/"\1","\2"/g;\

}'


Example BFL

● Shows the file id's and their last modification revision number

"file:///home/alec/Voga/laptop/gigs/OSDC/repo/build/BuildProject1/00000003/src/a.c","1"

"file:///home/alec/Voga/laptop/gigs/OSDC/repo/build/BuildProject1/00000003/h/a.h","1"


Storing the target & BFL

● Every time we commit a new built target to the repository– Add the new target to the repositorysvn nonrecursive add $(@D) \ $(REPO)/build/$(BUILDPROJECT)/$(BUILDNO)/

svn add $@ \ $(REPO)/build/$(BUILDPROJECT)/$(BUILDNO)/

– Concatenate the BFL's for the dependenciessort u $(BFL) > $(BFL_DIR)/$(@F).b

– Add the BFL to the repositorysvn file $(BFL_DIR)/$(@F).b propset BFL $@


Driver script sets up

● Build ID – a simple eight digit integer (global)● The Build Configuration. A simple string

displaying the build environmentBUILDCONFIG="$(cat /etc/SuSErelease |\

head 1 ) $(cc version|head 1 ) \

$(rpm q glibc)"

● The host name on which the build took place – actually not very important


Further work

● Better Error Checking!!● Save and re-use built intermediate files

– (but see constraints later)● Update tickets with “Fixed In Build” information


Further work contd.

● Integrate with build control tools– e.g. CruiseControl

● Integrate with meta make tools– e.g. cmake, autoconf/automake, ...

● Automate the production of release notes and other reports from the BFL and other version control information


How to re-use targets in later Builds

● Validate md5sum values on source files on original BFL against source files in Build areas – must match. SVN uses md5 as well

● Check build configuration for target on build tag against current environment – must match

● Check command line switches – must match● We don't care about dates and time


Outstanding Questions

● Does this scale?– More complex directory structures– Multiple languages (e.g. grammar files &

assembler)● What about languages and tools with hard to

calculate dependencies (e.g. Java, code generators,...)

● Would it be better to record the SVN UUID rather than the URL in the BFL


Outstanding Questions contd

● Can this work with other version control tools– Centralised: CVS, Version Manager,...– Distributed: git, bazaar, hg,...

● Will this work across platforms– differences in tools– md5sum problems because text files change?


Further reading

● Example scripts (written for this paper)● “Recursive Make Considered Harmful” by Peter

Miller http://www.canb.auug.org.au/~millerp/rmch/recu-make-cons-harm.html

● “Continuous Integration” by Martin Fowler http://www.martinfowler.com/articles/continuousIntegration.html

http://www.canb.auug.org.au/~millerp/rmch/recu-make-cons-harm.html

http://www.martinfowler.com/articles/continuousIntegration.html

Economy & Finance

OSDC 2006 Presentaton: Building with a Version Control Audit Trail