New DevOps Tools for Native Code · New DevOps Tools for Native Code LINUXCON North America...

C. Thomas Stoverwww.thomasstover.com

New DevOps Tools for Native Code

LINUXCON North AmericaWednesday, September 18th, 2013Hyatt HotelNew Orleans, Louisiana11:50 CST

Revision Control Source Configuration

Collaboration

Packaging &Deployment

Build Automation

Continuous Integration

Test Analysis

Project Management

Some DevOps Interest AreasSome DevOps Interest Areas

(Scope of BCA and Build Matrix)

Build Configuration Adjust (BCA)

•Web Site: bca.stoverenterprises.com•Git Hub: ctstover/Build-Configuration-Adjust•Google Groups Mailing List: bca-discus•License: GPLv3

NONE.NONE.PROJECT_NAME = "Hello World "BINARY.MAIN.NAME = helloBINARY.MAIN.FILES = ./src/hello.c

$ ./configureconfigure: running ./bca --configureconfigure: creating directories for build host NATIVE...configure: directory . already exists.configure: directory ./obj already exists.configure: generating Makefile.bcaconfigure: Next use GNU Make to process Makefile.bca, ie "make -f Makefile.bca" or "gmake -f Makefile.bca".$ make -f Makefile.bcagcc -c hello.c -o ./obj/MAIN-hello.ogcc ./obj/MAIN-hello.o -o ./native/hello$ ./native/hellohello world

Build Tool Roles

•Dependency based, as needed, compilation lines (Make)•Project definition to Makefile generation (Automake)•Portable library abstraction (LibTool)•External / componentized code base integration (pkg-config)•Dynamic source / macro configuration (AutoConf)•Build environment / platform details detection (AutoConf)•Compiler / tool detection selection (AutoConf)•Optional Feature detection selection (AutoConf)•Custom Build Logic (Hard Coded Make, Scripting, utilities in native code)

Build Tool Roles

•Dependency based, as needed, compilation lines (Make)•Project definition to Makefile generation (Automake)•Portable library abstraction (LibTool)•External / componentized code base integration (pkg-config)•Dynamic source / macro configuration (AutoConf)•Build environment / platform details detection (AutoConf)•Compiler / tool detection selection (AutoConf)•Optional Feature detection selection (AutoConf)•Custom Build Logic (Hard Coded Make, Scripting, utilities in native code)

BCA Design Ideas

● “normal” build logic: ./configure; make; make install● “normal” command line usage: --disable-*, --enable-*, --without-*, --host=, --prefix=, etc.●installing additional programs not required●human readable Makefiles●human readable project files●human readable build configuration files●Full command line / scriptable, and text file interfaces●Always a simple manual override procedure●minimal to no dependencies: C99-ish●Configure time, and Build time performance really do matter●just abandon non-*nix style build hosts●ground up incorporation of pkg-config●build visualization (human readable graphviz dot output) helps.

Unix Wars -> Portability in 2013

•Many of the detection tests in Autoconf are: valuable, esoteric, complicated, not worth replacing, not worth discarding, etc.

•Really, package configure (pkg-confg), is a good idea, really.

“Solution” - platform details checking now means either 1) pkg-config, 2) slightly modified autoconf script, 3) any other external script you want to make.

Big deal:

pkg-config files (.pc) are used for all component interrelationships, both external and internal.

(btw Hacking a .pc to force something to work is infinitely superior to forcing LibTool to do anything.)

External dependency

Internal dependency

Packaging vs. Source Installations

•A practical, non-hateful, binary package support abstraction for all meaningful platforms is simply not possible. Really. No seriously.

•You know what? That doesn't matter.

•Source installations are the original portable distribution format.

•Source builds/installations that ACTUALLY WORK, perpetuate maintained, quality in-distribution packages. (and happy users)

Optional GUI may increase source installation useability.

Fighting Recursive Makefiles

•Miller, P.A. (1998), Recursive Make Considered Harmful•Console output utterly disgusting and confusing•Parallel build configurations can be accommodated by a single monolithic Makefile.•Use make -j X to full potential

$ bca --generate-graphviz$ dot -Tpng -o bcaproject.png bcaproject.dot

New Work flow Model

Run a greater and greater number ofbuild configurations, before handingoff to continuous integration.

New Work flow Model

Corollary: Let the build system - notcontinuous integration - driveparametrized builds.

Build Matrix•Web Site: buildmatrix.stoverenterprises.com•Git Hub: ctstover/Build-Matrix•Google Groups Mailing List: build-matrix-disc

Redundant Facilities in Continuous Integration (CI)

●SSH●Cron●Bash / Scripting Languages●User Access Paradigms●GNU Screen (slightly)●mail●Linux / Unix in general●build system logic (slightly)●version control logic (slightly)

What are the missing pieces?

•A Scoreboard•A Test History•A Build History

Build Matrix Design Ideas

● ssh (or fork()/execv() ) is the only relevant transport for this●use system users ●SQLite + directory tree for big files as backing store●distributed: peer to peer push + pull + list operations●web interfaces, GUI tools, test analytics can each get there own full copy to play with / mine●again just C99 ●very low overhead●no persistent daemon●just a “dumping ground” for build output and test results●minimum scope possible

Leaving Plantation J/X-Unit

suite: stringstest: escapenull: passedtest: escapeempty: passedtest: escapenotneeded: passedtest: escapeqoutes: passedtest: escapespaces: passedtest: escapemixed: passedtest: containsnullsource: passedtest: containsnullsearch: passedtest: containsnot: passedtest: containstring: passedtest: stringarrayaddnull: passedtest: stringarrayaddstr: passedtest: freestringarray: passedtest: stringarraynoduplicates: passedtest: pathextractnullpath: passedtest: pathextractnulloutput: passed

BCA - very near first release candidateBuild Matrix - nearing feature complete alpha prototype

cts at thomasstover.com