View
242
Download
0
Category
Preview:
Citation preview
B3 | Modeling Unbreakable Builds
Eclipse Modeling DayNov 16-19, 2009
Ed Merks, Bjorn Freeman-Benson, Miles Daffin
Modeling Day 2009 1b3 | modeling unbreakable builds
agenda
1. usage scenario: eclipse release assembly
2. b3 project: overview
3. usage scenario: enterprise provisioning
Modeling Day 2009 2b3 | modeling unbreakable builds
eclipse as singleton
o Eclipse starts as one project: Java IDE.• Just one .zip file to download
• Just one update site for updates
• Extra features added manually through complex update menus and wizards
• Java IDE built using custom scripts and headless instances from IBM .
Modeling Day 2009 4b3 | modeling unbreakable builds
eclipse as a small collection
o Eclipse expands to a few core projects• Each still responsible for own builds
• Some copy the complex Java IDE build; others use simpler custom,scripts
• Each has its own downloads and update site(s)
• Each has its own release schedule and versioning scheme
• Consumers struggle with version compatibility and … (worse) …assembling the pieces for a working installation
Modeling Day 2009 b3 | modeling unbreakable builds 5
assembling the pieces
o Assembly has been a problem since the start of the software industry
o Assembly = “which source files are compiled and linked into an .exe” or "which versions of which components are packaged into coherent whole“
o Problem grew as software shifted from a single binary executable into multiple source files/ executables and then multiple libraries
Modeling Day 2009 6b3 | modeling unbreakable builds
many solutions
o Multiple generations of tools: make, autoconf, MSBuild, cook, ant, maven, and even winzip and msi.
o Commonality: assembles result X from inputs A, B, C and dependencies.
o Result X may be:
• a compiled binary
• an installed program...
• a library of compatible components
Modeling Day 2009 7b3 | modeling unbreakable builds
callisto and europa (2006/7)
o Need: consumers wanted single Eclipse release with all the correct pieces
• Solved by internalizing problem assembling a distro: the Callisto coordinated release.
o Consumers happy but assembling the single update site was painful
• Committer Dave Williams (IBM) wrote set of perl, grep, awk, and ant scripts to get the bits & package them into an update site.
• It broke more often then it worked!
Modeling Day 2009 8b3 | modeling unbreakable builds
new problems
o It broke due to:• New hidden dependencies
• New explicit dependencies people forgot to declare in the xml files
• Version numbers changed
• Included old & unused (thus broken) code
• Conflicting dependencies
o It worked by:
• loading everything into an Eclipse image
• checking to see if it worked
• slow and inherently error prone
Modeling Day 2009 9b3 | modeling unbreakable builds
ganymede and galileo (2008/09)
o “Dave-o-matic”• Committers Thomas Hallgren and Henrik Lindberg (Cloudsmith)
created new system to replace the custom assembly scripts
• Worked better because it used the (powerful) Eclipse/Buckminstertechnology to fetch components and resolve dependencies.
• Changing output format from the old update site (Ganymede) to new p2 site (Galileo) relatively trouble-free because descriptions stayed constant when the tool changed
• Still not perfect: custom external scripts and incomplete XML configuration files
Modeling Day 2009 10b3 | modeling unbreakable builds
what next?
o Simultaneous release requires same things as anyone working on complex component-based software
o Formal descriptions of.
• components
• dependencies
• processing
• “the result”
o Execution engine that understands all the different component metadata and file formats.
o In other words, modeling!
Modeling Day 2009 11b3 | modeling unbreakable builds
target problem: why are builds so #$%~! complicated?
o Software is developed with the latest technology…separation of concerns, dependency injection, declarative services, OSGi, modeling, etc.
o Is built & assembled using the oldest technology...hand-coded and monolithic scripts created through successive hacks
o So it's no surprise everything breaks whenever something changesor someone else tries to run it
Modeling Day 2009 13b3 | modeling unbreakable builds
b3 project objective
o New generation of Eclipse technology to supporting build & assembly processes that are simple, repeatable, and adaptable:
• declarative, model-driven, DSL-based process definition
• support for discoverable, reusable build/assembly actions
• support for composable build/assembly processes
• separation between process definition and execution (who/where/when)
• debuggable processes
o Project will deliver a successor to current PDE/Build and consolidate/clarify related technologies (p2, Buckminster, Athena, etc.)
Modeling Day 2009 14b3 | modeling unbreakable builds
b3 functional overview
D
E F
target
C
B
G H
workspace
A
A
B
C
D
build model
resolversmeta data translatorsconnectors
import(materializers)
b
cd
“advice” engine actors
for(;;) …javac –xyzant …gcc …
events
15Modeling Day 2009 b3 | modeling unbreakable builds
goal: maximal simplicity
Give me A.
= give me A, but I also need source for
debugging, and B has a faulty version range,
and…and...and...
“advice”
16Modeling Day 2009 b3 | modeling unbreakable builds
drill-down: b3 DSL
o B3 build files contain info about:• how the environment should be set up
• how the model should be advised
• what actions can be performed on the build file
o B3 build files are an XText DSL• concise, powerful way to specify build details based on "advice"
• comparison: XML-based approaches
• Alternatives to the build file are also possible (i.e. programmatically using EMF, MoDisco, Xtext )
17Modeling Day 2009 b3 | modeling unbreakable builds
drill-down: b3 DSL
o Declarative syntax for
• build unit (name, version, visibility, parallel or sequential execution)
• build unit interfaces (compare to p2 BU namespace)
• provided and required capabilities
• build part structure
• properties
• advice
• repositories (and more advanced resolution support)
o Script syntax for build part actions
• javascript like syntax for imperative programming
• functional programming additions
• literal queries (selecting model features, files, results etc.)
• set operations, select, reject, collect, exists, foreach
18Modeling Day 2009 b3 | modeling unbreakable builds
build file example
unit {requires {
eclipse.feature/org.myorg.myproj/1.0;}
repositories {platform:plugin:/;platform:resource:/;p2:http://www.someplace.good/updates-3.5;svn:svn+ssh://org.myorg.repo/productx;}
main {input { org.myorg.myproj#siteP2;
org.myorg.myproj.packaging;}
}}
19Modeling Day 2009 b3 | modeling unbreakable builds
broad implications
o “Building" is more than just producing a binary from source• Assembling components
• Producing a single exe
• Producing an Eclipse update site for features
• Producing an msi installable unit
• Installing an msi on a computer (!)
• Producing an internal library of “blessed” & compatible components
Modeling Day 2009 20b3 | modeling unbreakable builds
overview
o Enterprise constraints & consequences
o Enterprise provisioning challenges
o Past: delivering 3.2 - 3.4 with internal solution
o Present: delivering 3.5 using b3 predecessor
o Future: building on b3
Modeling Day 2009 22b3 | modeling unbreakable builds
enterprise constraints & consequences
o Constraints: cannot allow developers free access to external Eclipse environment
• Against firm policy
• Risky (malware,legal & licensing)
• Inefficient, chaotic & hard to support
o Consequences: restrict users to private Eclipse inside the firewall
o Requirements:
• Easy to keep up to date
• “Clean” IP and safe to use
• Provides transitive closure of all dependencies
Modeling Day 2009 23b3 | modeling unbreakable builds
provisioning challenges
o Objectives: • Provide everything our developers need.
• Ensure internal environment is stable and reliable.
• Reduce Eclipse TCO
o Requirements• Security & Legal Requirements
• Repository aggregation & Mirroring Requirements
• Installation Requirements
Modeling Day 2009 24b3 | modeling unbreakable builds
security & legal requirements (1)
o Primary requirement: restrict user access to internal repos for browsing, installs and updates
o Secondary requirement: purify repositories of malicious code and problematic IP
Modeling Day 2009 25b3 | modeling unbreakable builds
aggregation & mirroring requirements (2)
o Mirror Eclipse Platform + 'Jupiter Moon' repositories (core stuff)
o Shopping list of 3rd party software from: update sites; p2 repositories; downloads (update site, p2 repository, feature, plugin)
o Ensure all dependencies are met internally
o Add homegrown stuff to the mix
o Keep resulting configuration up to date
• Reduce lag between external and internal updates
• Ideally, get updates automatically
o Expose updated configuration to users in stages
o Edit the configuration easily (add/remove software)
Modeling Day 2009 26b3 | modeling unbreakable builds
installation requirements (3)
o Eclipse installer must:• install a version of eclipse based on a custom profile (e.g. a
shopping list of different plug-ins sourced from internal repositories)
• enable the install to be run in a custom environment (e.g. perforce executables on PATH so plugin will work)
• install an arbitrary selection of “other stuff” with Eclipse install (e.g. JDKs)
• create a “default” install configured with standard global preferences (e.g. java code format, installed JREs)
• support desktop shortcuts, etc.
• be reliable & easy to configure and maintain
Modeling Day 2009 27b3 | modeling unbreakable builds
past: internal solution for 3.2 - 3.4
o Internal solution was basically, a nightmare!• Mirroring: did it work? Gave up all Eclipse-based mirroring tools for
Ganymede SR2 (used wget from unixheads...)
• Proprietary mirroring tools not much help
• Test process was manual and awkward (install everything into a vanilla Eclipse!)
• Complex ant scripts needed to post-process mirrored repos
• prevent Eclipse from “phoning home”
• hard coded to unstable Eclipse implementation internals
• generate p2 metadata
• Installer was a hack
• Four internal repositories to be kept in sycn
• Could take 3 days to update the main 3rd party repo
Modeling Day 2009 28b3 | modeling unbreakable builds
present: b3 predecessor for 3.5
o B3 predecessor technologies (Buckminster Aggregator + P2 director) makes it manageable
• Use Buckminster Aggregator for mirroring, verification, testing, etc.
• Only two repositories needed to create releases: private Galileobase repo + public “mega“ repo aggregating everything else
• Only maintain 2 build models needed: one for each repo.
• Installer uses P2 director to install straight from the mega repo. Much simpler.
• Can build new release of the mega repo in 2-3 hours instead of days
Modeling Day 2009 29b3 | modeling unbreakable builds
future: building on b3
o Use b3 models to drive/simplify entire Eclipse delivery process• Model build from revision control to source repository (input for
aggregation)
• Add heuristics for coping with aggregation build problems: missing repositories, IUs, artifacts.
• Add heuristics for auto-update of pre-defined aggregation model
• Model more of the installation (non-eclipse IUs, global preferences, etc.)
• Intelligent error reporting: “this is what I found” … “this is what I fixed; this”… “ this is what I couldn’t fix and why”
• ->Ultimately, just run it all headlessly, once a week.!
• B3 IDE – easy-to-use tools for designing, running, testing and debugging b3 builds
o Extend into other build/assembly domains: Java; C++; .Net, etc..
Modeling Day 2009 30b3 | modeling unbreakable builds
Recommended