ESE2007 Buckminster Gany-matic - EclipseCon … Summit 2007 21 BUCKMINSTER HIGHLIGHTS • Resolution & materialization framework – Manages and resolves dependencies – Materializes

Buckminster

BUCKMINSTER & GANYMATIC Eclipse Summit Europe, 2007

Bjorn Freeman-Benson (Eclipse/Dash)�

Thomas Hallgren (Eclipse/Buckminster)�

Eclipse Summit 20072

AGENDA

1. Why Ganymede?

2. Ganymede & Ganymatic

3. How Ganymatic works

4. Using Ganymatic

5. Ganymatic Roadmap


1. WHY GANYMEDE?


WHAT IS ECLIPSE?

• Eclipse is many things, but for this talk, Eclipse is:

– A collection of plug-ins which…

– When combined in various sets…

– Provide useful tools for software developers

• Eclipse is “everything and nothing”…

• …that is both its power and its complexity


HOW DO USERS USE ECLIPSE?

• Users download and install an Eclipse distro

– Optionally use the update manager for additional features

• A distro is a set of Eclipse plug-ins

– Binaries


HOW DO DEVELOERS USE ECLIPSE?

• Developers load and code in a workspace

• A workspace is (source code for) a set of Eclipse plug-ins

• Binary distro ~ workspace…

• …a set of Eclipse plug-ins


THE CHALLENGE IN BUILDING A DISTRO

• Plug-in A needs plug-ins B and C, and they need plug-in D

• Each plug-in (we call them components) has a local view of its dependencies

• But to assemble a distro, you need the global view - the transitive closure

• As components (plug-ins) evolve, their dependencies change…


WHO BUILDS YOUR DISTRO?

• Eclipse distros are assembled by:

– Project teams (e.g., WTP all-in-one)�

– Companies (e.g., Innoopract Yoxos)�

– IT departments (e.g., Fidelity)�

– Planning Council (e.g., Callisto, Europa, Ganymede, …)�

• And by you


2. GANYMEDE & GANYMATIC


WHAT IS GANYMEDE?

• A simultaneous release

– Includes "tightly bound" & "loosely bound" projects

– Differing levels of guideline conformance, testing, packaging

– Defined by the Eclipse Management Organization

• An update site for that release

• A distro (or product) assembled from that update site


GANYMEDE VS. EUROPA

The next moon of Jupiter:

• Callisto: a release date

– Can we do a simultaneous release in spite of the cross dependencies?

• Europa: a process for one/static distro

– Can we automate the production of a multi plug-in distro?

• Ganymede: process & infrastructure for many/dynamic distros

– Extend that automation to the larger Eclipse community


GANYMATIC & GANYMEDE

Ganymatic is the infrastructure

• Ganymatic = Ganymede "releng"

– Current: assembles update sites & distros, manages dependencies

– Soon: runs tests, verifies community rules

– Future: measures performance, generates New & Noteworthy,

source bundles, etc.

• Built on Buckminster

– Dependency management

– Site assembly

– Runs on build server or on a local developer machine


GANYMATIC & ECOSYSTEM

• Opened to ecosystem through Project Dash (Tools for Committers) �

• “Tightly bound” project teams use Ganymatic to verify contribution

• “Loosely bound” project teams use Ganymatic to contribute

• Other project teams use Ganymatic to produce all-in-one distros

• Companies use Ganymatic to produce product distros

• User community uses Ganymatic to produce custom distros– “The Bjorn distro”

– “The Thomas distro”

– “The Wascana (CDT on Windows) distro”

– Your distro here


3. HOW GANYMATIC WORKS


RESULTS ON THE WEB

http://build.eclipse.org/ganymede/


RESULTS AND NOTIFICATIONS

• Ganymede Ganymatic runs on a regular schedule

• Failure notifications by email to contributing releng

• Summaries (successes, failures, all) by RSS


RULES AND CONFORMITY

• Pluggable rule checking

• Ganymede projects must be signed and pack200 compressed

• Ganymede projects must list their required JVM version

• Ganymede projects must use Orbit for third-party plug-ins

• Ganymede projects should use ICU4J internationalization

• Etc.


FUNCTIONAL OVERVIEW


SITE CONTRIBUTION FILE


BUCKMINSTER AS FOUNDATION

• Key is assembly & dependency management

– Ganymede is a “virtual distro”

– Managing dependencies significant effort

• Ganymatic relies on Buckminster to simplify

– Consumes a contribution description

– Uses Buckminster to assemble as virtual distro

• Also key is minimizing specifications

– Materializes binary virtual distros

– Materializes source code “debug & test” distros


BUCKMINSTER HIGHLIGHTS

• Resolution & materialization framework

– Manages and resolves dependencies

– Materializes components in context of choice

– Can build workspaces defined as “virtual distros”

– Supports CVS, SVN, update sites, other repositories/libraries

• Model agnostic meta-framework

– Spans repos, component & build types

– Supports complex assembly patterns

• Catalyst for distro creation

– Community input in dependency definition

– Simplified consumption of cross-project dependencies


TESTING - THEN WHAT?

• Site distros are assembled binaries

• Assembly does not detect API breakage

• A good distro includes tests

• Tests will detect a malfunction, then what?

• Finding bugs is difficult without source

• Building and maintaining a suitable workspace is time consuming


USING BUCKMINSTER/GANYMATIC FOR SOURCE

• Ganymatic has a unified way of describing dependencies

• Dependencies can be used to extract binaries (normal virtual distro) and/or source (workspace for debugging)�

• Buckminster can populate workspace with source/binary mix

– All source

– Source of your components, binaries of the rest

– Source of key components, binaries of the rest

– Etc.


4. USING GANYMATIC


VERY VERY EASY

• Get the site assembler from the Dash download site

• Get the Ganymatic code from the Dash repository

• Run As… > Ant Build…

• (wait a few minutes)�

• Examine results

• Release Engineering should be dull, boring, and repeatable

• Ganymatic is that: easy to configure, dull to demo :-)�


DOING MORE

• Build your own distro

• Modify the site contribution files (edit, add, remove, ?)�

• Contributions from other libraries of plug-ins

– Non-Ganymede Eclipse projects

– EPIC (epic.eclipse.org)�

– Your internal company tools

– Sourceforge plug-in projects

– Maven repositories

– Web sites specializing in providing components

• Run your local Ganymatic instance


DEBUGGING USING GANYMATIC

• (coming soon)�

• Ganymatic generates a Buckminster cspec:e.g., ganymede.cspec

• Using Buckminster, “Resolve and Materialize” that cspec

• Debug at will…


5. GANYMATIC ROADMAP


SCOPE & VISION

• Anyone can make a distro

– You, me, the Eclipse Foundation, Eclipse members, the ecosystem, …

• Anyone can make a distro

– Define

– Assemble

– Test

– Debug

– Develop

– Consistently…

• Expanding the Choice that Open Source provides

THANK YOU

visit us at:

www.eclipse.org/buckminster

www.eclipse.org/dash


BUCKMINSTER BACKGROUND SLIDES


Buckminster introduction

• Explain Buckminster Concepts

• Example – workspace provisioning

• Example – virtual distribution


WHAT IS BUCKMINSTER?

• Eclipse feature-set for specifying & materializing

component configurations

• Automates materialization & assembly of components

from disparate sources

• Recognizes metadata semantics of various component

types

• Supports either “headless” or GUI-driven execution

• Buckminster 0.2 released for Eclipse 3.3.x

• Included in Europa and (will be in) Ganymede


AT A GLANCE

Buckminster:

• materializes components to an Eclipse Workspace

• produces new artifacts inside or outside of the workspace (i.e ‘build’)�

Action: “build”


TRANSITIVE MATERIALIZATION

• When working on A components B and C are also needed, and they need D

• Buckminster materializes all pre-requisite components

• Can take action into account –dependency followed only when needed.


WHERE DOES BUCKMINSTER GET COMPONENTS?

• Components are loaded from repositories:

– CVS

– Subversion

– Maven

– URL (http, ftp, etc.)�

– Eclipse update site

• Extension point


HOW DOES BUCKMINSTER FIND REPOSITORIES?

• Buckminster has a Resource Map (RMAP)�

– Search path for components

and families of components

– Different paths for different

aspects of components


COMPONENT – WHAT IS IT?

• A named entity found in a repository– File(s) in file system, directory, an Eclipse project, a URL etc.– Can be “empty”; i.e. a configuration

• Has component meta data– Buckminster uses existing meta-data from Eclipse plug-ins and Maven

– Buckminster Component Specification (CSPEC) can also be used– If CSPEC is missing, it is generated on the fly

• Component Content Agnostic– Java, C, PHP, a web site, etc. etc.


GENERATED CSPEC

• No roundtrip engineering.

• No need to maintain same meta data in several places.

• Extension of generated component meta data is possible.


DEPENDENCIES AND VERSIONS

• Describes; if a component is needed which version (or range of versions) is acceptable

• Versions are mapped to version type used by repository


ATTRIBUTES & ACTIONS

• Attributes represents artifact groups

• Can have pre-requisites on other attributes

– Same or other component

• Attributes are static or computed (actions)�


ACTIONS

• Action attributes produce artifacts

– Eclipse builds

– ANT tasks

• Extension point

– Maven?

– Your own


Workspace provisioning

Getting started can be somewhat time consuming...

• What projects do I need and where do I find them?

• What workspace settings should I use?

• Is the target platform correctly configured?

• What other dependencies are there?

• ...

Where is all of this described?


And as the project matures...

• Developers often want the latest source of some branch

• Release management and QA engineers require fixed labels

• Using pre-built artifacts to reduce build-time becomes a requirement

• Source found in CVS and SVN, Features and plugins at update sites, jars at maven, other binaries at ...

• Keeping track of the exact bill of materials becomes crucial dueto licensing

And everything is evolving over time!


The Buckminster solution, CSPEC

A type and language agnostic Component Specification:

•Dependencies that have flexible constraints

•Actions triggered by events and/or manually

•Often generated from known meta-data (feature.xml, Maven POM, etc.)�


The Buckminster solution, CQUERY

A Component Query controlling aspects of the transitive resolution such as:

•scope of resolution (target platform,remote,...)�

•property based variability

•version ambiguity resolution

•skip of unneeded components


Component resolution phase I. Creating the BOM

• Component meta-data is loaded

from repositories:– CVS

– Subversion

– Maven

– URL (http, ftp, etc.)�

– Eclipse update site

• Transformed into CSPEC

• Extension points– Component Type

– Reader Type

– Version Type


The Bill of Materials (BOM) ��

• A BOM is the result of a transitive component resolution

• It's a description of exactly what to download and where to download it from

• All artifacts appointed using exact versions

• Reusable


Phase II, the materialization

• Uses the BOM created in Phase I as input.

• Downloads components to local disk

• Performs assembly actions such as bundle generation and relocation of embedded jar files.

• Binds projects to Eclipse workspace.


WHAT IS A “VIRTUAL DISTRO”?

Open source products “assembled” from disparate sources

• Typical Problems– Download & installation inefficiencies

– Conflicts between tooling, run-times

– Supporting consumer variability

– Licensing constraints

• Buckminster designed to support virtual distros– Mechanisms to specify product as configuration of components

– Mechanisms to publish product configurations

– Mechanisms to materialize configurations against component sources

• Applicability of virtual distro model– Open source projects

– Distro vendors

– Internal development


SOLUTION OVERVIEW

Definition ≡ Product = CSPECs + CQUERYs = BOMs

Sourcing ≡ Appoint locational dependencies

Bypass licensing issues

Publication ≡ "One-click" materialization link

Java Web Start

Consumption ≡ Headless to deployment target

Directly into Eclipse workspace

User-defined materialization variations


LombozCSPEC

LombozCQUERY

A

B

X1.0

2.0

Points to Lomboz component

Can override the conflict – 2.0 of X is

known to work for both A and B

LombozBOM

reads

produces

BuckminsterResolver

Download

Web page with Java

Web Start

Script referencing the BOM

URL to BOM

PUBLISHER’S SIDE


A

B

X

LombozBOM

Download

Gets BOM

Runs Materializatioin

CONSUMER’S SIDE

JREBM

Java Web Start

Fetched(if not present)�

BM materializer

invokes

L

Downloads to eclipse workspace

or freestanding.

Documents

ESE2007 Buckminster Gany-matic - EclipseCon … Summit 2007 21 BUCKMINSTER HIGHLIGHTS • Resolution & materialization framework – Manages and resolves dependencies – Materializes