HudsonSF JUG

Kohsuke KawaguchiSun Microsystems, Inc.

2

Rise of Continuous Integration

Offload from people, push to computers$

time

computers

people

Spend more CPU power to help you

First on your laptops and workstations IDEs are at the forefront

And then to the servers a.k.a. “Continuous Integration” More frequent build/test executions Static code analysis tools And more to come

3

Hudson

Open-source CI server Emphasis on ease of installation and use

“java -jar hudson.war” execution Or your OS-specific package Configure everything from browsers

Extensibility 230+ community-developed public plugins By 230+ contributors

Estimated 13,000 installations4

https://hudson-ci.org/

Adoption in all kinds of businesses

5

6

Support Subscriptions

Community releases every week Over 340 releases to date Rapid new features & bug fixes delivery

“Sun Continuous Integration Server” For enterprise that needs stability Bug fix only patch releases for 18

months

Basic Feature Set

Hudson notices that there is a change Check out the source code

CVS, Subversion, ClearCase, Mercurial, Accurev, Perforce, Git, StarTeam, …

Do builds Ant, Maven, shell script, NAnt, …

Record and publish results Close the feedback loop

RSS feeds E-mail/IM notification Tray application to notify developers

Tool integration Findbugs, cobertura, …

7

Demo

9

Why do I care?

Life before Hudson Dev forgets to commit a file RE nightly build breaks. It blocks SQE People waste time

Life after Hudson Hudson will tell you in 5 mins if a build

broke No one else notices that you forgot to

commit a file

10

Why do I care?

Life before Hudson Dev makes a change Monday AM SQE nightly finds a bug Monday night Dev fixes it Tuesday Lot of wasted time

Life after Hudson Dev makes a change Monday AM Hudson finds a regression 30 mins later Dev can fix it before lunch

11

Why do I care?

Life before Hudson SQE test runs every night, results sent out

in e-mail After the enthusiasm of the 1st week is

gone, nobody looks at them anymore Regressions go unnoticed until it’s too late

Life after Hudson Tests run Hudson after every commit E-mail sent out only when tests start failing▪ So it manages to keep people’s attention

Matrix projects

Run the same thing on different environments such as different

JDKs, different databases, and different OSes

Aggregate the results

12

FindBugs integration

13

14

Tracking changes

… and more

BROWSE WORKSPACES

BUILD TIME TREND REPORT

15

16

Inter-Project Support

You can host many projects on 1 Hudson And when you do, 1+1 becomes more

than 2 You can…

Execute tests separately and correlate them with builds

Mark builds as “promoted” based on test outcome

Track which build of libraries are used where

And more

Test/Build Separation

Tests often dominate the build time Multiple tests are run in parallel

Improves turn-around time

JAX-WS #10

Unit test with GF

Unit test with Tomcat

SQE test

JAX-WS #11 JAX-WS #12 …

Test w/o container

3OS x 2JDK = 6 tests

time

17

Test/Build Separation Technique

Copy bits from the build to test via wget by permalinks Ant has the equivalent <get> task

“Fingerprinting” to keep track of where binaries are used

$ wget –O foo.zip http://server/hudson/job/foo/lastSuccessfulBuild/ artifact/foo.zip

18

19

Build Promotion

CI produces a lot of “successful” builds Often overwhelming to downstream

consumers

Run tests as fast as you can If a build passes tests, promote it

20

Got the idea?

Automation Reduce turn-around time Make things transparent Remove people from the loop

Save people’s time Push jobs to servers, keep

workstations idle for you

Doing Distributed Builds

21

Why?

There’s only so much 1 computer can do

But lots of PCs = tedious What does Hudson do to help you?

22

Installing new slaves

For first 20 or so slaves, we did it manually Insert CD, click, type, click, type, click, … But that doesn’t scale

Then we automated Available as “Hudson PXE Plugin”

23

Automated System Installations

Slaves Power on, hit F12 PC boots from network

(PXE)

> Hudson + PXE plugin ISO images of OS

24

Automated System Installations

Slaves Power on, hit F12 PC boots from network

(PXE) Choose OS from menu Installs non-interactively

> Hudson + PXE plugin ISO images of OS

Your corporate IT guy & his DHCP

server

25

Distributed builds with Hudson

Master Serves HTTP requests Stores all important info

Slaves 170KB single JAR Assumed to be unreliable Scale to at least 100

Link Single bi-di byte stream No other requirements

Master

slave1

slave2

slave3

slave4

slave5

slave6

slave7

slave8

26

How master and slaves start talking

For Unix slaves, via SSH Only need SSHD and JRE on slaves We just need a host name

27

How master and slaves start talking

For Windows, DCOM We just need admin user name and

password No manual intervention Works even from Unix masters

28

How master and slaves start talking

Via Java Web Start When master cannot see slaves A separate socket connection is made

29

Automating JNLP launch

Once started, can be installed as Windows service

30

Automating JNLP launch

Emulate the JNLP client headless

$ java -jar slave.jar -jnlpUrl URL

31

Automated Tool Installation - JDK

JDK from http://java.sun.com/ Hudson automatically chooses the right

bundle Always up to date with new releases

32

Automated Tool Installation - Apache

Ant and Maven from Apache

33

Heterogeneous Cluster Challenge

Your builds/tests need to run in specific environment

Dependency on individual nodes hurts utilization

WombatWindows

test

Hudson Windows

test

Windows #1

jobs slaves

GlassFishWindows

test

Windows #2

Solaris#1

Hudson Solaris test

34

Labels to rescue

Label is a group of slaves Tie jobs to labels

WombatWindows

test

Hudson Windows

test

Windows #1

jobs slaves

GlassFishWindows

test

Windows #2

Solaris#1

Hudson Solaris test

Windows

Solaris

Windows #3

35

36

Concurrent Builds

time

Hudson #10

Hudson #11

Hudson #12

Hudson #13

Hudson #14

Hudson #15

Forecasting failures

Hudson monitors key health metrics of slaves Low disk space, insufficient swap Clock out of sync Extensible

Slaves go offline automatically

38

Clean up mess after builds

Kill run-away processes Daemons, background processes left by

your build Works on Windows, Linux, Mac, and

Solaris

39

Load Statistics Monitoring

40

When it’s time to add more slaves

41

Hudson EC2 plugin

Automatically provisions slaves on EC2 on demand Based on the current load

Picks the right AMI depending on demand

Starts slave agent Shuts down unused instances

42

Putting it all together

time

# o

f exe

cuto

rs

capacity

usagequeue length

43

Hudson “Appliance” on EC2

Run the master in the cloud too, if you like Hudson on stock OpenSolaris AMI Data stored persistently in Elastic Block

Storage▪ Dynamically expandable thanks to ZFS▪ Online, too

Packaged as a wizard

44

And a lot more I couldn’t show

IDE plugins iPhone/Android apps REST API and CLI Deployment Automation Hadoop, Selenium …

45

Conclusions

CI is here to stay Using lots of PCs is a key

Hudson is very easy to get started Once started, there are a lot you can do

46

Q&A

Kohsuke KawaguchiSun Microsystems, Inc.

Compartmentalization

Faster feedback Good engineering practice anyway Isolation of problems

full Metro build #10 full Metro build #11

time

JAXB #10 JAXB #11 JAXB #12

JAX-WS #20 JAX-WS #21 JAX-WS #22

48

Automated System Installations

Supports OpenSolaris, Ubuntu, CentOS, Fedora Trivial with most Linux

Cooperate with Windows, too

Quite useful outside Hudson, too No more broken CD drives No more CD-Rs

49