Continuous Integrationwith Hudson

Arun GuptaSun Microsystems, Inc.

Originally presented byKohsuke Kawaguchi & Jesse Glick

at JavaOne

Scheduled to be presented byFabiane Nardon

At FISL 10

Acknowledgements

Rise of Continuous Integration> Offload from people, push to computers

3

$

time

computers

us

“Never use humans for the job of a machine”

Spend more CPU power to help you> … even if it only helps a little

> First on your laptops and workstations IDEs are at the forefront

> And then to the servers a.k.a. “Continuous Integration” More frequent build/test executions Static code analysis tools And more to come

5

Hudson> Open-source CI server at java.net

> Emphasis on ease of installation and use “java -jar hudson.war” execution Configure everything from browsers

> Extensibility 140+ community-developed public plugins By 150+ contributors

> Estimated 13,000 installations6

https://hudson.dev.java.net/

It basically does builds and tests> Check out the source code

Subversion, Perforce, Git, Mercurial, CVS, …> Do builds and/or tests

Ant, Maven, MSBuild, shell script, …> Record results

Binary, test results, code coverage, static analysis> Notify people

E-mail, IM, RSS, tray apps, IDEs

7

Hudson Plugin Ecosystem

8

New Plugins

2008: 552009: 44 so far

Localized to 8 languages

9

And hopefully more to come…

10

11

Adoption in all kinds of businesses

Eclipse Community Survey

12

13

Why Distributed Builds?> You need to use multiple computers because…

You need different environments You need isolation

> There’s only so much you can do with 1 computer

14

Installing new slaves> For first 20 or so slaves, we did it manually

Insert CD, click, type, click, type, click, … But that doesn’t scale

> Then we automated Available as “Hudson PXE Plugin”

15

Automated System Installations

16

> Slaves Power on, hit F12 PC boots from network (PXE)

> Hudson + PXE plugin ISO images of OS

Automated System Installations

17

> Slaves Power on, hit F12 PC boots from network (PXE) Choose OS from menu Installs non-interactively

> Hudson + PXE plugin ISO images of OS

Your corporate IT guy & his DHCP server

Automated System Installations> Supports OpenSolaris, Ubuntu, CentOS, Fedora

Trivial with most Linux> Cooperate with Windows, too

> Quite useful outside Hudson, too No more broken CD drives No more CD-Rs

18

Distributed builds with Hudson> Master

Serves HTTP requests Stores all important info

> Slaves 170KB single JAR Assumed to be unreliable Scale to at least 100

> Link Single bi-di byte stream No other requirements

19

Heterogeneous Cluster Challenge> Your builds/tests need to run in specific

environment> Dependency on individual nodes hurts utilization

20

WombatWindows test

Hudson Windows test

Windows #1

jobs slaves

GlassFishWindows test

Windows #2

Solaris#1

Hudson Solaris test

Labels to rescue> Label is a group of slaves> Tie jobs to labels

21

WombatWindows test

Hudson Windows test

Windows #1

jobs slaves

GlassFishWindows test

Windows #2

Solaris#1

Hudson Solaris test

Windows

SolarisWindow

s #3

Making builds sticky> We want jobs to be mostly on the same slave

Faster check out Consistent results Minimizes disk consumption

> But does it softly

> Hudson uses consistent hash* to achieve this> More schedule controls become possible:

Use faster machines more frequently Slowly ramp up newly installed slaves

22

Coming to your

Hudson soon

* http://en.wikipedia.org/wiki/Consistent_hashing

Forecasting failures> Hudson monitors key health metrics of slaves

Low disk space, insufficient swap Clock out of synch Extensible

> Slaves go offline automatically> Catch problems before they break builds

23

Load Statistics Monitoring

24

When it’s time to add more slaves

25

Hudson made this extensible> Hudson detects excessive workload> Hudson notifies plugins> Plugins can provision more slaves

… assuming that you have that infrastructure

26

27

Amazon EC2: The Good> Pay as you go (10¢/h or so)

Loads on Hudson tend to be spiky> Programmable API> Instances launch at machine-speed> EC2 instances are forgetful

28

Amazon EC2: The Bad> Your data is still inside your firewall

Takes time to check out code … or to archive build artifacts Some data just can’t be moved

> EC2 instances are forgetful> Can your tests run in parallel?

29

Hudson EC2 plugin> Built on top of typica*> What does it do?

Automatically provisions slaves on EC2 on demand Picks the right AMI depending on demand Starts slave agent Shuts down unused instances

30* http://code.google.com/p/typica/

Hudson “Appliance” on EC2> Run the master in the cloud too, if you like

Hudson on stock OpenSolaris AMI Data stored persistently in Elastic Block Storage

Dynamically expandable thanks to ZFS Online, too

> Packaged as a wizard

31

Hudson EC2 plugin usage> Tell Hudson what AMIs you want to start

32

33

Conclusion> CI is here to stay

We’ll continue to push more workload to servers> Hudson makes this easy for you> Reap the benefit of a cluster in multiple ways

34

Resources> http://hudson.dev.java.net/

> Support Subscription hudson@sun.com

35

36

Arun Gupta

http://blogs.sun.com/arunguptahttp://hudson.dev.java.net/