Apache Java Cloud Projects!

Andrew BayerBay Area Apache CloudStack Users GroupAugust 7, 2013

Who am I?

Andrew Bayer

Build and Tools Architect at Cloudera

User of CloudStack

VP, Apache Whirr

PPMC, Apache jclouds

Bad at designing slide decks

What am I here to talk about?

How the Apache Java-based cloud projects play together!

CloudStack(of course!)jcloudsWhirr

Hold on a sec, gotta start the demo!

(it takes a while, y'see)

What's CloudStack?

...shouldn't you know that already?

Well, ok.

The project develops open source software for deploying public and private Infrastructure-as-a-Service (IaaS) clouds.

What can you do with CloudStack?

Run a cloud?

Seriously, what can't you do with CloudStack?

What's jclouds?

Java library for abstracting and interacting more directly with a plethora of compute, blobstore (i.e., S3 or Swift) and other cloud-related APIs and providers.

High-level abstractions allow you to use the same code to work with different APIs/providers.

Java interfaces directly to the various APIs let you drill down and get more specific control.

What can you do with jclouds?

Cloud automation in your Java/JVM application

Provision instancesUse object storesControl your cloudWrite a management layer on top of one or more clouds (hey, I did that!)

What's Whirr?

A set of Java libraries (built on top of jclouds) for provisioning and deploying distributed systems.

Cloud-neutral, or bring your own node

Common service API with service-specific details for configuring/provisioning

Smart defaults so you can get going easily, but easy to override

What can you do with Whirr?

Set up an Apache Hadoop cluster on your favorite cloud in 5 minutes of work.Deploy a distributed application onto existing physical hosts (yes, Whirr works with bare metal!)Write your own service to deploy your own appIntegrate all the above into a Java/JVM app

All at Apache!

CloudStack and jclouds

jclouds has support for the full CloudStack API

User, Domain and Root APIs(if something's missing, open a JIRA and we'll add it!)

jclouds compute abstraction allows you to use the same code with multiple clouds

Or you can drill down to the full CloudStack-specific API and do, well, anything!

CloudStack and jclouds

CloudStack is a first class API in jclouds

Right up there with EC2, OpenStack, and (soon) Google Compute

Provisioning, security groups, image creation one abstraction works with all these clouds

jclouds: Testing, Testing, Testing

Full battery of API tests

Unit tests to verify correctness of API calls

Live tests exercise every API call against an actual CloudStack setup

Wait, All the APIs?

You mean the Root Admin APIs too?

Creating/deleting zones, acounts, etc?

Yup.

But isn't that risky to run against a live CloudStack setup?

Ah, I've got an answer for that!

So how do you do that safely?

The CloudStack simulator, of course!

Build CloudStack, run the simulator(and the marvin scripts to configure it, etc)

Point jclouds' tests at the simulator

Test everything! Break nothing!

A bit more detail...

You do need to grab the root admin API creds, and create domain admin/normal users as well

Not all tests will actually work anything that expects to SSH into a VM or attach a disk, etc

We'll be marking simulator-friendly tests so you can run mvn clean install -Plive -Pcloudstack-simulator and just run those tests

So what's using jclouds and CloudStack now?

A good number of tools, like...

- The Jenkins jclouds plugin(for provisioning Jenkins slaves on the fly)- Pallet(config management like Chef/Puppet, in Clojure)- CloudCat(Cloudera's internal cloud management app)...and of course, Whirr!

Is Whirr like Chef or Puppet too?

Short answer? No.

Chef/Puppet/Pallet/etc are great at getting nodes to a target state, initially and/or regularly

But they're not good at dealing with a set of distributed apps that need to coordinate with each other while getting set up

Whirr, on the other hand, is.

Deployment Orchestration Engine!

That's the term I use to describe Whirr

Coordinating installation and configuration, across hosts/instances, in pre-defined stages

What services does Whirr support?

Among others:

HadoopHBaseZooKeeperCassandraChefPuppetSolrKerberos

and more...

My personal favorite: whirr-cm

(I'm admittedly a bit biased)That's Whirr support for Cloudera Manager

Deploys a fully configured CDH (Cloudera's Distribution for Apache Hadoop) cluster, including Cloudera Impala and Search, using Cloudera Manager

All in one command!

Cloudera Manager

I should probably mention a bit more about CM.

It manages, reports, configures, monitors, deploys the complete CDH Hadoop stack, with lots of handy features and integrations, all exposed via a great web UI and a REST API (which has publicly available Java and Python libraries as well).

whirr-cm works everywhere

...well, nearly everywhere.

Since it's jclouds-powered, it works with any cloud supported by jclouds.

But it does require certain things like reverse-resolvable public IPs, as do some other Whirr services.

Running whirr

Set up a config file for your cluster see the recipes dir for examples

For CloudStack, you have to specify jclouds.endpoint=http://your.server/client/api

Run bin/whirr launch-cluster config path/to/your.properties to provision/deploy

Run bin/whirr destroy-cluster config path/to/your.properties to destroy

Oh yeah, that demo...

It's whirr-cm!

Let's see how it's doing...

Another caveat

Whirr 0.8.2 (the latest release) has some problems with CloudStack

Namely, security groups/firewalls and keypairs

These will be fixed in 0.9.0, coming out this fall

Patch already available - WHIRR-725

How do I get whirr-cm?

First option: download or build Whirr, and then download or build whirr-cm see https://github.com/cloudera/whirr-cm

Second option: install Whirr from the CDH deb/rpm packages CDH 4.3.0 uses whirr-cm 1.2.

Links!

CloudStack - http://cloudstack.apache.org/

jclouds - http://jclouds.incubator.apache.org/

Whirr - http://whirr.apache.org/

whirr-cm - https://github.com/cloudera/whirr-cm

Installing Whirr from CDH

CloudCat - https://github.com/abayer/cloudcat/ (for CloudStack-specific API use with jclouds)

Thanks for listening!