Embed Size (px)
Citation preview
connecting java (and clojure) to the cloud
[email protected]@jcloudshttp://blog.jclouds.org
1Monday, July 12, 2010
Agenda• Motivation
• jclouds
• BlobStores
• Provisioning
2Monday, July 12, 2010
Motivation• Cloud Storage
• Vendors
• Differences
3Monday, July 12, 2010
what is key value storage?
global name space
key, value with metadata
http accessible
sites on demand
unlimited scaling
4Monday, July 12, 2010
key value storage concepts
GLOBAL NAME SPACE
KEY/ VALUE
5Monday, July 12, 2010
THE VENDORS
6Monday, July 12, 2010
They aren’t the same
7Monday, July 12, 2010
• FILE SIZE
• RESUMABILITY
8Monday, July 12, 2010
CONTENT DELIVERY NETWORK
REPLICATION
SLA
9Monday, July 12, 2010
Consistency Model
10Monday, July 12, 2010
AUTHORIZATION
11Monday, July 12, 2010
api complexity
12Monday, July 12, 2010
CODE AND SIGN THE HTTP REQUEST
PUT /sushi.avi HTTP/1.1 Host: adriansmovies.s3.amazonaws.comContent-Length: 734859264Date: Wed, 01 Mar 2006 12:00:00 GMTAuthorization: signature x-amz-meta-Chef: Kawasaki
PUT /<api version>/<account>/adriansmovies/sushi.avi HTTP/1.1 Host: storage.clouddrive.com Transfer-Encoding: chunkedX-Auth-Token: session-token X-Object-Meta-Chef: Kawasaki
PUT /adriansmovies/sushi.avi HTTP/1.1 Host: <account>.blob.core.windows.netContent-Length: 734859264Date: Wed, 01 Mar 2006 12:00:00 GMTAuthorization: SharedKey <app>:signaturex-ms-meta-Chef: Kawasaki
POST /namespace/adriansmovies/sushi.avi HTTP/1.1 Content-Length: 734859264Date: Wed, 01 Mar 2006 12:00:00 GMTx-emc-uid: <uid> x-emc-signature: signature x-emc-meta: Chef=Kawasaki
13Monday, July 12, 2010
GET /ws/IMFS/GetStorageNodeExtended.ashx?&fileOverwrite=true&ipRestricted=true&destFolderPath= adriansmovies&sizeBytes= 734859264&firstByteExpiration=6000&lastByteExpiration=259200&sessionToken=session-token HTTP/1.1
POST /Upload.ashx?uploadToken=from_above&destFolderPath=adriansmovies HTTP/1.1Host: from_above Content-Length: 734859382Content-Type=multipart/form-data; boundary=--jclouds--Authorization=Basic GpjbG9=----jclouds--Content-Disposition: form-data; name="sushi.avi"; filename="sushi.avi"Content-Type: application/octetstring...
PUT /ws/Metadata/SetMetadata.ashx?&path=Folders/adriansmovies/sushi.avi&sessionToken=session-token&metadata=Chef:Kawasaki HTTP/1.1
CODE AND SIGN THE HTTP REQUEST
14Monday, July 12, 2010
POST /<api version>/containers/id_of_ adriansmovies/contents HTTP/1.1 Content-Length: 734859382Content-Type=multipart/form-data; boundary=--jclouds--Authorization=Basic GpjbG9=----jclouds--Content-Disposition: form-data; name="sushi.avi"; filename="sushi.avi"Content-Type: application/octetstring...
PUT /<api version>/files/from_above/metadata/Chef HTTP/1.1 Content-Length: 8Content-Type: text/plainAuthorization: Basic GpjbG9=Kawasaki
CODE AND SIGN THE HTTP REQUEST
15Monday, July 12, 2010
do you want to
• Deal with Errors
• Deal with Concurrency
• Deal with Cloud Complexity
16Monday, July 12, 2010
jclouds
open source
feels like java (and clojure)
portability between clouds
deal with web complexity
unit testability
thread-safe and scalable
17Monday, July 12, 2010
Tools we provide
• Abstractions
• BlobStore ( atmos, azure, rackspace, s3 )
• Compute ( vcloud, ec2, gogrid, ibmdev, rackspace, rimu )
• Clojure bindings
• Third-party library integration
18Monday, July 12, 2010
Alternatives to jclouds
• Roll-your-own
• Jersey, RESTEasy
• EC2-based cloud apis
• typica, jets3t
• Dasein Cloud API
• Service provided SDKs
19Monday, July 12, 2010
BlobStore• Java Code
• Clojure Code
20Monday, July 12, 2010
java
// init context = new BlobStoreContextFactory().createContext( "s3", accesskeyid, secretaccesskey); blobStore = context.getBlobStore();
// create container blobStore.createContainerInLocation(null, "mycontainer");
// add blob blob = blobStore.newBlob("test"); blob.setPayload("testdata"); blobStore.putBlob(containerName, blob);
21Monday, July 12, 2010
commons vfs
vfs > open blobstore://user:key@cloudfiles/mycontainerOpened blobstore://cloudfiles/mycontainer/Current folder is blobstore://cloudfiles/mycontainer/vfs > lsContents of blobstore://cloudfiles/mycontainer/README.txt0 Folder(s), 1 File(s)vfs > close
22Monday, July 12, 2010
clojure
(ns demo (:use org.jclouds.blobstore) (:use clojure.contrib.pprint))
(def blobstore (blobstore-context service account secret ))
(pprint (containers blobstore))
(pprint (blobs blobstore "mycontainer" ))
23Monday, July 12, 2010
Provisioning• The Good, the Bad, and the Ugly
• Java Code
• Clojure Code
24Monday, July 12, 2010
The Good
provisioning (and re-provisioning) is cheap
APIs = automation
tools exist
25Monday, July 12, 2010
The Bad
forgetting to turn things off
licensing
erratic service quality
26Monday, July 12, 2010
The Ugly
cloud apis are sometimes unreliable
apis are very different across clouds
features are very different across clouds
accidental complexity
27Monday, July 12, 2010
Things to consider when provisioning
Can you create an image?
Can you push credentials or files?
Do you need to VPN in?
How is storage provisioned?
How close are your dependencies?
28Monday, July 12, 2010
Java Code
29Monday, July 12, 2010
jclouds github jclouds/jclouds
service = new ComputeServiceContextFactory().createContext( “rimuhosting”, user, password ).getComputeService();
template = service.templateBuilder().any().biggest().build();
template.getOptions().installPrivateKey(privateRSA) .authorizePublicKey(publicRSA) .runScript(installGemsAndRunChef);
nodes = service.runNodesWithTag(“webserver”, 5, template);
30Monday, July 12, 2010
dasein sourceforge dasein-cloud
CloudProvider provider = providerClass.newInstance();
ProviderContext context = new ProviderContext();
context.setAccountNumber(accountNumber);context.setAccessPublic(apiKeyBytes);context.setAccessPrivate(privateKeyBytes);
provider.connect(context);
ServerServices services = provider.getServerServices();
server = services.launch(imageId, size, dataCenterId, serverName, keypairOrPassword, vlan, analytics, firewalls);
31Monday, July 12, 2010
whirr github tomwhite/whirr
ServiceSpec serviceSpec = new ServiceSpec();
serviceSpec.setProvider("gogrid");serviceSpec.setAccount(account);serviceSpec.setKey(key);serviceSpec.setSecretKeyFile(secretKeyFile);serviceSpec.setClusterName(clusterName);
service = new HadoopService(serviceSpec);
ClusterSpec clusterSpec = new ClusterSpec( new InstanceTemplate(1, HadoopService.MASTER_ROLE), new InstanceTemplate(1, HadoopService.WORKER_ROLE));
cluster = service.launchCluster(clusterSpec);proxy = new HadoopProxy(serviceSpec, cluster);proxy.start();
32Monday, July 12, 2010
jclouds-chef github jclouds/jclouds
context = ChefContextFactory.createContext(server, identity, key);
rsaPrivateKey = context.getApi().createClient(clientName);
cookbooks = context.getApi().listCookbooks();
ChefAsyncClient nonBlockingClient = context.getAsyncApi();
nonBlockingClient.uploadCookbook(“apache2”, new File(“/cookbooks/apache2.tgz”));
33Monday, July 12, 2010
(code clojure)
34Monday, July 12, 2010
jclouds github jclouds/jclouds
(def service (compute-service “ec2” account key :ssh :log4j))
(with-compute-service [service] (def template (build-template :run-script bootstrap)) (def node (run-node "couchdb" template)) (create-volume :node node :size 250))
35Monday, July 12, 2010
crane github clj-sys/crane
(def hadoop-config (conf "/path/to/conf.clj"))
(def compute (ec2 (creds "/path/to/creds.clj")))
(launch-hadoop-cluster compute hadoop-config)
36Monday, July 12, 2010
pallet github hugoduncan/pallet
(defnode webserver [] :bootstrap [(public-dns-if-no-nameserver) (automated-admin-user)] :configure [(chef)])
(with-compute-service [service] (converge {webserver 3}) (cook webserver "/cookbooks/apache-chef-demo"))
37Monday, July 12, 2010
