64
1 Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Embed Size (px)

Citation preview

Page 1: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

1

Taming the Cloud Databasewith

Apache jcloudshttp://rack.to/ace14db

Page 2: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Before we begin: Setup

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

2

� Virtual Machine pre-setup provided� http://rack.to/ace14vm� Avoid downloading during this presentation

Page 3: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Before we begin: Setup

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

3

� Alternatively you need� Oracle Java 7

� http://www.webupd8.org/2012/01/install-oracle-java-jdk-7-in-ubuntu-via.html

� maven� http://maven.apache.org/download.cgi

� git config --global core.autocrlf false

Page 4: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Before we begin: Setup

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

4

� Testing on your machine (no subscriptions)� Devstack

� http://devstack.org/� http://blog.phymata.com/2014/04/18/devstack-icehouse-on-the-

rackspace-cloud/

� Providers: Rackspace, HP, others� http://www.openstack.org/marketplace/public-clouds/

� http://rack.to/ace14

Page 5: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Before we begin: Setup

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

5

git clone https://github.com/zack-shoylev/jclouds-developer-examples.git

Multiple branches

git clone https://github.com/rackerlabs/jclouds-examples.git

Branch trove-example-only

Page 6: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Introductions

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

6

➢ Zack Shoylev➢ irc: zacksh twitter: @zackshoylev

freenode #jcloudsAlso email

@rackspace.com

Page 7: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Introductions

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

7

Developer Experiencehttps://developer.rackspace.com/support/

Page 8: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Goals

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

8

➢ jclouds➢ Create a database in the cloud➢ Abstractions and how to contribute➢ How to add support for an API

Page 9: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

The Cloud

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

9

➢ Networked and distributed computing➢ Variety of services

➢ Compute (servers)➢ Storage (files)➢ Databases➢ Email➢ …

Page 10: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

The Cloud

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

10

➢ Can’t someone else do it?

Page 11: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

The Cloud

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

11

Page 12: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

The Cloud

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

12

➢ Public cloud (external provider)➢ Private cloud (internal deployment)➢ Hybrid cloud

➢ Public + Private

Rackspace example

Page 13: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

The Cloud

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

13

YourSystem

SDK

CloudAP

I

VMVM

VM

➢ This can be an application server➢ Or your home machine➢ Or belong to one of your end users➢ Or a cloud VM➢ Or a smartphone

jclouds

Page 14: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

The Cloud

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

14

➢ Advantages➢ Metered pricing (pay as you go)➢ Project scalability (unlimited)➢ Safer (offsite/redundant)➢ Economies of scale

➢ Expertise➢ Support

Page 15: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

The Cloud

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

15

➢ Disadvantages➢ Less hardware control

➢ Provider-controlled downtime➢ Virtualization (efficiency)

➢ Provider lock-in➢ Provider-specific apis/sdks/features➢ Expensive to switch clouds or deploy locally➢ jclouds minimizes this disadvantage

Page 16: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

The Cloud

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

16

➢ Database specific advantages➢ Optimized by provider

➢ Settings➢ Container virtualization

➢ Automated software updates➢ Redundant data storage➢ Backup➢ Migration➢ Choice➢ Clustering (upcoming)➢ Support

Page 17: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

OpenStack

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

17

➢ Cloud OS➢ Open-sourced by Rackspace and NASA

(2010)➢ Free➢ Supported

➢ AT&T, HP, IBM, Red Hat, Rackspace, Dell, Cisco, Intel, VMware, …

Page 18: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

OpenStack

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

18

➢ http://stackalytics.com/

Page 19: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

jclouds

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

19

➢ Cloud SDK➢ Apache project➢ Java (and Clojure)➢ Easy➢ Portable➢ Cloud-agnostic➢ Community➢ Open source

Page 20: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

jclouds

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

20

➢ https://jclouds.apache.org/

Page 21: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

jclouds

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

21

➢ HTTP requests, responses, retries➢ Authentication and re-authentication➢ Pagination➢ Polling➢ Rate limits➢ Retries➢ Abstractions➢ Logging➢ Less code

Page 22: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

jclouds

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

22

Page 23: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

jclouds

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

23

➢ Services➢ Storage➢ Compute➢ VM Images➢ Load Balancers➢ DNS➢ Databases➢ …

➢ Providers➢ Openstack

➢ Rackspace➢ HP

➢ Amazon➢ Azure➢ …

Page 24: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

jclouds

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

24

AWS aws-ec2 US-VA,US-CA,IE,SG

Bluelock bluelock-vcloud-zone01 US-IN

CloudSigma US cloudsigma-lvs US-NV

CloudSigma CH cloudsigma-zrh CH-ZH

CloudSigma DC cloudsigma-wdc US-DC

DigitalOcean digitalocean

ElasticHosts GB elastichosts-lon-b GB-LND

ElasticHosts GB elastichosts-lon-p GB-LND

ElasticHosts US elastichosts-sat-p US-TX

Go2Cloud go2cloud-jhb1 ZA-GP

GoGrid gogrid US-CA,US-VA

Green House Data greenhousedata-element-vcloud

US-WY

HP hpcloud-compute US-NV

Ninefold ninefold-compute AU-NSW

OpenHosting openhosting-east1 US-VA

Rackspace UK (First Gen) cloudservers-uk GB-SLG

Rackspace US (First Gen) cloudservers-us US-IL,US-TX

Rackspace UK (Next Gen) rackspace-cloudservers-uk GB-SLG

Rackspace US (Next Gen) rackspace-cloudservers-us US-IL,US-TX

SeverLove serverlove-z1-man GB-MAN

SkaliCloud skalicloud-sdg-my MY-10

SoftLayer softlayer

ComputeProviders

Page 25: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

jclouds

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

25

➢ Showcase database code➢ Best practices➢ Compare with compute code➢ Gotchas and workarounds➢ Abstractions

Page 26: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Requirements

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

26

➢ Maven 3➢ Java 7+➢ jclouds➢ Windows or Linux [etc..] (thanks Java!)

Page 27: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Java Project

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

27

SkeletonProject

Create User

Create Instance

Initialize API

Create Database

Test Connection

Cleanup

Page 28: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Java Project

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

28

➢ Maven Dependencies➢ Selective subset➢ Versioning➢ jclouds-labs

Page 29: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

POM

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

29

https://github.com/jclouds/jclouds-examples/blob/master/rackspace/pom.xml

<dependency>

<groupId>org.apache.jclouds.provider</groupId>

<artifactId>rackspace-clouddatabases-us</artifactId>

<version>${jclouds.version}</version>

</dependency>

<dependency>

<groupId>mysql</groupId>

<artifactId>mysql-connector-java</artifactId>

<version>5.1.25</version>

</dependency>

Page 30: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

POM

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

30

mvn dependency:copy-dependencies "-DoutputDirectory=./lib"

Page 31: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Logging

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

31

// This module is responsible for enabling logging Iterable<Module> modules = ImmutableSet.<Module> of(new SLF4JLoggingModule());

ComputeServiceContext context = ContextBuilder.newBuilder(provider) .credentials(username, apiKey) .modules(modules) // don't forget to add the modules to your context! .buildView(ComputeServiceContext.class);

logback.xml<configuration scan="false"> … <appender name="WIREFILE" class="ch.qos.logback.core.FileAppender"> <file>target/test-data/jclouds-wire.log</file>

<encoder> <Pattern>%d %-5p [%c] [%thread] %m%n</Pattern> </encoder> </appender>

Page 32: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Java Project

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

32

SkeletonProject

Create User

Create Instance

Initialize API

Create Database

Test Connection

Cleanup

Page 33: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Initialize

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

33

private final TroveApi troveApi;

troveApi = ContextBuilder

.newBuilder("rackspace-clouddatabases-us")

.credentials(username, apiKey)

.buildApi(TroveApi.class);

Page 34: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Architecture

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

34

Cloud Databases

Trove

jclouds-database

Increasing level of abstraction

Provider settings:Endpoint, etc.

Implementation code

Abstractions

Page 35: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Architecture

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

35

Cloud Databases

Trove

jclouds-database

Increasing level of abstraction

Provider settings:Endpoint, etc.

Implementation code

Abstractions

Not implemented

Page 36: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Initialize

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

36

private final FlavorApi flavorApi;

flavorApi = troveApi.getFlavorApiForZone(ZONE);

Page 37: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

APIs

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

37

➢ TroveApi➢ FlavorApi

➢ InstanceApi

➢ DatabaseApi

➢ UserApi

➢ Utils

Page 38: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Java Project

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

38

SkeletonProject

Create User

Create Instance

Initialize API

Create Database

Test Connection

Cleanup

Page 39: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Create Instance

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

39

Flavor flavor = Iterables.getFirst(flavorApi.list(), null);

Instance instance = instanceApi.create(flavorId, size, name);

flavor.getId()

Volume size in GB

Page 40: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Create Instance

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

40

Instance updatedInstance = awaitAvailable(instance, instanceApi);

Page 41: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Create Instance

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

41

Instance updatedInstance = awaitAvailable(instance, instanceApi);

Polls status, waits until ready. Will not retry by itself!

Page 42: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Actual Code

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

42

InstanceApi instanceApi = api.getInstanceApiForZone(zone);

for (int retries = 0; retries < 10; retries++) {

Instance instance = null;

try {

instance = instanceApi.create(flavorId, size, name);

} catch (Exception e) {

Uninterruptibles.sleepUninterruptibly(15, TimeUnit.SECONDS);

logger.error(Arrays.toString(e.getStackTrace()));

continue;

}

Instance updatedInstance = awaitAvailable(instance, instanceApi);

if (updatedInstance != null) {

return updatedInstance;

}

instanceApi.delete(instance.getId());

InstancePredicates.awaitDeleted(instanceApi).apply(instance);

}

return null;

Page 43: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Actual Code

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

43

Utils.

getWorkingInstance(

String zone,

String name,

String flavorId,

int size)

Page 44: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Java Project

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

44

SkeletonProject

Create User

Create Instance

Initialize API

Create Database

Test Connection

Cleanup

Page 45: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Create Database

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

45

troveApi = ContextBuilder.newBuilder(PROVIDER)

.credentials(username, apiKey)

.buildApi(TroveApi.class);

instanceApi = troveApi.getInstanceApiForZone(ZONE);

databaseApi = troveApi

.getDatabaseApiForZoneAndInstance(ZONE,getInstance().getId());

getSomeApiForXandYandZ – fairly common in jclouds

Page 46: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Create Database

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

46

boolean result = databaseApi.create(NAME);

Page 47: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Java Project

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

47

SkeletonProject

Create User

Create Instance

Initialize API

Create Database

Test Connection

Cleanup

Page 48: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Create User

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

48

userApi = troveApi

.getUserApiForZoneAndInstance(ZONE, instance.getId());

boolean result = userApi.create(UNAME, PASSWORD, DBNAME);

User nameUser passwordDatabase name

Page 49: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Java Project

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

49

SkeletonProject

Create User

Create Instance

Initialize API

Create Database

Test Connection

Cleanup

Page 50: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Test Connection

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

50

Load balancer needed!

CreateLoadBalancer createLB = CreateLoadBalancer.builder()

.name(NAME)

.protocol("MYSQL")

.port(3306)

.algorithm(LoadBalancer.Algorithm.RANDOM)

.nodes(addNodes)

.virtualIPType(VirtualIP.Type.PUBLIC)

.build();

Page 51: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Test Connection

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

51

loadBalancer = lbApi.create(createLB);

Page 52: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Test Connection

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

52

StringBuilder connString = new StringBuilder();

connString.append("jdbc:mysql://");

connString

.append(getVirtualIPv4(getLb().getVirtualIPs()));

connString.append("/");

connString.append(DBNAME); connString.append("?user=");

connString.append(UNAME);

connString.append("&password=");

connString.append(PASSWORD);

Page 53: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Test Connection

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

53

Connection conn =

DriverManager.getConnection(connString.toString());

Page 54: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Test Connection

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

54

Statement stmt = conn.createStatement();

ResultSet rs = stmt.executeQuery("SELECT 3+5");

rs.first();

System.out.format(" 3+5 is %s%n", rs.getInt(1));

Page 55: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Going forward

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

55

➢ jclouds examples➢ https://github.com/jclouds/jclouds-examples

➢ More docs➢ http://jclouds.incubator.apache.org/documentation/➢ http://javadocs.jclouds.cloudbees.net/

➢ Maven alternatives?➢ http://jclouds.incubator.apache.

org/documentation/userguide/installation-guide/➢ Contribute!

➢ https://wiki.apache.org/jclouds/How%20to%20Contribute

Page 56: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Java Project

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

56

➢ Linux

➢ Compile➢ javac -classpath "lib/*:src/main/java/:src/main/resources/"

src/main/java/org/jclouds/examples/rackspace/*.java

➢ Run➢ java -classpath "lib/*:src/main/java/:src/main/resources/" org.jclouds.examples.

rackspace.clouddatabases.CreateInstance username apikey

Page 57: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Java Project

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

57

➢ Windows

➢ Compile➢ javac -classpath "lib/*;src/main/java/;src/main/resources/"

src/main/java/org/jclouds/examples/rackspace/*.java

➢ Run➢ java -classpath "lib/*;src/main/java/;src/main/resources/" org.jclouds.examples.

rackspace.clouddatabases.CreateInstance username apikey

Page 58: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Going forward

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

58

➢ TroveApi➢ Backup Extension

➢ Settings Extension

➢ Clustering

➢ Abstraction layer

Page 59: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Going forward

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

59

➢ TroveApi➢ Backup Extension

➢ Settings Extension

➢ Clustering

➢ Abstraction layer

Rackspace

Microsoft

Amazon

CouchDB

Salesforce

Page 60: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Going forward

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

60

➢ TroveApi➢ Backup Extension

➢ Settings Extension

➢ Clustering

➢ Abstraction layer

Rackspace

Microsoft

Amazon

CouchDB

Salesforce

Page 61: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Going forward

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

61

Collaborate!

http://jclouds.apache.org/

http://jclouds.apache.org/community/

Page 62: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Implementing an API

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

62

Fork <link to my fork>

Alternatively, fork jclouds-labs-openstack (for openstack).

Page 63: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Implementing an API

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

63

On to the IDE

Page 64: Taming the Cloud Database with Apache jclouds, ApacheCon Europe 2014

Taming the Cloud Database with Apache jclouds http://rack.to/ace14db

64

Thank you!http://[email protected]

Zack ShoylevSoftware Developer

zacksh #jclouds@zackshoylev