Hybrid cloud openstack meetup

Copyright © GigaSpaces 2014. All rights reserved.1 Copyright © GigaSpaces 2016. All rights reserved.

Orchestrating In Hybrid Environments

DeWayne FilppiGigaspaces Technologies2/10/2016

Kubernetes, Openstack, and beyond

Copyright © GigaSpaces 2014. All rights reserved.2

Agenda

• Describe real-world effort orchestrating Kubernetes on Openstack

• Not just Kubernetes, but Kubernetes + non-Kubernetes hosted services in a single orchestration

• Add to that manual and automated metric-driven scaling inside Kubernetes, and of Kubernetes itself


Flash Kubernetes Review

• Container Orchestration (docker, rkt)

• Master/minion architecture

• Containers considered immutable

• Pod concept

• Replication controller concept

• Service concept

• kubectl utility


Kubernetes Overview


Target Architecture


Orchestration Steps

• Preparation– Create Docker containers for Nodejs and metrics collection– Instrument containers to configure externals– Create RC and service descriptor templates– Create Riemann config for threshold detection

• Runtime– Create network/subnet/router for VMs on Openstack– Create VMs for Kubernetes and MongoDb on network– Create VM for Riemann event processor– Deploy Riemann and start with detection config– Save Kubernetes, Mongo Ips/ports, Riemann IPs/ports– Install MongoDb on Mongo VM(s)– Set env vars in RC template (Mongo and Riemann location etc…)– ……


Orchestration Steps

• Runtime (continued)

– Copy rendered templates to Kubernetes master

– Use kubectl on master to create replication controller (Pod with nodejs and collector)

• Causes nodejs to start and connect to external Mongodb cluster

• Causes metric collection to begin and pushed in Riemann

– Use kubectl on master to create service endpoint

• Creates externally accessible single IP to access Pod collection


Orchestration Steps

• Runtime (continued)

– When threshold breached, execute kubectl rcscale.

– When Kubernetes runs out of space, expand Kubernetes by adding another node/minion and updating the master

– Wrap all the above with retry logic


Analysis

• Automating that is a lot of work

• Complex/error prone

• Making cloud neutral difficult


TOSCA Modeling

• Topology Orchestration for Cloud Applications (OASIS)

• Not really. More like Orchestration for Applications.

• Models deployments as DAG of “nodes”

• Nodes are anything that needs orchestrating (hardware, software, VMs, containers, networks, floating Ips, etc….)

• Everything connected by relationships


TOSCA Modeling

Example


TOSCA Modeling

• An orchestrator applies “workflows” to the node graph/model.

• TOSCA defines a type system that lets custom types to be defined.

• Types have operations tied to code

• Example:

– A “Server” node for Openstack would define operations needed to operate nova/neutron/cinder etc….


Modeling Our Target Architecture

• Define custom types for Kubernetes:

– Master

– Minion/Node

– Define operations on these to install and configure Kubernetes

• Probably delegate to Salt, Puppet, Ansible, Chef, etc..



• Define custom types for MongoDb:

– Mongod, Mongocfg

– Mongos

– Define operations on these to install and configure MongoDb

• Probably delegate to Salt, Puppet, Ansible, Chef, etc..



• Define custom type for a generic Microservice

• Define a relationship that installs the microservice on the Kub master

• Note that the Microservice type can be configured to upload arbitrary to the master node



• Complete model (simplified)• Assume flat network


Rendering The Architecture

• A standard workflow will walk the model and execute associated code.• First relationships are evaluated to determine order.• A tree of tasks is constructed, and the tasks are executed

• Note the VMs are independent.• They are all rendered/instantiated in parallel



• Now the next independent set can be done• Install Master and Mongod



• Now the last independent set can be done• Install “Node” and Microservice


Note on Cloud Portability Concerns

• Three approaches– Abstraction (AKA “least common denominator”)

• Not necessarily all that bad if the targets are reasonably isomorphic.

– Pluggability• Future proof

• No “least common denominator”

• Potential difficulty translating between targets

– Hybrid • Write a plugin that serves as a façade for multiple

clouds


Note on Cloud Portability Concerns

• Example Equivalent Node Typescloudify.nodes.Root:

interfaces:cloudify.interfaces.lifecycle:

create: {}configure: {}start: {}stop: {}delete: {}

cloudify.nodes.SecurityGroup:derived_from: cloudify.nodes.Root

cloudify.openstack.nodes.SecurityGroup:derived_from: cloudify.nodes.SecurityGroupproperties:

description:type: stringdefault: ''

rules:default: []

disable_default_egress_rules:default: false

interfaces:cloudify.interfaces.lifecycle:create:

implementation: openstack.neutron_plugin.security_group.create

inputs:args:

default: {}openstack_config:

default: {}delete:

implementation: openstack.neutron_plugin.security_group.delete

inputs:openstack_config:

default: {}

cloudify.aws.nodes.SecurityGroup:derived_from: cloudify.nodes.SecurityGroupproperties:description:

type: stringrequired: true

rules:default: []description: >

You need to pass in either src_group_id(security group ID) OR cidr_ip,

and then the following three: ip_protocol, from_port and to_port.

interfaces:cloudify.interfaces.lifecycle:

create: aws.ec2.securitygroup.createdelete: aws.ec2.securitygroup.delete


Concrete Implementation

• Kubernetes – A server/VM definition

master_host:type: cloudify.openstack.nodes.Serverinstances:deploy: 1

properties:cloudify_agent:user: ubuntu

image: {get_input: image}flavor: {get_input: flavor}

relationships:- target: kubernetestype: cloudify.relationships.contained_in

- target: master_security_grouptype: cloudify.openstack.server_connected_to_security_group

• Under the hood, the orchestrator uses construct proper nova/neutron API calls.

• Same concept applies to Nodejs architecture



• Kubernetes – A security group definition

• Security groups are independent and created immediately

master_security_group:type: cloudify.openstack.nodes.SecurityGroupproperties:security_group:name: master_security_groupdescription: kubernetes master security group

rules:- remote_ip_prefix: 0.0.0.0/0 # for remote installport: 22

- remote_ip_prefix: 0.0.0.0/0port: { get_property: [ master, master_port ]}

- remote_ip_prefix: 0.0.0.0/0 # for minionsport: 4001

- remote_ip_prefix: 0.0.0.0/0 # for serviceport: 30000



• Kubernetes – Custom Microservice Typecloudify.kubernetes.Microservice:

derived_from: cloudify.kubernetes.Baseproperties:

name:description: the name of the service

image:description: the image to rundefault: ''

port:description: the port for the servicedefault: -1

target_port:description: the target port to mapdefault: {get_property: [SELF,port]}

protocol:description: the service protocol { TCP|UDP } TCP defaultdefault: TCP

replicas:description: the number of instances to rundefault: 1

run_overrides:description: json overrides for kubectl rundefault: '‘

……….



• Kubernetes – Custom Microservice Type: Behavior

cloudify.kubernetes.Microservice:

……….interfaces:

cloudify.interfaces.lifecycle:start:implementation: kubernetes.kube_plugin.tasks.kube_run_expose

stop:implementation: kubernetes.kube_plugin.remote_tasks.kube_delete

• Note that arbitrary code is tied to lifecycle methods.• Specific workflow recognize certain interfaces• The standard “install” and “uninstall” workflows recognize nodes that

implement “cloudify.interfaces.lifecycle”.• Note also that the “Microservice” type is simply automation, it’s not tied

to a VM, e.g. TOSCA ‘nodes’ can be purely logical constructs.



• Kubernetes – Custom Microservice Type: Behavior

• kubernetes.kube_plugin.tasks.kube_run_expose Is a reference to python function (kube_run_expose)

• It looks at the Microservice node and properties, finds the defined Kubernetes descriptor files, performs substitutions and deploys them in the Kub cluster.



• Kubernetes – Custom Microservice Type:

nodecellar:type: cloudify.kubernetes.Microserviceproperties:name: nodecellarssh_username: ubuntussh_keyfilename: /root/.ssh/agent_key.pemconfig_files:

- file: pod.yamloverrides:

- "['spec']['template']['spec']['containers'][0]['env'][1]['value'] = '@{mongo_proxy,mongo_info,ip}'"- "['spec']['template']['spec']['containers'][0]['env'][2]['value'] = '@{mongo_proxy,mongo_info,port}'"- "['spec']['template']['spec']['containers'][1]['env'][0]['value'] = '%{management_ip}'"- "['spec']['template']['spec']['containers'][1]['env'][2]['value'] = '%{deployment.id}'"- "['spec']['template']['spec']['containers'][1]['env'][3]['value'] = '%{node.id}'"- "['spec']['template']['spec']['containers'][1]['env'][4]['value'] = '%{instance.id}'"

- file: service2.yaml

relationships:- type: cloudify.kubernetes.relationships.connected_to_master

target: kubernetes_proxy- type: cloudify.relationships.connected_to_proxy

target: mongo_proxy



• Kubernetes Native DescriptorapiVersion: v1kind: ReplicationControllermetadata:name: nodecellar

spec:replicas: 1selector:app: nodecellar

template:metadata:name: nodecellarlabels:app: nodecellar

spec:containers:- name: nodecellarimage: dfilppi/nodecellar:v2workingDir: /command: ["bash","start.sh"]ports:- containerPort: 3000

hostPort: 3000hostIP: 0.0.0.0

env:- name: NODECELLAR_PORT

value: '3000'- name: MONGO_HOST

value: '15.125.81.204'

- name: MONGO_PORTvalue: '27400'

- name: diamonddimage: dfilppi/diamond:v1workingDir: /command: ["bash","start.sh"]ports:- containerPort: 5672hostPort: 5672hostIP: '0.0.0.0'

env:- name: CH_SERVERvalue: '10.67.79.2'

- name: CC_PORTvalue: '3000'

- name: CC_DEPLOYMENTvalue: ''

- name: CC_NODEvalue: ''

- name: CC_INSTANCEvalue:



• Kubernetes – Containers

• Orchestration info from blueprint passed in environment via overrides

• Container has simple start script that grabs environment and uses it to configure service

cfg=ConfigObj("/diamond/diamond.conf",list_values=False)cfg['collectors']['ConnCollector']['enabled']="true"cfg['collectors']['ConnCollector']['path_prefix']=os.getenv("CC_DEPLOYMENT")cfg['collectors']['ConnCollector']['port']=os.getenv("CC_PORT")cfg['collectors']['ConnCollector']['hostname']='.'.join([ip,os.getenv("CC_NODE"),os.getenv("CC_INSTANCE")])

cfg['handlers']['CloudifyHandler']['server']=os.getenv('CH_SERVER')cfg['handlers']['CloudifyHandler']['port']=os.getenv('CH_PORT',5672)cfg['handlers']['CloudifyHandler']['user']=os.getenv('CH_USER','cloudify')cfg['handlers']['CloudifyHandler']['password']=os.getenv('CH_PASSWORD','c10udify')cfg.write()



• Service Metrics Approach

– Scenario requires metrics from Nodejs

– Solution: provide Diamond collector container

– Deploy collector container in Pod with Nodejs

– Configure Diamond using container environment:

• Target host (server running Riemann)

• Information for queue. Point to correct deployment



• Service Metrics

– Diamond container pushes metrics to Riemann

• Sends number of connection in local Pod

– Riemann evaluates metrics. When threshold breached, executes Kubernetes scale workflow on orchestrator

– Orchestrator calls scale on Kubernetes master



• Reimann Overview

– High speed event processor

– Minimal state (only last sample of each metric)

– Calls user “streams” with metric

– Uses functional programming model

– Provides an API with utility streams and functions

– Streams are high order functions (functions that return functions)

– Implementation language is Clojure



(where (service #"{{service_selector}}")

(let [ hosts (atom #{}) ](fn [e]

; store or remove host from set depending on whether it has expired.(let [ key (str (:host e) "." (:service e))]

(do(if (expired? e)

(swap! hosts disj key)(swap! hosts conj key)

);save the host count(riemann.index/update index (assoc e :host nil :metric (max 1 (count @hosts)) :service "hostcount" ))

)))) )

(where (not (nil? (riemann.index/lookup index nil "hostcount")))(where (not (expired? event))

(moving-time-window {{moving_window_size}}(smap folds/mean(fn [ev]

(let [hostcnt (:metric (riemann.index/lookup index nil "hostcount"))conns (/ (:metric ev) (max hostcnt 1))cooling (not (nil? (riemann.index/lookup index "scaling" "suspended")))

](if (and (not cooling) ({{scale_direction}} hostcnt {{scale_limit}}) ({{scale_direction}} {{scale_threshold}} conns))(do

(process-policy-triggers ev)(riemann.index/update index {:host "scaling" :service "suspended" :time (unix-time) :description "cooldown flag" :metric 0 :ttl

{{cooldown_time}} :state "ok"}))))))))


Microservice Recap

• Containers prepared for Pod/RC

• Orchestrator feeds connection info from Nodejs to Mongo and Diamond collector to Riemann via environment in Kub descriptors

• Orchestrator copies edited descriptors to Kub master

• Kubectl “create” called remotely

• Nodejs connects to Mongo in cloud

• Diamond connects to Riemann and pushes metrics

• Riemann commands orchestrator to scale Kubremotely (kubectl rc scale)


Take Aways

• TOSCA makes complex orchstrations (more) understandable

• TOSCA hides cloud (or non-cloud) implementation details from orchestration itself

• An orchestrator can render a TOSCA blueprint on any infrastructure (containers, VMs, hardware, anything with an API)

• An orchestrator can coordinate other orchestrators in hybrid context (Kubernetes, Cloud, Virtual, Physical)

Copyright © GigaSpaces 2014. All rights reserved.36 Copyright © GigaSpaces 2016. All rights reserved.

Thank You

Software

Hybrid cloud openstack meetup