ChefConf 2013 Talk: Writing a Knife cloud plugin for your shiny VMWare vCloud Director

Writing a Knife Plugin for your shiny new VMWare vCloud

Director

Chirag Jog, CTO Clogenychirag@clogeny.com

Innovation → Execution → Solution → Delivered

http://www.clogeny.com © 2012 Clogeny Technologies

About Me

Built cutting edge products:• Hybrid Cloud

Migration Appliances• Enterprise Java PaaS• Multi-cloud single

pane of glass Dashboard

Contributed to several knife plugins• VMWare vCloud

Director• Google Compute

Engine• Windows Azure• Citrix Cloudstack• Terremark• Amazon EC2

Co-founder/CTO at Clogeny

http://www.clogeny.com © 2012 Clogeny Technologies

Agenda

What is the knife plugins ?

The Knife cloud plugin framework

VMWare vCloud Director API Model

Code Walkthrough of the knife-vcloud

plugin

Demo of the knife vcloud plugin

http://www.clogeny.com © 2012 Clogeny Technologies

What are knife cloud plugins ?

Extend Knife to support additional functionality

Virtual Machine Life cycle Server Create

Linux and Windows Bootstrapping

Server Destroy

List available cloud resources Images

Flavors / Server Types

Networks

Servers

http://www.clogeny.com © 2012 Clogeny Technologies

Available Cloud Plugins

http://www.clogeny.com © 2012 Clogeny Technologies

Knife Plugin Usage

gem install knife-<cloudname>

http://www.clogeny.com © 2012 Clogeny Technologies

Knife Plugin Usage

gem install knife-<cloudname>knife <cloudname> server create

http://www.clogeny.com © 2012 Clogeny Technologies

Knife Plugin Usage

gem install knife-<cloudname>knife <cloudname> server create• Flavor or Server size• Image• Zone/Region• Template/Distro• Chef Node Name• SSH params – ssh username, password, keypair,

keypair file• WinRM params – username, password,

AD/Kerberos params• Cloud Vendor Credentials

http://www.clogeny.com © 2012 Clogeny Technologies

Knife Plugin Usage

knife <cloudname> server delete• Chef Node Name• Purge Node/Client

knife <cloudname> server listknife <cloudname> image listknife <cloudname> flavor list

http://www.clogeny.com © 2012 Clogeny Technologies

High Level Diagram

...

knife-ec2

knife-rackspace

Knife-vcloud

….

AWS Driver

Rackspace Driver

Fog library

vCloud Driver

http://www.clogeny.com © 2012 Clogeny Technologies

High Level Diagram

...

knife-ec2

knife-rackspace

Knife-vcloud

….

AWS Driver

Rackspace Driver

Fog library

vCloud Driver

http://www.clogeny.com © 2012 Clogeny Technologies

fog.io

Knife Cloud plugins use fog to communicate with the Cloud services

http://www.clogeny.com © 2012 Clogeny Technologies

Fog

Fog - Multi-Cloud Services Library

Open source, Community driven

Portable, Powerful, Established

Simulated Fog behavior using Mock

Organizes resources – servers, images,

networks, vapps as collection of models.

http://www.clogeny.com © 2012 Clogeny Technologies

Using Fog

$ gem install fog

Fetching: fog-1.10.1.gem (100%)

Successfully installed fog-1.10.1

1 gem installed

http://www.clogeny.com © 2012 Clogeny Technologies

Using Fog Interactive

$ fog Welcome to fog interactive!>> connection = Fog::Compute.new(:provider => <Cloud Provider>, credential params..) >> images = connection.images.all[<Fog::Compute::Cloud::Image id=…, name=“ubuntu”>, <Fog::Compute::Cloud::Image id=…, name=“centos”>…]>> servers = connection.servers.all>> server = connection.servers.create(server_def_params)

http://www.clogeny.com © 2012 Clogeny Technologies

Using Fog Interactive

$ fog Welcome to fog interactive!>> connection = Fog::Compute.new(:provider => <Cloud Provider>, credential params..) >> images = connection.images.all[<Fog::Compute::Cloud::Image id=…, name=“ubuntu”>, <Fog::Compute::Cloud::Image id=…, name=“centos”>…]>> servers = connection.servers.all>> server = connection.servers.create(server_def_params)

http://www.clogeny.com © 2012 Clogeny Technologies

Using Fog Interactive

$ fog Welcome to fog interactive!>> connection = Fog::Compute.new(:provider => <Cloud Provider>, credential params..) >> images = connection.images.all[<Fog::Compute::Cloud::Image id=…, name=“ubuntu”>, <Fog::Compute::Cloud::Image id=…, name=“centos”>…]>> servers = connection.servers.all>> server = connection.servers.create(server_def_params)

http://www.clogeny.com © 2012 Clogeny Technologies

Using Fog Interactive

$ fog Welcome to fog interactive!>> connection = Fog::Compute.new(:provider => <Cloud Provider>, credential params..) >> images = connection.images.all[<Fog::Compute::Cloud::Image id=…, name=“ubuntu”>, <Fog::Compute::Cloud::Image id=…, name=“centos”>…]>> servers = connection.servers.all>> server = connection.servers.create(server_def_params)

http://www.clogeny.com © 2012 Clogeny Technologies

High Level Diagram

...

knife-ec2

knife-rackspace

Knife-vcloud

….

AWS Driver

Rackspace Driver

Fog library

vCloud Driver

http://www.clogeny.com © 2012 Clogeny Technologies

Knife plugin – Fog collections

knife-cloud

knife cloud server create

knife cloud server delete

knife cloud server list

knife cloud flavor list

knife cloud image list

servers

images

flavors

Fog Collections

http://www.clogeny.com © 2012 Clogeny Technologies

Knife plugin – Fog collections

knife-cloud

knife cloud server create

knife cloud server delete

knife cloud server list

knife cloud flavor list

knife cloud image list

servers

images

flavors

Fog Collections

http://www.clogeny.com © 2012 Clogeny Technologies

knife cloud server createclass Chef class Knife class CloudServerCreate < Knife

banner "knife vcloud server create (options)”

option :flavor, :short => "-f FLAVOR_ID", :long => "--flavor FLAVOR_ID", :description => "The flavor ID of server (m1.small, m1.medium, etc)", :proc => Proc.new { |f| Chef::Config[:knife][:flavor] = f }

option :image, :short => "-I IMAGE_ID", :long => "--image IMAGE_ID", :description => "The image ID for the server", :proc => Proc.new { |i| Chef::Config[:knife][:image] = i }

option :chef_node_name …. option :zone … Several bootstrapping options

http://www.clogeny.com © 2012 Clogeny Technologies

knife cloud server createclass Chef class Knife class CloudServerCreate < Knife

banner "knife vcloud server create (options)”

option :flavor, :short => "-f FLAVOR_ID", :long => "--flavor FLAVOR_ID", :description => "The flavor ID of server (m1.small, m1.medium, etc)", :proc => Proc.new { |f| Chef::Config[:knife][:flavor] = f }

option :image, :short => "-I IMAGE_ID", :long => "--image IMAGE_ID", :description => "The image ID for the server", :proc => Proc.new { |i| Chef::Config[:knife][:image] = i }

option :chef_node_name …. option :zone … Several bootstrapping options

Inherit from the Chef::Knife class

http://www.clogeny.com © 2012 Clogeny Technologies

knife cloud server createclass Chef class Knife class CloudServerCreate < Knife

banner "knife vcloud server create (options)”

option :flavor, :short => "-f FLAVOR_ID", :long => "--flavor FLAVOR_ID", :description => "The flavor ID of server (m1.small, m1.medium, etc)", :proc => Proc.new { |f| Chef::Config[:knife][:flavor] = f }

option :image, :short => "-I IMAGE_ID", :long => "--image IMAGE_ID", :description => "The image ID for the server", :proc => Proc.new { |i| Chef::Config[:knife][:image] = i }

option :chef_node_name …. option :zone … Several bootstrapping options

Usage Banner

Inherit from the Chef::Knife class

http://www.clogeny.com © 2012 Clogeny Technologies

knife cloud server createclass Chef class Knife class CloudServerCreate < Knife

banner "knife vcloud server create (options)”

option :flavor, :short => "-f FLAVOR_ID", :long => "--flavor FLAVOR_ID", :description => "The flavor ID of server (m1.small, m1.medium, etc)", :proc => Proc.new { |f| Chef::Config[:knife][:flavor] = f }

option :image, :short => "-I IMAGE_ID", :long => "--image IMAGE_ID", :description => "The image ID for the server", :proc => Proc.new { |i| Chef::Config[:knife][:image] = i }

option :chef_node_name …. option :zone … Several bootstrapping options

Parameter Options like Server size,

Image, Zone and Bootstrapping

Options

Usage Banner

Inherit from the Chef::Knife class

http://www.clogeny.com © 2012 Clogeny Technologies

knife cloud server create (run)def run - validate compulsory parameters ….

connection = Fog::Compute.new(…) … Cloud provider based checks … server = connection.servers.create(params) … Cloud provider specific logic – Enable Networking, Associate IP Address, Reconfigure Server … if windows_image? wait_for_winrm else wait_for_sshd end

// Server is now up and runningBootstrap the Node

end

http://www.clogeny.com © 2012 Clogeny Technologies

knife cloud server create (run)def run - validate compulsory parameters ….

connection = Fog::Compute.new(…) … Cloud provider based checks … server = connection.servers.create(params) … Cloud provider specific logic – Enable Networking, Associate IP Address, Reconfigure Server … if windows_image? wait_for_winrm else wait_for_sshd end

// Server is now up and runningBootstrap the Node

end

Knife commands are run by calling the `#run` method on an instance of your command class

http://www.clogeny.com © 2012 Clogeny Technologies

knife cloud server create (run)def run - validate compulsory parameters ….

connection = Fog::Compute.new(…) … Cloud provider based checks … server = connection.servers.create(params) … Cloud provider specific logic – Enable Networking, Associate IP Address, Reconfigure Server … if windows_image? wait_for_winrm else wait_for_sshd end

// Server is now up and runningBootstrap the Node

end

Knife commands are run by calling the `#run` method on an instance of your command class

Create the fog connection object

http://www.clogeny.com © 2012 Clogeny Technologies

knife cloud server create (run)def run - validate compulsory parameters ….

connection = Fog::Compute.new(…) … Cloud provider based checks … server = connection.servers.create(params) … Cloud provider specific logic – Enable Networking, Associate IP Address, Reconfigure Server … if windows_image? wait_for_winrm else wait_for_sshd end

// Server is now up and runningBootstrap the Node

end

Knife commands are run by calling the `#run` method on an instance of your command class

Create the fog connection object

Create and configure the server

http://www.clogeny.com © 2012 Clogeny Technologies

knife cloud server create (run)def run - validate compulsory parameters ….

connection = Fog::Compute.new(…) … Cloud provider based checks … server = connection.servers.create(params) … Cloud provider specific logic – Enable Networking, Associate IP Address, Reconfigure Server … if windows_image? wait_for_winrm else wait_for_sshd end

// Server is now up and runningBootstrap the Node

end

Knife commands are run by calling the `#run` method on an instance of your command class

Create the fog connection object

Create and configure the server

Wait for WinRM/SSH Service to start

http://www.clogeny.com © 2012 Clogeny Technologies

knife cloud server create (run)def run - validate compulsory parameters ….

connection = Fog::Compute.new(…) … Cloud provider based checks … server = connection.servers.create(params) … Cloud provider specific logic – Enable Networking, Associate IP Address, Reconfigure Server … if windows_image? wait_for_winrm else wait_for_sshd end

// Server is now up and runningBootstrap the Node

end

Knife commands are run by calling the `#run` method on an instance of your command class

Create the fog connection object

Create and configure the server

Wait for WinRM/SSH Service to start

Windows or Linux Bootstrapping

http://www.clogeny.com © 2012 Clogeny Technologies

Bootstrapping

Installs Chef and runs the chef-client to communicate with a Chef Server• via SSH• via Windows Remote

Management(WinRM)

Ref: http://www.lemen.com/imageBootstrap1.html

http://www.clogeny.com © 2012 Clogeny Technologies

Knife plugin – Fog collections

knife-cloud

knife cloud server create

knife cloud server delete

knife cloud server list

knife cloud flavor list

knife cloud image list

servers

images

flavors

Fog Collections

http://www.clogeny.com © 2012 Clogeny Technologies

knife cloud server deleteclass Chef class Knife class CloudServerDelete < Knife

banner "knife vcloud server delete (options)”

def run validate compulsory parameters …. connection = Fog::Compute.new(…)

Check whether server(s) exists or not@name_args.each do |server_id|

begin server = connection.servers.get(server_id) if !server

ui.warn(“Server does not exists”)next

end confirm("Do you really want to delete this server”) server.destroy ui.warn("Deleted server #{server.id}") #Destroy Chef Node # Destroy Chef Clientend

Inherit from the Chef::Knife class

Usage Banner

Find the Server and Delete it

http://www.clogeny.com © 2012 Clogeny Technologies

Knife plugin – Fog collections

knife-cloud

knife cloud server create

knife cloud server delete

knife cloud server list

knife cloud flavor list

knife cloud image list

servers

images

flavors

Fog Collections

http://www.clogeny.com © 2012 Clogeny Technologies

knife cloud image listclass Chef class Knife class CloudImageList < Knife

banner "knife cloud image list (options)"

def run image_list = [ ui.color('ID', :bold), ui.color('Name', :bold) ]

connection.images.sort_by(&:name).each do |image| image_list << image.id.to_s image_list << image.name end

puts ui.list(image_list, :uneven_columns_across, 2) end end endend

Inherit from the Chef::Knife class

Usage Banner

List Resource and Display it

http://www.clogeny.com © 2012 Clogeny Technologies

Knife plugin – Fog collections

knife-cloud

knife cloud server create

knife cloud server delete

knife cloud server list

knife cloud flavor list

knife cloud image list

servers

images

flavors

Fog Collections

http://www.clogeny.com © 2012 Clogeny Technologies

knife plugin directory structure

knife-vcloud├── Gemfile├── knife-vcloud.gemspec├── lib ├ chef├ knife-vcloud├ version.rb

├ knife├ vcloud_server_create.rb├ vcloud_server_delete.rb├ vcloud_image_list.rb├── spec├── README.md └── LICENSE

http://www.clogeny.com © 2012 Clogeny Technologies

knife plugin directory structure

knife-vcloud├── Gemfile├── knife-vcloud.gemspec├── lib ├ chef├ knife-vcloud├ version.rb

├ knife├ vcloud_server_create.rb├ vcloud_server_delete.rb├ vcloud_image_list.rb├── spec├── README.md └── LICENSE

Parent Directory

http://www.clogeny.com © 2012 Clogeny Technologies

knife plugin directory structure

knife-vcloud├── Gemfile├── knife-vcloud.gemspec├── lib ├ chef├ knife-vcloud├ version.rb

├ knife├ vcloud_server_create.rb├ vcloud_server_delete.rb├ vcloud_image_list.rb├── spec├── README.md └── LICENSE

Required to build the gem

Parent Directory

http://www.clogeny.com © 2012 Clogeny Technologies

knife plugin directory structure

knife-vcloud├── Gemfile├── knife-vcloud.gemspec├── lib ├ chef├ knife-vcloud├ version.rb

├ knife├ vcloud_server_create.rb├ vcloud_server_delete.rb├ vcloud_image_list.rb├── spec├── README.md └── LICENSE

Version Info

Required to build the gem

Parent Directory

http://www.clogeny.com © 2012 Clogeny Technologies

knife plugin directory structure

knife-vcloud├── Gemfile├── knife-vcloud.gemspec├── lib ├ chef├ knife-vcloud├ version.rb

├ knife├ vcloud_server_create.rb├ vcloud_server_delete.rb├ vcloud_image_list.rb├── spec├── README.md └── LICENSE

Actual Code

Version Info

Required to build the gem

Parent Directory

VMWare vCloud Director

http://www.clogeny.com © 2012 Clogeny Technologies

VMWare vCloud Director

Reference: http://www.vmguru.nl/wordpress/2012/01/vmware-vcloud-director-design-guidelines/

http://www.clogeny.com © 2012 Clogeny Technologies

vApps

Container of one or more VMs• Package up Multi-tier

Applications into vApps

• Operate on VMs as one unit

• Can be created by scratch

• Can be imported from outside the cloud

Reference: http://download3.vmware.com/vcat/documentation-center/index.html#page/Consuming%20a%20vCloud/3c%20Consuming%20a%20VMware%20vCloud.2.077.html

http://www.clogeny.com © 2012 Clogeny Technologies

Catalogs

Catalogs are collections of vApps, vApps Templates and mediaExamples:• Infrastructure as Service Catalogs• Pre-installed Windows/Linux VMs• OS Media files• App Catalogs

http://www.clogeny.com © 2012 Clogeny Technologies

vCloud Layered Network

External Network

Organization

Network

vApp Network

External Network : Providing a connection to the outside worldOrganization Network : Network contained within an organizationvApp Network : Network contained within a vApp

Ref: http://hol-prt.cloudfoundry.com/HOL-PRT-04_EN/HOL-PRT-04-m1/lessons/vCloud_Director_networking_and_Cisco_Nexus_1000V.html

http://www.clogeny.com © 2012 Clogeny Technologies

knife-vcloud

Git repo: https://github.com/opscode/knife-vcloudSupport to provision one-VM vApps onto the Org-level networkSupported Commands:• knife vcloud server create• knife vcloud server list • knife vcloud server delete• knife vcloud image list• knife vcloud network list

http://www.clogeny.com © 2012 Clogeny Technologies

knife-vcloud

Git repo: https://github.com/opscode/knife-vcloudSupport to provision one-VM vApps onto the Org-level networkSupported Commands:• knife vcloud server create• knife vcloud server list • knife vcloud server delete• knife vcloud image list• knife vcloud network list

Lets focus on it!

http://www.clogeny.com © 2012 Clogeny Technologies

Deploying a vApp

Ref: http://hol-prt.cloudfoundry.com/HOL-PRT-04_EN/HOL-PRT-04-m1/lessons/vCloud_Director_networking_and_Cisco_Nexus_1000V.html

Select the vApp Template.

http://www.clogeny.com © 2012 Clogeny Technologies

Deploying a vApp

Ref: http://hol-prt.cloudfoundry.com/HOL-PRT-04_EN/HOL-PRT-04-m1/lessons/vCloud_Director_networking_and_Cisco_Nexus_1000V.html

Select the vApp Template.Select the Network

http://www.clogeny.com © 2012 Clogeny Technologies

Deploying a vApp

Ref: http://hol-prt.cloudfoundry.com/HOL-PRT-04_EN/HOL-PRT-04-m1/lessons/vCloud_Director_networking_and_Cisco_Nexus_1000V.html

Select the vApp Template.Select the NetworkProvide a name, administrator Password to the vApp being deployed.

http://www.clogeny.com © 2012 Clogeny Technologies

Deploying a vApp

Select the vApp Template.Select the NetworkProvide a name, administrator Password to the vApp being deployed.Configure the virtual machines inside the vApp, including • computer name, • CPU and Memory• Network and IP Assignment.

http://www.clogeny.com © 2012 Clogeny Technologies

Deploying a vApp

Select the vApp Template.Select the NetworkProvide a name, administrator Password to the vApp being deployed.Configure the virtual machines inside the vApp, including • computer name, • CPU and Memory• Network and IP Assignment.

Bootstrapping!

http://www.clogeny.com © 2012 Clogeny Technologies

Deploying a vApp(Deep-Dive)

Select the vApp Template.

http://www.clogeny.com © 2012 Clogeny Technologies

Deploying a vApp(Deep-Dive)

Select the vApp Template.

http://www.clogeny.com © 2012 Clogeny Technologies

Deploying a vApp(Deep-Dive)

Select the vApp Template.

http://www.clogeny.com © 2012 Clogeny Technologies

Deploying a vApp(Deep-Dive)

Select the vApp Template.

http://www.clogeny.com © 2012 Clogeny Technologies

Deploying a vApp(Deep-Dive)

Select the vApp Template.

http://www.clogeny.com © 2012 Clogeny Technologies

Deploying a vApp (Deep-Dive)

Select the vApp Template.Select the Network

http://www.clogeny.com © 2012 Clogeny Technologies

Deploying a vApp (Deep-Dive)

Select the vApp Template.Select the Network

http://www.clogeny.com © 2012 Clogeny Technologies

Deploying a vApp (Deep-Dive)

Select the vApp Template.Select the Network

http://www.clogeny.com © 2012 Clogeny Technologies

Deploying a vApp (Deep-Dive)

Select the vApp Template.Select the NetworkBuild the server spec and deploy the vApp

http://www.clogeny.com © 2012 Clogeny Technologies

Deploying a vApp (Deep-Dive)

Select the vApp Template.Select the NetworkBuild the server spec and deploy the vApp

http://www.clogeny.com © 2012 Clogeny Technologies

Deploying a vApp (Deep-Dive)

Select the vApp Template.Select the NetworkBuild the server spec and deploy the vApp

http://www.clogeny.com © 2012 Clogeny Technologies

Deploying a vApp (Deep-Dive)

Select the vApp Template.Select the NetworkBuild the server spec and deploy the vApp

http://www.clogeny.com © 2012 Clogeny Technologies

Deploying a vApp (Deep-Dive)

Additional Configuration

http://www.clogeny.com © 2012 Clogeny Technologies

Deploying a vApp (Deep-Dive)

Additional Configuration

http://www.clogeny.com © 2012 Clogeny Technologies

Deploying a vApp (Deep-Dive)

Additional Configuration

http://www.clogeny.com © 2012 Clogeny Technologies

Deploying a vApp (Deep-Dive)

Additional Configuration

http://www.clogeny.com © 2012 Clogeny Technologies

Deploying a vApp (Deep-Dive)

Additional Configuration

http://www.clogeny.com © 2012 Clogeny Technologies

Deploying a vApp (Deep-Dive)

Save Additional Configuration

http://www.clogeny.com © 2012 Clogeny Technologies

Deploying a vApp (Deep-Dive)

Power On!

http://www.clogeny.com © 2012 Clogeny Technologies

Deploying a vApp (Deep-Dive)

Wait until ready

http://www.clogeny.com © 2012 Clogeny Technologies

Deploying a vApp (Deep-Dive)

Linux Bootstrapping!

http://www.clogeny.com © 2012 Clogeny Technologies

Deploying a vApp (Deep-Dive)

Linux Bootstrapping!

http://www.clogeny.com © 2012 Clogeny Technologies

Deploying a vApp (Deep-Dive)

Linux Bootstrapping!

http://www.clogeny.com © 2012 Clogeny Technologies

Deploying a vApp (Deep-Dive)

Windows Bootstrapping!

http://www.clogeny.com © 2012 Clogeny Technologies

Deploying a vApp (Deep-Dive)

Windows Bootstrapping!

http://www.clogeny.com © 2012 Clogeny Technologies

knife vcloud server create (Linux)knife vcloud server create --image <vApp-Linux-template ID>--node-name linux_node_test --network <Network ID>--ssh-user <SSH Username>--ssh-password <SSH Password>--distro chef-full

http://www.clogeny.com © 2012 Clogeny Technologies

knife vcloud server create (Windows)

knife vcloud server create --image <vApp-Windows-template ID>--node-name windows_node_test--network <Network ID>--bootstrap-protocol winrm--winrm-password <Windows WinRM Password>--template-file ../knife-windows/lib/chef/knife/bootstrap/windows-chef-client-msi.erb -VV

http://www.clogeny.com © 2012 Clogeny Technologies

Demo

http://www.clogeny.com © 2012 Clogeny Technologies

knife vcloud server create

Source Code: https://github.com/opscode/knife-vcloud/blob/master/lib/chef/knife/vcloud_server_create.rb

http://www.clogeny.com © 2012 Clogeny Technologies

knife vcloud server delete

Identify the vApp to deletePower Off the vAppDelete the vApp

http://www.clogeny.com © 2012 Clogeny Technologies

knife vcloud server delete

Identify the vApp to deletePower Off the vAppDelete the vApp

http://www.clogeny.com © 2012 Clogeny Technologies

Resource Listing

• knife vcloud server list• knife vcloud image list• Knife vcloud network list

http://www.clogeny.com © 2012 Clogeny Technologies

knife vcloud server list

http://www.clogeny.com © 2012 Clogeny Technologies

knife vcloud image list

http://www.clogeny.com © 2012 Clogeny Technologies

knife vcloud network list

http://www.clogeny.com © 2012 Clogeny Technologies

Demo

http://www.clogeny.com © 2012 Clogeny Technologies

Chefs: now with Knife expertise!

Ref: http://s3.hubimg.com/u/1691762_f520.jpg

Thank You!

Innovation → Execution → Solution → Delivered

Feel free to reach out :Email: chirag@clogeny.com

Twitter: @cheezo