Download pdf - My CloudConnect 2013 Talk: Infrastructure Automation in the Cloud

Innova'on → Execu'on → Solu'on → Delivered

Infrastructure Automation for the Cloud

Chirag Jog, CTO Clogeny [email protected]

Clogeny Confidential

http://www.clogeny.com © 2013 Clogeny Technologies

About Me

•  Strong Experience in DevOps, Infrastructure Automation

•  Built cutting edge products: •  Hybrid Cloud Migration Appliances •  Enterprise Java PaaS •  Multi-cloud single pane of glass Dashboard

•  Experience building highly scalable automated cloud applications

•  Leading engineering teams

Co-founder/CTO at Clogeny


What is my talk about?






What is Infrastructure?

App LBs

App Servers

NoSQL

DB slaves

Cache

DB Cache

DBs


What is Infrastructure?

DC1

DC3

DC2


Problems in Managing Infrastructure

"   Configuration Drift

"   Changes causes Outages

"   Wall of Confusion – Dev and Ops

"   Non Functional Requirements: •  Backups, security, monitoring, synchronization,

performance, capacity planning

" Dev, Test and Prod – All Different Environments

"   Mystery Machine: On But don’t know why?

"   Code Complete != Production Ready


Infrastructure is Hard !!


Old Problem: Infrastructure is Hard



"   Proprietary Solutions



"   Solve very little of the problem "   Enterprise Focused "   Large Professional Services Bills "   On-premise "   Acquired by companies with large consulting

organizations "   Google, Amazon, Microsoft built their own tools

but its secret sauce

"   Proprietary Solutions


Infrastructure is Changing !!


Cloud Computing


New Problem: Infrastructure is Changing

"   Easier to get .. but harder to manage

"   Demand is dynamic

"   Developers are crucial to Operations

"   Web/Cloud services are proliferating

"   Manual Configuration no longer a crutch

"   Few tools solve all the problems

"   No Shrink Wrap software


Infrastructure is Code

"   API Driven Abstractions "   Infrastructure is now an Application

App LBs

App Servers

NoSQL

DB slaves

Cache

DB Cache

DBs


Agile Infrastructure: The Final Frontier for Agile


Agile Infrastructure

"   Infrastructure is Code: Apply the lessons learnt from Software Development

"   Build more Flexible systems and faster feedback

"   Beyond Continuous Integration: Continuous Delivery

"   10+ deploys per Day – Flickr, 2009


Amazon.com

"   Mean time between deployments: 11.6s

"   Max # of deploys per hour : 1079

"   Mean # of host simultaneously receiving a

deployment: 10,000

"   Max # of hosts receiving a deployment: 30,000





Agile Infrastructure Techniques

u Configuration Management

u Version Control Everything

u Build from source

u One step deploy

u Monitoring

u Continuous Delivery

u Work Together – Dev and Ops


What is Infrastructure (again) ?

• Nodes • Networking • Files • Directories • Symlinks • Mounts • DB Schema

Infrastructure is a collection of resources:

•  Routes •  Users • Groups •  Tasks •  Packages •  Software •  Services • Configurations •  Stuff


Infrastructure as Code (really?)

package "ntp" do ! action :install !end !

service "ntpd" do ! action [:enable,:start] !end !

template "/etc/ntpd.conf" do ! source "ntpd.conf.erb" ! owner "root" ! group "root" ! mode 0644 ! action :create ! variables(:time_server => “time.example.com”) ! notifies :restart, “service[ntpd]”!end !


Configuration Management

"   Infrastructure as Code "   Put systems into a known state "   Audit and enforce consistency "   Manage server lifecycle "   Reason about services, instead of systems "   Apply dev-test-prod cycle to infrastructure


Configuration Management Tools


Version Control Everything

"   Network Configurations "   System Configurations "   Application Configurations "   Application Code "   Database Schema "   Documentation "   Anything else that matters!


Build from Source

"   Build Application and Infrastructure from source

" Dev, Test and Prod in sync

"   Test from a known state

"   Scaling

"   Disaster Recovery


One Step Deploy

"   One automated process from version control to live services

"   Lower the time and cost to deploy


Monitoring

"   Define ‘Normal’ Conditions "   Monitor Baselines, Charts, trends "   Monitor, Monitor and Monitor !

Ref: http://consultingblogs.emc.com/blogs/gracemollison/image_thumb_123A5775.png


Continuous Integration

"   Check in regularly "   Create an automated and comprehensive test

suite. "   Test New builds "   Assert Services are running "   Deploy early and often "   No manual steps! "   Never go home on a broken build "   Build a pipeline!

Poll Master and

Compile

Run Unit Tests

Package

Deploy

Run Smoke/

Integration Tests

Release


Continuous Delivery Pipeline

SCM Build CI Analyze Deploy Config

Private Clouds

Public Clouds

Repo Infrastructure

Continuous Integration

Continuous Deployment


Work Together – Dev and Ops

" Dev and Ops see the same thing, in the same place

"   Share Metrics "   Share the Repository •  Configs in sync with Application Code

"   Minimize Surprise "   Always ship trunk/master


How often do we see this?


Work Together – Dev and Ops

" Devs need to understand the infrastructure "   Ops needs to understand the Application "   Manage Flows – Avoid Fires "   Build a culture – Avoid the blame game "   Break the silos and work together! "   Collaboration: not only when things go wrong "   No Rock star mentality


What is DevOps ?

DevOps: is a software development method that stresses communication, collaboration and integration between Dev and Ops.


What is DevOps ?

DevOps really is: "   Processes "   Tools "   Culture


DevOps

"   Processes and Tools •  Automated infrastructure •  Shared version control •  One step build and deploy •  Shared metrics

"   Culture •  Respect •  Trust •  Healthy attitude about failure •  Avoiding Blame •  Superhero Culture


Conclusion

ü Agile Infrastructure ü Embrace DevOps ü Business Agility ü  Improved Time to Market ü Ahead of the competition


Questions ?


References

"   Dev2Ops - http://dev2ops.org

"   Flickr’s 10 deploys per day - http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr

"   Agile Infrastructure - http://www.slideshare.net/littleidea/agile-infrastructure-agile-2009

" DevOps – The war is over - http://www.slideshare.net/jedi4ever/devops-the-war-is-over-if-you-want-it

" DevOps, the future is here – http://www.slideshare.net/KrisBuytaert/devops-the-future-is-here-its-just-not-evenly-distributed-yet

"   Jesse Robbins’s DevOps talk - http://www.slideshare.net/jesserobbins/devops-interop-las-vegas-jesse-robbins-opscode