Lessons from Rule the Stack:

Rapid OpenStack* ProvisioningPete Chadwick, Michael Kadera, Dirk Müller, Adam Spiers

Agenda

• Rule the Stack competition

• Past competitions: challenges and results

• Translating lessons learned to benefit real world deployments

• Looking forward to the Austin Summit

Rule the Stack competition

4

Community competition to showcase OpenStack readiness, tools and skills

• Demonstrate OpenStack readiness through showcasing RESULTS

• Demonstrate community leadership and involvement• Create buzz/excitement and FUN through friendly

competition

Objective

Atlanta Paris Vancouver

Great momentum and enthusiasm:

• “It was a great experience, one that I plan to repeat”

• http://www.hitchnyc.com/i-ruled-the-stack-and-lived-to-tell-about-it/

• “I'm sure every contestant had fun and there was quite a good mood in the area reserved for the contest…. There were even more contestants (including some who are not vendors of an OpenStack distribution), which is really good to see. I hope we'll see even more in Tokyo!”

• http://www.vuntz.net/journal/post/2015/05/26/SUSE-Ruling-the-Stack-in-Vancouver

Soundbites from past competitors

6

Why hold a competition?

● Combat two perceptions:○ OpenStack is hard○ It takes a long time to deploy

● Demonstrate enterprise readiness● Generate some buzz● Include community point of view

○ Challenge people to attempt approaches not done in the past○ Uncover/promote new ideas and areas of improvement

Past competitionsChallenges and results

Past three competitions

9

Competition setup for head to head battle Atlanta & Paris: 8 node configuration per team

Vancouver: 6 node configuration per team

Systems 1, 2 & 3 Systems 4 & 5 Systems 6, 7 & 8

2 x Intel® Xeon® E5-2650v216GB RAM1x480GB SSD

2 x Intel® Xeon® E5-2650v2128GB RAM1x800GB SSD

2 x Intel® Xeon® E5-2650v264GB RAM3x480GB SSD

Systems 1, 2, 3 & 4 Systems 5 & 6

2 x Intel® Xeon® E5-2650v232GB RAM1x800GB SSD

2 Intel® Xeon® E5-2697v364GB RAM1x800GB SSDTPM

10

Time deductions:

● 30 min: “Live Upgrade” from Havana to Icehouse

● 10 min: “Live Migrate” with block storage

● 5 min: Running OpenStack HA

● 5 min: Launch VMs as collection via Heat

Atlanta - Pure Speed

11

Atlanta - Results

● Dirk Müller @ SUSE completed in 3:14 mins ● Ryan Moe @ Mirantis completed in 7:49 mins with

deductions for Heat and VM live-migration

12

Atlanta - How did SUSE end up at 3 min 14 ?

● 26s SUSE Linux Enterprise install OS including OpenStack packages (OEM Installer)

● Kexec● 32s Booting Operating System● 30s waiting for DHCP timeout● 106s OpenStack initialisation + service start

○ Using openSUSE’s OpenStack-QuickStart○ Installing+configuring up single tenant/single VM/single Node Cloud

with Neutron, Glance Cinder and Nova

13

Atlanta - Lessons learned for participants

● Do not rely on○ DHCP, external route, DNS, NTP○ Manual steps

● Use Kiwi*○ Image Builder for creating compressed, self-installing OS image on an

USB stick

● Minimal solution wins○ No multi-node

http://opensuse.github.io/kiwi/

14

Atlanta - Lessons learned for future competitions

● Time benefits were unbalanced○ 5 min for HA vs 5 min for deploying Heat

● Goal was to be open for everyone○ Rules were not announced in advance○ Many tried git checkouts with devstack

● Contest rules need to ○ Require preparation○ Require realistic deployments

15

Paris - Challenges

HA with penalties for failures when

● One controller node goes down

● Individual service goes down of cloud breaker’s choice

● Whole cluster of controllers goes down

● One Nova* node shut down

16

Paris - Results

Adam Spiers (& Dirk) from SUSE with 53 minutes

17

Paris - How did SUSE win the HA install?

● Adam had a workshop on this at the Paris Summit:○ “Automated Deployment of a Highly Available OpenStack Cloud”

● So SUSE already had everything prepared, tested and optimized to install in less than 1 hour

○ Vagrant files with prepared images

○ SUSE OpenStack Cloud using Crowbar’s batch build feature

● Nobody else finished the competition successfully

● Also, there was no HA for power :-)

18

Paris - Lessons learned for future competitions

● More difficult challenge reduced the number of participants

● Competition details should be available in advance

● Realistic deployment requirements vs. time benefits

19

Vancouver - Challenge

● Speed & accuracy● Provisioning time● Latest OpenStack release

○ Penalty for earlier release● Rolling upgrade● Performance tuning with “enhanced

platform awareness”● VM deployment with Heat● Secure compute environment with

Trusted Compute Pools● VM live migration

The Ruler of the Stack with the overall fastest scored time

1. Dirk @ SUSE, with a scored time of negative 9 minutes 54 seconds with bonuses

2. Adam and Vincent @ SUSE with a scored time of 10 minutes 17 seconds

Notable mention

○ “Setting the Bar” – Roman and Sergey @ Mirantis were the only competitors to take the challenge and deploy OpenStack on Summit Day 1 (scored time of 1 hour 16 minutes, which held for 3 days)

○ “Determined Competitor Award” – Walter @ Rackspace put up with his timeslots occurring during our router outage, helped us debug issues, and spent more time working to get his stack deployed than anyone

○ “Mighty Girl / Newbie Award” – Ella @ Red Hat earned the first female competitor

Vancouver - Results

How did SUSE win the Ruler of the Stack?Dirk’s Negative 9 minutes 54 seconds

● Refreshed Atlanta OpenStack image on top of SLE12 and OpenStack Kilo

● Split into two images, one for controller and one for compute nodes

● Deployed additional things like e.g. Horizon for easier demonstration

● Deployed everything in one go using 6 USB sticks

● Systemd

Vincent’s/Adam’s 10 minutes 17 seconds● Appliance of SUSE OpenStack

Cloud 5 product install ● Taking 10 minute penalty for

deploying Juno● Demonstrated additional

features to reduce penalty● Left out HA

How did SUSE win the 2nd Ruler of the Stack?

23

Vancouver - Lessons Learned

● Automation

○ Open Build Service, Kiwi, Crowbar, SUSE Linux Enterprise Server OEM install

● Newest OpenStack version before product release is a challenge

● Prepare and know all networking+h/w details

24

Vancouver - Lessons for Future

● Realistic deployments● Focus on manageability and Deployment quality● “Live Upgrade” never attempted

Translating lessons learnedHow competition results can help your real world deployment

27

Man is a Tool-Using Animal. Without tools man is nothing,

with tools he is all. ―Thomas Carlyle

Parameters

Components

1400

11

Why use an OpenStack distribution?

Cloud Orchestration (Heat*)

Dashboard(Horizon*)

Cloud APIs(OpenStack and EC2)

RequiredServices

Message QDatabase

Identity(Keystone*)

Images(Glance*)

Hypervisor

Compute(Nova*)

Operating System

Physical Infrastructure: x86-64, Switches, Storage

OpenStack Management Tools OS and Hypervisor

Object(Swift*)

Network(Neutron*)

Telemetry(Ceilometer*)

Physical Infrastructure

Block(Cinder*)

Required components

RequiredServicesMessageQDatabase

Inst

all F

ram

ewor

k

Physical Infrastructure: x86-64, Switches, Storage

Adapters

Vendor Solutions

Highly Available Services

Shared Files(Manila*)

OpenStack components

•Speed of execution

•Simplified implementation

•Repeatable and reliable

•Scalable

Benefits of an OpenStack distribution

Many options… See commercial and supported options at:https://www.openstack.org/marketplace/distros/

31

Real world lessons

● Fit-for-purpose planning● Networking● Hardware (re)boot times - BIOS

POST incredibly slow● OS installation speed● Packages vs. filesystem images● Requirement for cloud to remain

functioning after removing installation media and rebooting

● Tools + modular approach, not one size fits all

32

Were there community benefits?

● High Availability○ Proved HA can be automated to simplify deployment○ Identified bugs in Pacemaker+Corosync stack

● Reliability enhancements○ Identified and fixed OpenStack race conditions

● Provoked healthy discussions○ Deployment is (still) important!

Looking forwardAustin Summit

34

Define Rule the Stack for Austin!

Join us for a Birds-of-a-Feather discussion to shape our next competition

Wednesday, October 2811:45 AM - 12:30 PM

Design Summit Lounge

35

Ideas for future competitions

● Keep deployment but de-emphasize speed● Challenges based on ongoing operations● Live upgrade● HA for compute nodes / guest instances

36

Participating in the Intel Passport Program?Are you playing? Be sure to get your Passport Stamp for attending this session! See me or my helper in the back at the end!

Not Playing yet? What are you waiting for? See me or my helper in the back at the end and we can get you started!

Don’t forget to return your stamped passport to the Intel Booth #H3 to enter our raffle drawing! 3 Stamps = 1 Raffle Ticket