Using SaltStack to DevOps the enterprise

USING SALTSTACK TO DEVOPS THE ENTERPRISEChristian McHugh

About me

● Just zis guy, you know?● Been working professionally since 1998 in systems

administration/ops and management● Been involved in DevOps work and advocacy since 2010● Author of saltstack jenkins plugin

Agenda

● Enterprise concerns● Articulate the argument● Tool use

Enterprise Terminology

● Service● Service Level Agreement● Configuration Management Database

Demo!

● Deploy: salt-run state.orchestrate orch.stackdeploy

● Scale: ab -t 600 -n 1000000 -c 100 192.168.122.197/

● Patch: cli version: salt-run state.orchestrate orch.patch-web 'pillar={"target":"web1"}'

● Report: salt '*' coalfire.report --output newline_values_only > /tmp/report.csv

Demo: Landscape/service deploy● salt-run state.orchestrate orch.stackdeploy

○ web1: http://192.168.122.184/○ balance1: http://192.168.122.197/○ balance1 stats: http://192.168.122.197:8998/admin?stats

Demo: scaling● ab -t 600 -n 1000000 -c 100 192.168.122.197/

○ https://efhdevops.slack.com/messages/salttest/

Demo: Patching● cli version: salt-run state.orchestrate orch.patch-web 'pillar=

{"target":"web1"}'

Demo: Reporting

● salt '*' coalfire.report --output newline_values_only > /tmp/report.csv

Req 1 Debian Debian-8 Application saltmaster 127.0.1.1 DC1 8.2 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65...

Req 2 Debian Debian-8 Application node1 192.168.122.213 DC2 8.2 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65...

Mission Accomplished?

● Move from demo to deployment● Multiple teams involved

○ VM deployment team○ Storage○ Windows Operations (Wintel)○ Unix○ Networking○ Security○ Compliance○ App team○ Dev team○ Architecture○ Patching team○ Testing team○ Help desk (update cmdb)○ DevOps team

● *** Management buy-in ***● Cost/Benefit analysis

○ Costs for people to do work today?

■ New project spin up time

■ Employee hours patching

■ Security response

■ Compliance auditing

■ Track ticket buckets for where time spent

○ Cost to bring down service for maintenance?

○ Cost of existing duplicated tools (inventory management, runbooks, patching tools)

○ Cloud?

○ Overhead of app deployment

■ Features and bug fixes are the differentiating value add

Informs tool availability and work priority

What’s required?

Value Stream Map

Request VM Create Base Build Done

Send email: 5minHelpdesk rework request: 1 day

VM Build: 1 hour Firewall Conf: 2 weeksAdd User: 1-7 daysInstall Role: 7 days

Wait: 2 weeks

Wait: 2 weeks

1 day 1 hour 22 - 28 days

Wait Time:28 days

Work Time: 23 - 29 days

Total Time: 51 - 57 days

You don’t have to boil the ocean (all at once)

Have kids? Ever managed people? Does what works for one child or employee always work for everyone?

Processes also need to adapt to real world usage

Don’t start with silo removal, just ease the communication and current workflows

Perhaps you set up syndic or just gitfs and let silo team manage most of their config. Task enterprise architecture team to set base, and allow teams to layer on top.

How Important is Management Engagement?

“The most critical success factor to deploying DevOps and transitioning into a high performing organization, is changing the mindset from siloed operations to enabling continuous delivery” -- Gartner

DevOps is not a project, it is a culture of Continuous Improvement

Talk Business

When engaging management, keep scope in mind. Tell a story of how recommendations will affect business outcomes

Not great● Makes patching better

Good● Reduces patching times by 90%

Best● Reduces patching hours from 330 to 3 resulting in $13,200 of monthly cost

reduction

Communicate with management

Know your/their terminologyAsk about what is important to themSpeak in terms they understand

Should you talk big picture, or lay out the individual steps? Perhaps you describe fully managed services, or you talk about individual metric improvements

What gets measured gets improvedKeep them collectable, actionable, and auditable.Operational

● Headcount● Cost of change/release● Cost of incident/security response (patching)● Mean time to recover (MTTR)

Service Velocity● Deployment Frequency● Deployment lead time

Customer Value● Response Time● Epics Delivered

Business Value● Time to New Service● Time to report (inventory, sox, pci, security…)

Simple Goals

Keep things simple. Implement something and iterate.It’s okay to be wrong. Don’t let egos take over.Don’t rush off to implement solutions; discuss and get agreement on what you are trying to accomplish.

● *** Management buy-in ***● Communicate

○ Get stakeholders into the same room (teams from before)

○ Ensure management attends meetings○ Post constant progress updates/reminders○ Ensure signoff on all decisions (roman vote)○ Brown bag sessions

■ Demos, examples, progress reports■ Record low hanging fruit

● Determine end state● Quick wins

○ Build toward end goal, but keep deliverables to days

What else is required?

So Salt?

Salt is super modular. Use it for code deploys or full stack orchestration.

Write modules to handle current paint points: reporting, one-shot jobs, repetitive tasks, job scheduling

Once in place, proselytize the event bus

Create an interface for required functionality and delegate the authority. Use salt ACL system or frontend like Jenkins

Continuous Improvement - bite sized

● Use Salt to delegate capability○ Patching

■ Develop remote execution workflow■ Give patching team a jenkins button

Continuous Improvement - bite sized

● Use Salt to delegate capability○ Patching

■ Develop remote execution workflow■ Give patching team a jenkins button

■ Develop service definition● SLA/OLA● Outage windows● $ of outage

■ Develop service monitoring■ Develop service tests■ Safely implement automatic patching

In short

Ensure everyone is serious1. Get & keep management involved2. Get players in the same room: DevSecNetCompArchStorOps

○ Cross functional team with champions3. Communicate constantly: end goal, training, status, pain points, desires, low

fruit4. Determine current costs5. Develop priority list6. Rock it, one quick item at a time

The End

Questions?Thoughts?Disagreements?

Please rate and provide feedback via the conference app/site

Why use salt?

● Simplicity (simple to use, but powerful)● Approachability for non programmers● Messaging layer● Compliance

What worked well

Salt-virt and salt-cloud

Zabbix monitoring

Testinfra

Salt for CD

Autoscailing loadbalancers: salt events and mine

Salt-proxy for firewall management

Salt elasticsearch reporting