USING SALTSTACK TO DEVOPS THE ENTERPRISEChristian McHugh
About me
● Just zis guy, you know?● Been working professionally since 1998 in systems
administration/ops and management● Been involved in DevOps work and advocacy since 2010● Author of saltstack jenkins plugin
Agenda
● Enterprise concerns● Articulate the argument● Tool use
Enterprise Terminology
● Service● Service Level Agreement● Configuration Management Database
Demo!
● Deploy: salt-run state.orchestrate orch.stackdeploy
● Scale: ab -t 600 -n 1000000 -c 100 192.168.122.197/
● Patch: cli version: salt-run state.orchestrate orch.patch-web 'pillar={"target":"web1"}'
● Report: salt '*' coalfire.report --output newline_values_only > /tmp/report.csv
Demo: Landscape/service deploy● salt-run state.orchestrate orch.stackdeploy
○ web1: http://192.168.122.184/○ balance1: http://192.168.122.197/○ balance1 stats: http://192.168.122.197:8998/admin?stats
Demo: scaling● ab -t 600 -n 1000000 -c 100 192.168.122.197/
○ https://efhdevops.slack.com/messages/salttest/
Demo: Patching● cli version: salt-run state.orchestrate orch.patch-web 'pillar=
{"target":"web1"}'
Demo: Reporting
● salt '*' coalfire.report --output newline_values_only > /tmp/report.csv
Req 1 Debian Debian-8 Application saltmaster 127.0.1.1 DC1 8.2 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65...
Req 2 Debian Debian-8 Application node1 192.168.122.213 DC2 8.2 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65...
Mission Accomplished?
● Move from demo to deployment● Multiple teams involved
○ VM deployment team○ Storage○ Windows Operations (Wintel)○ Unix○ Networking○ Security○ Compliance○ App team○ Dev team○ Architecture○ Patching team○ Testing team○ Help desk (update cmdb)○ DevOps team
● *** Management buy-in ***● Cost/Benefit analysis
○ Costs for people to do work today?
■ New project spin up time
■ Employee hours patching
■ Security response
■ Compliance auditing
■ Track ticket buckets for where time spent
○ Cost to bring down service for maintenance?
○ Cost of existing duplicated tools (inventory management, runbooks, patching tools)
○ Cloud?
○ Overhead of app deployment
■ Features and bug fixes are the differentiating value add
Informs tool availability and work priority
What’s required?
Value Stream Map
Request VM Create Base Build Done
Send email: 5minHelpdesk rework request: 1 day
VM Build: 1 hour Firewall Conf: 2 weeksAdd User: 1-7 daysInstall Role: 7 days
Wait: 2 weeks
Wait: 2 weeks
1 day 1 hour 22 - 28 days
Wait Time:28 days
Work Time: 23 - 29 days
Total Time: 51 - 57 days
You don’t have to boil the ocean (all at once)
Have kids? Ever managed people? Does what works for one child or employee always work for everyone?
Processes also need to adapt to real world usage
Don’t start with silo removal, just ease the communication and current workflows
Perhaps you set up syndic or just gitfs and let silo team manage most of their config. Task enterprise architecture team to set base, and allow teams to layer on top.
How Important is Management Engagement?
“The most critical success factor to deploying DevOps and transitioning into a high performing organization, is changing the mindset from siloed operations to enabling continuous delivery” -- Gartner
DevOps is not a project, it is a culture of Continuous Improvement
Talk Business
When engaging management, keep scope in mind. Tell a story of how recommendations will affect business outcomes
Not great● Makes patching better
Good● Reduces patching times by 90%
Best● Reduces patching hours from 330 to 3 resulting in $13,200 of monthly cost
reduction
Communicate with management
Know your/their terminologyAsk about what is important to themSpeak in terms they understand
Should you talk big picture, or lay out the individual steps? Perhaps you describe fully managed services, or you talk about individual metric improvements
What gets measured gets improvedKeep them collectable, actionable, and auditable.Operational
● Headcount● Cost of change/release● Cost of incident/security response (patching)● Mean time to recover (MTTR)
Service Velocity● Deployment Frequency● Deployment lead time
Customer Value● Response Time● Epics Delivered
Business Value● Time to New Service● Time to report (inventory, sox, pci, security…)
Simple Goals
Keep things simple. Implement something and iterate.It’s okay to be wrong. Don’t let egos take over.Don’t rush off to implement solutions; discuss and get agreement on what you are trying to accomplish.
● *** Management buy-in ***● Communicate
○ Get stakeholders into the same room (teams from before)
○ Ensure management attends meetings○ Post constant progress updates/reminders○ Ensure signoff on all decisions (roman vote)○ Brown bag sessions
■ Demos, examples, progress reports■ Record low hanging fruit
● Determine end state● Quick wins
○ Build toward end goal, but keep deliverables to days
What else is required?
So Salt?
Salt is super modular. Use it for code deploys or full stack orchestration.
Write modules to handle current paint points: reporting, one-shot jobs, repetitive tasks, job scheduling
Once in place, proselytize the event bus
Create an interface for required functionality and delegate the authority. Use salt ACL system or frontend like Jenkins
Continuous Improvement - bite sized
● Use Salt to delegate capability○ Patching
■ Develop remote execution workflow■ Give patching team a jenkins button
Continuous Improvement - bite sized
● Use Salt to delegate capability○ Patching
■ Develop remote execution workflow■ Give patching team a jenkins button
■ Develop service definition● SLA/OLA● Outage windows● $ of outage
■ Develop service monitoring■ Develop service tests■ Safely implement automatic patching
In short
Ensure everyone is serious1. Get & keep management involved2. Get players in the same room: DevSecNetCompArchStorOps
○ Cross functional team with champions3. Communicate constantly: end goal, training, status, pain points, desires, low
fruit4. Determine current costs5. Develop priority list6. Rock it, one quick item at a time
The End
Questions?Thoughts?Disagreements?
Please rate and provide feedback via the conference app/site
Why use salt?
● Simplicity (simple to use, but powerful)● Approachability for non programmers● Messaging layer● Compliance
What worked well
Salt-virt and salt-cloud
Zabbix monitoring
Testinfra
Salt for CD
Autoscailing loadbalancers: salt events and mine
Salt-proxy for firewall management
Salt elasticsearch reporting