Mentoring Devs Into DevOpsJustin Carmony Director of Development Deseret Digital Media

Lets Measure The Audience

• Who here is a…

• System Administrator?

• Developer?

• Manager / Management?

• “DevOp?”

Confession:I’m a Developer

Self-Taught OpsBecause There Was No One Else To Do It

About Me• Director of Development

for Deseret Digital Media

• Utah PHP UsergroupPresident

• I Make (and Break) Web Stuff (10 years)

• Salt User in Production since 0.8

(I <3 Salt)

This Presentation• Lessons learned at DDM & previous jobs

• Insight into our process of increasing “DevOps”

• We’re still learning, but this what we’ve found.

• Slides will be posted online, so don’t worry about copying slide content.

• Feel free to ask on-topic questions during, and we’ll have questions at the end.

About DDM

• Deseret Digital Media runs local website like KSL.com, DeseretNews.com

• Running National and International Websites like OK.com, familia.com.br, etc.

• ~10 million pageviews a day across sites.

• ~150 internal VMs, a few dozen physical machines, some AWS sprinkled around.

Lets Start With a Story!

You Work for an Awesome Tech Company

Team Is Working Hard to Build New Things!

You launch your awesome product!

A Few More Features…

… and next thing you know…

Awesome Job Team, We Rock!

We Need !Real-Time XYZ Feature!

ASAP!

We Need !Real-Time XYZ Feature!

ASAP!

We Need !Real-Time XYZ Feature!

ASAP!

&#$%!

“Huh, it works if you ! just turn off caching…”!

- Dev @ 80th Hour This Week

“I’m sure this ! will work…”

“Our servers are melting!”

“We Need a Better Solution!”

So…

Where Do We Start?

We Have This Problem

Challenges We Faced

• Giant mesh-up of technologies

• Tightly-coupled & fragile infrastructure

• Debugging production only bugs was difficult

• Bugs that were part code, part environment were a nightmare to track down.

So One Day… We Had A Genius Idea!

Lets Hire a DevOp!

I’m Not JokingWe Actually Said This

Two Problems with this “Idea”

Problem #1 - We Didn’t Understand What We Really Wanted

Step 1: Hire a DevOp!

Step 2: ????????????!

Step 3: Profit!

Step 1: Hire a DevOp!

Step 2: ????????????!

Step 3: Profit! Everything Works !Perfectly!

Problem #2 - People Who Are Great At Dev & Ops Are Hard To Find

Expectation:

Reality:

Honest Team Discussion: What is it we’re really looking for?

We Discovered a Few Things

What does DevOps Mean To Us?

• DevOps: Dev & Ops, a Culture of Collaboration

• Our Goal: “10 deploys a day without issues”

• Everyone shares the goal of quick development of features AND a stable system that stays up.

Team StructureDevs: 30 Ops: 2

Team StructureDevs: 30 Ops: 2

DevOps: 1

Team StructureDevs: 30 Ops: 2

DevOps: 1

Team StructureDevs: 30 Ops: 2

DevOps: 1

Hiring one person won’t just solve all our problems!

Team Realizations

• Hardest problem already solved: awesome team

• No foreseeable rapid expansion, must operate at our current scale

• Each Project’s Director of Development was acting as the bridge between Dev and Ops, but would become a bottleneck.

Teams Already Had Some Ad-Hoc DevOps Tools

- Real-time Logging - Capistrano Deploys

- Nagios Alerts - Server Metrics - Puppet for File Mgmt

- App Stats w/ Graphite - Graphite Dashboards - Salt for Cfg Management

- Homebrewed Metrics Sys. - Homebrewed Alert System

Teams Already Had Some Ad-Hoc DevOps Tools

- Real-time Logging - Capistrano Deploys

- Nagios Alerts - Server Metrics - Puppet for File Mgmt

- App Stats w/ Graphite - Graphite Dashboards - Salt for Cfg Management

- Homebrewed Metrics Sys. - Homebrewed Alert System

Step 1: Hire a DevOp!

Step 2: ????????????!

Step 3: Profit! Everything Works !Perfectly!

Step 1: Hire a DevOp!

Step 2: ????????????!

Step 3: Profit! Everything Works !Perfectly!

We Formed A Strategy

Step #1: Promote Dev to DevOp Role

WAIT!Isn’t that the advice you just

said was a bad idea?!

DevOp Engineer

• Well Defined Role:

• Ownership over the TOOLS to improve DevOps efforts.

• Resource for other teams to help use DevOps Tools.

• Easy to work with, aptitude for systems & ops, likes to try new things.

Promoting From Within

• A seasoned dev for your team already knows:

• Your Pain Points

• Your System’s Quirks

• How the “Chaos Works”

• Knows the people & personalities on your team

Step #2: Change Team Structure

Team StructureDevs: 30 Ops: 2

Team StructureDevs: 30 Ops: 2

Team StructureGoal: Spread Out Expertise By Increasing

Ops Experience & Skills Among Devs

Dev Ops

Team StructureGoal: Spread Out Expertise By Increasing

Ops Experience & Skills Among Devs

Dev Ops

Team Structure

Dev Ops

Team Structure

Dev Ops

Increasing Ops Among Devs

• Identify Devs who liked “Ops” & wanted to Learn

• Pair Dev with Op / Director

• Learning Dev works on things, not Op /Director.

• Pair program if needed.

Step #3: Increase Everyone’s Insight

Step #3: Increase Everyone’s Insight

Metrics• Everyone has access to Network, Server, and

Application Metrics.

• Consolidate & reduce places to look. We try to pipe everything to StatsD / Graphite

• Each developer trained to add & track metrics in production.

• We’re okay with 98% uptime of stats to avoid complexity.

Real-Time Logging

Real-Time Logging• Harder & more complicated at scale

• Still trying to solve well, we have lots of logs.

• Start with small window of data (i.e. 48 hours) and start to expand window.

• We’re trying Logstash, ElasticSearch, and Kibana right now.

• Generate Statistics off our Logs

Tracking Changes

• Everything, everything, everything in git (we use GitHub)

• Everyone has access to all repos • Everyone does work through Pull Requests • Everyone has their work code reviewed *

* - Your can merge w/o a review, but must be willing to defend your choice

Deploys

Everyone Can Deploy

• Automated our deployment process to a single step.

• Everyone can deploy, deployments are logged

• Easy rollback is a requirement!

• Implementing feature flags to turn off single parts of our application.

TestsUnit

Functional Integration

Acceptance etc

Automated Tests

• If you want to trust your Devs, you need tests

• Legacy apps we wrote Integration Tests

• New Apps & Refactored Legacy Parts have Unit Tests

• Continuous Integration to make sure tests run

So, where’s the salt?

Step #4: Devs Use The Ops Tools

Devs can grok salt

Safe Environment For Devs to Learn

Safe Environment For Devs to Learn

salt \* cmd.run "rm -rf /tmp /*"

Safe Environment For Devs to Learn

salt \* cmd.run "rm -rf /tmp /*"

Salt is awesome, but it can’t !recover from that

Dev Salt MasterDevs Can Look Into Every Server

Dev Salt Master• Every server has two minions:

• Admin Salt (aka root)

• Dev Salt (aka bob)

• Each connect to different master server:

• All Devs have access to Dev Salt Master

• Trusted Devs get access to Admin Salt Master

Everything Salty in GitReminder:

Dev Environment

• Developers own the Dev Environment

• Dev Teams manage the Salt States for their Env

• Vagrant + Salt for their Env

• Who makes changes? Developers

• DevOp helps advise & offer support

Team Structure

Dev Ops

Stage Environment• Stage & Production use same salt repos, different

branches

• Developers make all the changes for Application Servers

• All Changes through Pull Requests

• We’ll worry about env changes before code

• Small changes we quickly release, large or long running branches are scary & dangerous

Production Environment

• Merge change to Production Branch

• salt \* state.highstate

• Reminder: Small quick changes over time, never a large change at once.

Environment Caveats• Ops & DevOps Manage VM Hosts, Physical Load

Balancers, FireWalls, etc

• Ops & DevOps manage servers that deal with data:

• MySQL

• MongoDB

• etc

Mentoring Devs

Mentoring Devs

• Not every Dev will become an amazing DevOp

• Thats okay!

Level of “DevOps” Skills

• Thinks about their impact on Ops: Everyone

• Able to debug issues with production: Most

• Able to make changes to environments: Many

• “Awesome DevOp”: Some

Team’s “DevOps” Skills

0

25

50

75

100

Think Debug Change DevOp

Current Goal

So Everything Is Awesome for us, right?

Honesty Slide: We Have Skeletons In Our Closets

Where We Are At

• All Dev Environments using Vagrant + Salt

• All New Stage & Prod Environments are Salty

• Some Legacy Stage & Production Envs are Salty

• Continuously working on getting out stuff salty.

Making This Work For Your Team

Honest Introspection• Determine for your

team what are your… • Strengths • Weaknesses • Problems • Goals

Increase Team’s Insight • Make sure devs can see

& understand how their code performs

• Increase responsibility of team for those metrics.

• If they break it, they fix it. Do not always bail them out.

• Everyone can see everything.

Increase Team’s Insight • Make sure devs can see

& understand how their code performs

• Increase responsibility of team for those metrics.

• If they break it, they fix it. Do not always bail them out.

• Everyone can see everything.

Mentor Those With Desire / Aptitude

• Give Developers Safe Environment to Learn

• Let them submit code-reviewed changes for Stage & Production

• When teaching / mentoring, let the learner drive, kindly offer advice and help.

• It takes time, but worth the investment.

A Few Final Thoughts

Team Culture Matters

Positive Influence

Questions?

Thank YouJustin Carmony

Email: justin@justincarmony.comTwitter: @JustinCarmony

IRC: carmony #salt #uphpuWebsite: justin@justincarmony.com

p.s. we’re hiring, email / pm / tweet me