Rock Stars, Builders, and Janitors: You're Doing It Wrong, New Relic [FutureStack16]

Confidential ©2008-15 New Relic, Inc. All rights reserved.

Rockstars, Builders and Janitors:You’re doing it wrong!

Alice Goldfuss, Site Reliability Engineer, New Relic

Confidential ©2008-15 New Relic, Inc. All rights reserved. 2

Rockstars JanitorsBuilders@alicegoldfuss

You’re doing it wrong


This document and the information herein (including any information that may be incorporated by reference) is provided for informational purposes only and should not be construed as an offer, commitment, promise or obligation on behalf of New Relic, Inc. (“New Relic”) to sell securities or deliver any product, material, code, functionality, or other feature. Any information provided hereby is proprietary to New Relic and may not be replicated or disclosed without New Relic’s express written permission.

Such information may contain forward-looking statements within the meaning of federal securities laws. Any statement that is not a historical fact or refers to expectations, projections, future plans, objectives, estimates, goals, or other characterizations of future events is a forward-looking statement. These forward-looking statements can often be identified as such because the context of the statement will include words such as “believes,” “anticipates,” “expects” or words of similar import.

Actual results may differ materially from those expressed in these forward-looking statements, which speak only as of the date hereof, and are subject to change at any time without notice. Existing and prospective investors, customers and other third parties transacting business with New Relic are cautioned not to place undue reliance on this forward-looking information. The achievement or success of the matters covered by such forward-looking statements are based on New Relic’s current assumptions, expectations, and beliefs and are subject to substantial risks, uncertainties, assumptions, and changes in circumstances that may cause the actual results, performance, or achievements to differ materially from those expressed or implied in any forward-looking statement. Further information on factors that could affect such forward-looking statements is included in the filings New Relic makes with the SEC from time to time. Copies of these documents may be obtained by visiting New Relic’s Investor Relations website at ir.newrelic.com or the SEC’s website at www.sec.gov.

New Relic assumes no obligation and does not intend to update these forward-looking statements, except as required by law. New Relic makes no warranties, expressed or implied, in this document or otherwise, with respect to the information provided.


• Built new features / tools / systems

Scaled existing onesCleaned up existing ones

In the past 3 months, have you mainly:


I’m Alice

5

SRE @



• Define the rolesExamine the current

environmentDiscuss a better way


Rockstar


Rockstar StrengthsCurious and open-minded

Can iterate fastAlways looking to the next thing


Rockstar WeaknessesGet bored easilyNo patience for maintenance

Not used to thinking about end user experience


Builder

11


Builder StrengthsUnderstand scaleValue reliabilityGood at catching “what ifs” that others miss


Builder WeaknessesCan move slow while trying to build out

Have hard time letting things go

Can be resident doom-sayers


Janitor

14


Janitor StrengthsUnderstands the weaknesses in your systems

Really good at bandaids

Able to do a lot with a little


Janitor WeaknessesCan often resort to cleaning manually instead of automating

Often gets stuck down in the weeds

Resistant to change


Rockstars

Janitors

Builders


Rockstars

Janitors

Builders


Organic == Good?


GMO


GMO

Greater Mobility, Okay?


• They get stuck


• They get stuckThey get tunnel vision


Current State of Tech



117 responses


47.9% Currently Rockstars


55.2% Want to be Rockstars


19.7%

CurrentlyBuilders


35.3%Want to beBuilders


32.5% CurrentlyJanitors


9.5%

Want to beJanitors


47.9%

19.7%

32.5% Current State

Rockstars

Builders

Janitors


55.2% 35.3%

9.5%

Preferred State

RockstarsBuilders

Janitors


70.9%

70.9% of Rockstarsare Devs


58.3%

58.3% ofJanitorsare Ops


54.4%54.4% ofBuildersare Devs


55.0%

55% who want to be Builders are Ops


70.9%

58.3%

54.4%

Dev OpsDev

Rockstars Builders Janitors


26.3% 26.3% thinkwork distributionon team is unfair


49.1%

49.1% thinkwork distributionin company is unfair


people unhappy / not doing the work they want


can try to make something like reliability a first class citizen, but people confuse that with not changing anything, so it’s a tough sell

Focus on Reliability


can try rotating responsibilities across teams, but some teams will be better at some tasks than others, and this will evolve back into an imbalanceRotate

Teams


what to do?

What to do?


Ursula Le Guin


The Left Hand of Darkness


“No physiological habit is established, and the mother of several children

may be the father of several more.”


“Burden and privilege are shared out pretty equally; everybody has the same risk to run or choice to make. ”


Have everyone do everything


what??



That’s not what this is



Remove the Blinders


Build better56


Foster new skills

57


Rotate



4 engineers


1 week janitor1 week builder2 weeks rockstar


50.0% 25.0%

25.0% Proposed State

Rockstar

Builder

Janitor


55.2% 35.3%

9.5%

Preferred State

RockstarsBuilders

Janitors


Technical debt


Velocity!



Productivity =

Size(TechDebt)Size(System)1 -

Griffith, C. Izurieta, H. Taffahi, and D. Claudio, “A simulation study of practical methods for technical debt management in agile software development,” in Proceedings of the 2014 winter simulation conference, Piscataway, NJ, USA, 2014, pp. 1014-1025. http://www.gregerwikstrand.com/technical-debt-reduction/


case study


Decommissioned 1,139 applications in 2013 alone

Reduced systems 7,000 to 4,200 in three years

69

“The Digital Transformation Symphony: When IT and Business Play in Sync” http://www.slideshare.net/capgemini/the-digital-cio

http://www.slideshare.net/capgemini/the-digital-cio


Reduced debt by €14 billion

“The Digital Transformation Symphony: When IT and Business Play in Sync” http://www.slideshare.net/capgemini/the-digital-cio

http://www.slideshare.net/capgemini/the-digital-cio


Where to start?


Ops team


firefighting


pile of sustaining work


pile of sustaining work

Don’t punish your ops team for your dev team’s

SHIT TY CODE


• What are you worried about?



What could we do better?



What could we do better?What do you want to add?


devs overburdened


devs overburdened

GOOD




Rockstar tasksNew feature / tool / system

Research new projects

Demo mockups


Builder tasksImprove testingImprove service resiliency

Understand greater impact


Janitor tasksBug fixesRefactoringConfig cleanup





What could we do better?



What could we do better?What do you want to add?


47.9%

19.7%

32.5% Current State

Rockstars

Builders

Janitors


• Define the roles




environment


47.9%

19.7%

32.5% Current State

Rockstars

Builders

Janitors


55.2% 35.3%

9.5%

Preferred State

RockstarsBuilders

Janitors



environmentDiscuss a better way



50.0% 25.0%

25.0% Future State

Rockstar

Builder

Janitor


Thanks!

98

@alicegoldfuss


This document and the information herein (including any information that may be incorporated by reference) is provided for informational purposes only and should not be construed as an offer, commitment, promise or obligation on behalf of New Relic, Inc. (“New Relic”) to sell securities or deliver any product, material, code, functionality, or other feature. Any information provided hereby is proprietary to New Relic and may not be replicated or disclosed without New Relic’s express written permission.

Such information may contain forward-looking statements within the meaning of federal securities laws. Any statement that is not a historical fact or refers to expectations, projections, future plans, objectives, estimates, goals, or other characterizations of future events is a forward-looking statement. These forward-looking statements can often be identified as such because the context of the statement will include words such as “believes,” “anticipates,” “expects” or words of similar import.

Actual results may differ materially from those expressed in these forward-looking statements, which speak only as of the date hereof, and are subject to change at any time without notice. Existing and prospective investors, customers and other third parties transacting business with New Relic are cautioned not to place undue reliance on this forward-looking information. The achievement or success of the matters covered by such forward-looking statements are based on New Relic’s current assumptions, expectations, and beliefs and are subject to substantial risks, uncertainties, assumptions, and changes in circumstances that may cause the actual results, performance, or achievements to differ materially from those expressed or implied in any forward-looking statement. Further information on factors that could affect such forward-looking statements is included in the filings we make with the SEC from time to time. Copies of these documents may be obtained by visiting New Relic’s Investor Relations website at ir.newrelic.com or the SEC’s website at www.sec.gov.

New Relic assumes no obligation and does not intend to update these forward-looking statements, except as required by law. New Relic makes no warranties, expressed or implied, in this document or otherwise, with respect to the information provided.


1 https://www.flickr.com/photos/lydur/22006675124/1 https://www.flickr.com/photos/qilin/49135690116 https://commons.wikimedia.org/wiki/File:St_Anger_Album_Artwork.png22 https://www.flickr.com/photos/tamakisono/1490306821940 https://www.flickr.com/photos/46799485@N00/447076140943 https://www.flickr.com/photos/christinagallivan/8660375888/44 Photo by Jack Liu, 201445 https://queerscifi.com/for-readers-the-new-pronouns/49 https://www.flickr.com/photos/greencolander/320392221150 https://upload.wikimedia.org/wikipedia/commons/e/ee/Woodwards_building_Vancouver_demolition_2.jpg

Image credits

https://upload.wikimedia.org/wikipedia/commons/e/ee/Woodwards_building_Vancouver_demolition_2.jpg


53 https://www.flickr.com/photos/3059349393/8000156483/54 https://www.flickr.com/photos/dalbera/4923032712/56 https://www.flickr.com/photos/moonlightbulb/8025527585/57 https://www.flickr.com/photos/dafnecholet/537420094862 https://www.flickr.com/photos/alebonvini/475765119963 https://www.flickr.com/photos/dave_see/1513268418464 https://www.flickr.com/photos/diversey/1623358150/70 https://www.flickr.com/photos/expertinfantry/71 https://www.flickr.com/photos/adambelles/5044281763/72 https://www.flickr.com/photos/cogdog/9090732482/77 https://www.flickr.com/photos/twalmsley/554571234179 https://www.flickr.com/photos/athomeinscottsdale/3279949186/80 https://www.flickr.com/photos/modofodo/14964870008/

Data & Analytics

Rock Stars, Builders, and Janitors: You're Doing It Wrong, New Relic [FutureStack16]