65
DevOps Paul Nasrat · http://nasrat.livejournal.com · @nasrat Friday, 9 July 2010

Dev ops ljc 2010

  • View
    2.286

  • Download
    2

Embed Size (px)

DESCRIPTION

Talk from London Java Community 2010

Citation preview

Page 1: Dev ops   ljc 2010

DevOpsPaul Nasrat · http://nasrat.livejournal.com · @nasrat

Friday, 9 July 2010

Page 2: Dev ops   ljc 2010

about:me

Friday, 9 July 2010

Page 3: Dev ops   ljc 2010

DevOps == Culture + Tools

Friday, 9 July 2010

Page 4: Dev ops   ljc 2010

Culture

Friday, 9 July 2010

Page 5: Dev ops   ljc 2010

Velocity

Friday, 9 July 2010

Page 6: Dev ops   ljc 2010

DevOpsDays

Friday, 9 July 2010

Page 7: Dev ops   ljc 2010

10 Deploys A Day

• John Allspaw

• Paul Hammand

Friday, 9 July 2010

Page 8: Dev ops   ljc 2010

Development versus Operations

Friday, 9 July 2010

Page 9: Dev ops   ljc 2010

Spock ScottyLittle bit weird

Sits closer to the bossThinks too hard

Pulls levers & turns knobsEasily excitedYells a lot in emergencies

John Allspaw - Velocity 2009Friday, 9 July 2010

Page 10: Dev ops   ljc 2010

http://www.flickr.com/photos/albinoflea/195857838/Friday, 9 July 2010

Page 11: Dev ops   ljc 2010

Hero Culture

http://www.flickr.com/photos/ohrphan/999704273Friday, 9 July 2010

Page 12: Dev ops   ljc 2010

Silos

http://www.flickr.com/photos/zoomzoom/304135268/Friday, 9 July 2010

Page 13: Dev ops   ljc 2010

Trust and Respect

Friday, 9 July 2010

Page 14: Dev ops   ljc 2010

What’s my job again?

Friday, 9 July 2010

Page 15: Dev ops   ljc 2010

Keep sh*t running?

Friday, 9 July 2010

Page 16: Dev ops   ljc 2010

Keep sh*t running?

Friday, 9 July 2010

Page 17: Dev ops   ljc 2010

change

Friday, 9 July 2010

Page 18: Dev ops   ljc 2010

68%Friday, 9 July 2010

Page 19: Dev ops   ljc 2010

32%Friday, 9 July 2010

Page 20: Dev ops   ljc 2010

£Friday, 9 July 2010

Page 21: Dev ops   ljc 2010

All of this has happened

before, and it will happen

again...

http://en.wikipedia.org/wiki/File:Cylon_Centurion.jpg

Friday, 9 July 2010

Page 22: Dev ops   ljc 2010

• Theory of Constraints

• Lean

• Toyota Production System

• Agile

Friday, 9 July 2010

Page 23: Dev ops   ljc 2010

Tools

Friday, 9 July 2010

Page 24: Dev ops   ljc 2010

Automated Infrastructure

Friday, 9 July 2010

Page 25: Dev ops   ljc 2010

• EC2/S3

• Heroku

• Appengine

• Eucalyptus

• VMware

Cloud as Inspiration

Friday, 9 July 2010

Page 26: Dev ops   ljc 2010

Storage• mogilefs

• S3

• NFS

• HDFS

Friday, 9 July 2010

Page 27: Dev ops   ljc 2010

REST

CRUD + curl == :)

Friday, 9 July 2010

Page 28: Dev ops   ljc 2010

Infrastructure = Code

Friday, 9 July 2010

Page 29: Dev ops   ljc 2010

• Provisioning & Imaging OS

• Roles & Configuration

Friday, 9 July 2010

Page 30: Dev ops   ljc 2010

#fail

Friday, 9 July 2010

Page 31: Dev ops   ljc 2010

http://www.flickr.com/photos/lifeinelgin/2533204950/Friday, 9 July 2010

Page 32: Dev ops   ljc 2010

http://www.flickr.com/photos/rustie/472315037/

Friday, 9 July 2010

Page 33: Dev ops   ljc 2010

Share

Friday, 9 July 2010

Page 34: Dev ops   ljc 2010

Shared Version Control

Friday, 9 July 2010

Page 35: Dev ops   ljc 2010

Shared Domain

Friday, 9 July 2010

Page 36: Dev ops   ljc 2010

Shared Measurements

Friday, 9 July 2010

Page 37: Dev ops   ljc 2010

Twitter - sysadmin 2.0

“Data analysis, visualization, and other techniques for seeing patterns in data are going to be an increasingly valuable skill set. Employers take notice!”“Web Squared: Web 2.0 Five Years On”, Tim O’Reilly, Web 2.0 Summit, 2009

Friday, 9 July 2010

Page 38: Dev ops   ljc 2010

Uptime

90.0% 36 days, 12 hours

95.0% 18 days, 6 hours

99.0% 87 hours, 36 minutes

99.50% 43 hours, 48 minutes

99.90% 8 hours, 48 minutes

99.99% 52 minutes, 33 seconds

99.999% 5 minutes, 15 seconds

99.9999% 32 seconds

Friday, 9 July 2010

Page 39: Dev ops   ljc 2010

Numbers you should know - Jeff Dean

L1 cache reference 0.5 ns

Branch mispredict 5 ns

L2 cache reference 7 ns

Mutex lock/unlock 25 ns

Main memory reference 100 ns

Compress 1K bytes with Zippy 3,000 ns

Send 2K bytes over 1Gbps network 20, 000 ns

Read 1 MB sequentially from memory 250, 000 ns

Round trip within same datacenter 500, 000 ns

Disk seek 10,000,000 ns

Read 1 MB sequentially from disk 20,000,000 ns

Send packet CA->.NL->CA 150,000,000 ns

http://bit.ly/ladis09Friday, 9 July 2010

Page 40: Dev ops   ljc 2010

http://www.flickr.com/photos/altemark/273968506Friday, 9 July 2010

Page 41: Dev ops   ljc 2010

public class QuickStart {

public static void main(String[] args) { NagiosSettings settings = new NagiosSettingsBuilder() .withNagiosHost("nagiosHostNameOrIPAddress") .withPort(5667) .withEncryption(Encryption.XOR) .create(); MessagePayload payload = new MessagePayloadBuilder() .withHostname("hostname of machine sending check") .withLevel(Level.OK) .withServiceName("Service Name") .withMessage("should work if everything set up OK") .create(); NagiosPassiveCheckSender sender = new NagiosPassiveCheckSender(settings); try { sender.send(payload); } catch (NagiosException e) { e.printStackTrace(); } catch (IOException e) { e.printStackTrace(); } }}

http://code.google.com/p/jsendnsca/

Friday, 9 July 2010

Page 42: Dev ops   ljc 2010

Performance Test

Friday, 9 July 2010

Page 43: Dev ops   ljc 2010

Capacity Plan

Friday, 9 July 2010

Page 44: Dev ops   ljc 2010

Friday, 9 July 2010

Page 45: Dev ops   ljc 2010

• Scribe

•Hadoop

• Solr

Log everything

Friday, 9 July 2010

Page 46: Dev ops   ljc 2010

Log Setting servlet

private List<Logger> sortLoggers(Enumeration<Logger> currentLoggers) { List<Logger> loggers = new ArrayList<Logger>(); while(currentLoggers.hasMoreElements()) { loggers.add(currentLoggers.nextElement()); }

Collections.sort(loggers, new Comparator<Logger>(){ @Override public int compare(Logger logger1, Logger logger2) { return logger1.getName().compareTo(logger2.getName()); } });

return loggers; }

Friday, 9 July 2010

Page 47: Dev ops   ljc 2010

Continual deployment

Friday, 9 July 2010

Page 48: Dev ops   ljc 2010

IMVU

50Friday, 9 July 2010

Page 49: Dev ops   ljc 2010

Continuous Deploy

•CI is the backbone

• Simple Deploy

• Simple Revert, Roll forward

• Alerting

•Monitoring

Friday, 9 July 2010

Page 50: Dev ops   ljc 2010

wordpress.com

http://toni.org/2010/05/19/in-praise-of-continuous-deployment-the-wordpress-com-story/

Friday, 9 July 2010

Page 51: Dev ops   ljc 2010

kaching

• SOA

• ZooKeeper

• Deployment Manager

• Inflight releae limit - 2

Friday, 9 July 2010

Page 52: Dev ops   ljc 2010

Deploy at scale

• Bittorrent

• Incremental deploys

• Simple deployment tool

Friday, 9 July 2010

Page 53: Dev ops   ljc 2010

Application

Configuration

Data

Friday, 9 July 2010

Page 54: Dev ops   ljc 2010

Read Only Mode

Friday, 9 July 2010

Page 55: Dev ops   ljc 2010

dbdeploy

Friday, 9 July 2010

Page 56: Dev ops   ljc 2010

Configuration Lifecycle

Friday, 9 July 2010

Page 57: Dev ops   ljc 2010

http://www.flickr.com/photos/ahnmyrrh/3620801348/Friday, 9 July 2010

Page 58: Dev ops   ljc 2010

package com.gu.management.switching;

public interface Switchable extends SwitchState { void switchOff(); void switchOn();

...}

http://github.com/tackley/guardian-management

Friday, 9 July 2010

Page 59: Dev ops   ljc 2010

@Singletonpublic class GuiceSwitchboardServlet extends SwitchboardServlet { @Inject public GuiceSwitchboardServlet(Collection<Switchable> switchables) { super(switchables); }}

Friday, 9 July 2010

Page 60: Dev ops   ljc 2010

Dark Launcheshttp://www.flickr.com/photos/25035545@N04/2477315249/

Friday, 9 July 2010

Page 61: Dev ops   ljc 2010

Bucket Testing

Friday, 9 July 2010

Page 62: Dev ops   ljc 2010

Communications

Friday, 9 July 2010

Page 63: Dev ops   ljc 2010

http://github.com/RJ/irccat

Friday, 9 July 2010

Page 64: Dev ops   ljc 2010

Further Info

• agile-system-administration Google Group

• http://londondevops.org/

• devops-toolchain Google Group

Friday, 9 July 2010

Page 65: Dev ops   ljc 2010

Projects

• http://github.com/tackley/guardian-management

• http://github.com/lg/murder

• http://code.google.com/p/jsendnsca/

• http://puppetlabs.com/

• http://github.com/RJ/irccat

• http://hadoop.apache.org/

• http://github.com/facebook/scribe

Friday, 9 July 2010