© Neustar, Inc. 1

Todd Minnella

San Diego Web Performance Meetup

March 7, 2012

Musings on Web

Performance from

an Ops Guy

About Todd

© Neustar, Inc. 2

» Over 25 years doing

IT operations/support

» Experience with both

academic and enterprise computing

» Currently, managing the Systems Operations

team for Webmetrics/BrowserMob at Neustar

» Favorite operating system is Tru64

» Loves solving problems!

Agenda

© Neustar, Inc. 3

»Foundations of Web Performance

»Getting to Stability

»Building High Performing Systems

»High Quality Operations

Start with the Basics . . .

© Neustar, Inc. 4

Fast

Available

Useful Service

Useful Site/Service

© Neustar, Inc. 5

Does your site work?

Available

© Neustar, Inc. 6

Is your site running?

Fast

© Neustar, Inc. 7

Is your site performing well?

Useful Site/Service

© Neustar, Inc. 8

If your application or service

isn’t useful for customers, or

doesn’t work well

Talk to your Product, QA or

Engineering teams!

Getting to Stability

© Neustar, Inc. 9

Solve for Stability First

© Neustar, Inc. 10

»Track problems that have operational impact

»Triage these problems, and focus on the BIG

challenges

»Be willing to question the way things are –

ask why a lot!

Be Transparent Inside and Out

© Neustar, Inc. 11

»Share your problems with interested/affected

stakeholders

»Start with internal, and share to external

customers (if you can)

»Problem reports can help build righteous

outrage and get you help

A Few Troubleshooting Thoughts

© Neustar, Inc. 12

» Get to know what normal looks like

» Understand your dependencies (and dependents)

» Know your limits (and when you reach them)

» Consider the effects of high load and latency

» Make sure high availability/fault tolerant features

aren’t making things worse

Building High Performing

Systems

© Neustar, Inc. 13

Design for Performance

© Neustar, Inc. 14

»Know your requirements

» If you’re involved at the design phase, strive

for simplicity

»Don’t optimize too early (get the application

running first)

»Consider your customers’ bottlenecks

High Performing Web Apps

© Neustar, Inc. 15

»Test and monitor at production scale where

possible

»Monitor production thoroughly

»Watch for changes!!!

»Be aware of changes in your hosting

ecosystem

A Performance Problem

© Neustar, Inc. 16

» Planned migration of WM UI to new hardware . . .

Performance Improved!

© Neustar, Inc. 17

» And the results of modifying our load balancer profile:

And a Caution . . .

© Neustar, Inc. 18

» Know the changes affecting your monitors:

» Stay abreast of changes from your monitoring vendor(s)!

High Quality Operations

© Neustar, Inc. 19

Ops Philosophy

© Neustar, Inc. 20

»Manage change (but don’t quash it)

»Make decisions based on data

»Be transparent and honest

»Take ownership of issues

»Be a customer

Suggested Reading

© Neustar, Inc. 21

Web Operations

John Allspaw and Jesse Robbins

The Visible Ops Handbook

Kevin Behr, Gene Kim & George Spafford

The Practice of System and Network Administration

Thomas A. Limoncelli, Christina J. Hogan & Strata R. Chalup

Complete Web Monitoring

Alistair Croll & Sean Power

Any of the Short Topics in System Administration series

Published by the USENIX Association

Suggested Tools

© Neustar, Inc. 22

Cacti

http://www.cacti.net/

Splunk

http://www.splunk.com/

Wireshark

http://www.wireshark.org/

QuickBase

http://quickbase.intuit.com/

ServiceNow

http://www.service-now.com/

© Neustar, Inc. 23

Questions

and

Open Discussion