Upload
john-schneider
View
1.455
Download
0
Embed Size (px)
DESCRIPTION
@rossbruniges and I talked with our fellow Mozilla Foundation engineers and development teams about getting the data for building a data driven operation using statsd, graphite, geckoboard, google analytics, and newrelic.
Citation preview
Why measure anything?
We want to continually improve, to further the MoFo mission.
Humans, and also organizations, can't know if they can or should do something differently
unless they get feedback or data to prompt the question:
"How can we/I do better at this?"
Absent of any data, everything looks fine
First Requirement of Improvement
Realize there is something that needs improvement
__________________
Knowing the truth is often scary and uncomfortable, but at least you know when to run!
Fast, Confident, and Awesome
If an org, team, or individual:
-Knows, definitively, how they are doing now
-Knows, definitively, how they are doing after a change
-Knows, definitively, what variables changed when
Then that org, team, or individual:
-will be much faster to determine success vs. failure, making it possible to resolve faster.
-will inevitably grow and learn, further and faster
-generates more trust...with users, contributors, org, team
and maybe most importantly...
-We'll be more confident to try more things, knowing we can quickly detect and resolve.
-Thus, as we will try more things, it will lead more often to success.
Had Good Metrics Indicating Dogs Were Tame...Confident.
What kind of stats are we talking?
Not just CPU and App Performance!
metrics << app / productmetrics << projects / teams
metrics << users / customersmetrics << contributorsmetrics << operations
metrics << performancemetrics << developers
metrics << support
Proposed Layers of Metrics
1. Organization Layer - "How is Mozilla Foundation doing at its mission?"
2. Team Layer - "How well is this team working together and accomplishing goals? "
3. Application/Service Layer - "How are things with X application or service?"
4. Operations Layer - "How are we doing, logistically?"
Organizational Metrics ExamplesHow many visits to all of our sites this year vs. last year?
How many people have donated to MoFo, and how much?
How many times were we or any of our sites mentioned on social media and the news?
How many new "webmakers" did we bring into our ecosystem across all applications and teams?
Team Metrics Examples
How many hours were worked/PTO'd on the team?
How many projects is that team involved in this week?
How many collaborator / volunteer / staff lines of code / features were added?
How many milestones did the team meet this month?
Application Metrics Examples
What was the average response time for popcorn.webmaker.org this week?
How many videos were edited this week?
How many new videos were created this week?
What was the # of page visits for popcorn on average by hour?
What was the ratio of errors to successes for popcorn's key transactions this week?
Operations Metrics ExamplesHow much disk is currently being used?
How much memory and CPU utilization do we have in production?
How many gigabytes of logs were generated this week?
What was the slowest loading page per application this week?
How much did our infrastructure cost?
What tools will we be using?
Primary Tools● Newrelic● StatsD/Graphite● Google Analytics● Geckoboard
Secondary Tools (Not covering today)● AWS, Pingdom, Opsview
New Relic (Server Monitoring)
New Relic
Real time data on your server and app performanceExamples:● what code is running slowly or erroring?● how long are end users waiting before they
can interact with our content?● How many database writes per minute?● End user performance monitoring ● Which deploy caused a problem?
Google Analytics
Google Analytics
Numbers on who is using your app, how and when they are using it
● what content is popular with who?● which parts of the site are causing people to
leave?● where are our users coming from?● what things have they done while on our
sites?● Can track campaigns, goals (ie, made a
popcorn video), conversion rates
statsd/graphite
Multiple stats, one stream
statsd/graphite
statsd - "Collects data by increment counters and timers, but does not show, sort, or average(etc....)"
graphite - Shows the data in a presentable way, allowing you to line up multiple metrics and perform calculations, such as sort, avg, std deviation
(Examples of a few things we might count/time)● someone made a project, remixed a project● someone has been issued a badge● someone has decided to delete their profile
StatsD examples (counting new vids in popcorn)(note the data structure, env.app.what)
Time something (PHP example)$start = microtime(true);postvideo();StatsD::timing("production.butter.postvideo.response.200", (microtime(true) - $start) * 1000);
Count something (PHP)StatsD::increment("production.butter.postvideo.response.200");
Geckoboard
Geckoboard
A friendly view into all of our high level stats.
● this is where Mark/Ryan will come in● this is where we highlight the individual
metrics we get judges against● one stop-shop over all MoFo/webmaker
properties● encouraged to make public
Metrics Flow
Things to Beware Of
● Presenting data out of context for audience● Instrumentation of anything to New Relic,
Google Analytics, Geckoboard should go over ssl
● Do not use New Relic auto instrumentation to capture any params in the database
● Gameable Metrics
Go fourth and metric!
See what you can do with these client libs:
● statsd:○ https://django-statsd.readthedocs.org/en/latest/○ https://github.com/sivy/node-statsd
● geckoboard:○ http://pypi.python.org/pypi/django-geckoboard○ https://github.com/danjenkins/node-geckoboard-
push ○ https://github.com/Hebo/node-geckoboard
● newrelic custom instrumentation:○ https://newrelic.com/docs/instrumentation/custom-
metric-collection
Links/ReferencesGood read with a further intro for statsd/graphiteHow we will setup Opsview to watch StatsD/GraphiteGoogle Analytics: Setting up goals and tracking conversions
Easy setup instructions for statd/graphite server(if you want to test locally)