Embed Size (px)
DESCRIPTION
Measure anything, measure everything. Effortless monitoring with Statsd, Collectd and Graphite can increase software development productivity and quality at the same time.
Citation preview
"Monitoring as Software Validation"
Measure anything,Measure everything
Serena [email protected]
Incontro DevOps Italia
Bologna, 21 Feb. 2014
Monitoring:If it moves... you can track it!
Monitor everything
Network Machine Application
Why?●Learn from your
infrastructure●Anticipate failure●Speed up changes
Metrics and Events
Metric: Time + Name + ValueEvent: Time + Name
It can be anything
Graphite
An all-in-one solution for storing and visualizing real-time
time-series data
Key features:Efficient storage and ultra-fast retrieval.Easy!!
http://graphite.wikidot.com/
Graphite
Graphite Web
The front-end of Graphite. It provides a dashboard for retrieval and visualization of our metrics and a powerful plotting API.
Graphite components
Carbon
The core of Graphite. Carbon listens for data in a format, aggregate it and try to store it on disk as quickly as possible using whisper.
Whisper
The data storage. An efficient time series based database.
Organization of your data
Everything in Graphite has a path with components delimited by dots. servers.hostname.metricapplications.appname.metric
Paths reflect the organization of the data:
Pushing in your data:Carbon configuration (and limitations)
Carbon listens for data (1) and aggregates them (2). One can set the two specific behaviors by changing appropriate variables in the configuration files.
1) How often your data will be collected? It needs to have theretention time set to a specific value.For a timespan X I want to store my data at intervals of y (seconds/hours/days/months). What happens if I send two metrics at the same time? Carbon retains only the last one!
2)How do your metrics aggregate? It needs specific keywords to apply functions to aggregate the data (e.g., “min”, “max”, “sum”..).
import statsd
HOST = 'hostname.server.com'PORT = 8181PREFIX = 'myprefix'
def initialize_client(host, port, prefix): client = statsd.StatsClient(host, port, prefix) return client
def send_data(data_name, value, client): client.gauge(data_name, value)
client = initialize_client(HOST, PORT, PREFIX)
…..CODE.....
send_data('Energy', 1000, client)
Fast and flexible monitoring: StatsD
StatsDFront-end application for
Graphite (by Etsy)Buffers metrics locallyAggregates the data for
us Flushes periodically data to GraphiteClient libraries available in any languageSend any metric you like
https://github.com/etsy/statsd/
Data Types in StatsD
Graphite usually stores the most recent data in 1-minute averaged timestep, so when you’re looking at a graph, for each stat you are typically seeing the average value over that minute.
Type Definition Example
Counters Per-second rates Page views
Timers Event duration Page latency
Gauges Values How many views do you have
Sets Unique values passed to a key
Number of registered users accessing your website
CollectDA unix daemon that gathers system statistics
Plugin to send metrics to CarbonVery useful for system metrics
Fast and flexible monitoring: CollectD
Application-level statistics:StatsD
e.g. The number of times a function is called
System-level statistics:CollectD
e.g. the memory usage
We can combine them in a dashboard!
Case study:“Company A”
A project not testing friendly ... ...The Design phase was almost skipped!
We were asked to translate an existing (Matlab!) application (into Python)
Metrics Driven Development!
Case study:“Company A”
Task: exploring a space of solutions to find the best one
Method:Simulated annealing
ProbabilityRandom Number
Track the evolution of the process instead of parsing a (boring) log file to (1) correlate the consequences of
having P(x) > random number and (2) visually inspect the real-time changing of P(x) values during the simulation
Metrics Driven Development!
Case study:“Company B”
A project where multiple applications have to interact in order to manage the elaboration of a
huge number of pictures every day
Case study:“Company B”
Monitor to …1) see the asynchronous activation of the applications2) gather a regular pattern3) CHECK FOR CHANGES IN THAT PATTERN!
Monitor your system (cpu, ram...) and applications together to see
if the hardware suits their requirements or not
Case study:“Company B”
Monitor your system (cpu,ram...) and
applications together to see if the hardware suits their
requirements or not.E.g. picture upload time
Vs packet received/transmitted Vs memory free/used
and so on...
Database queries per second?Async tasks currently in queue?
How is the application behaving?Images resized and stored?
Error and warning rates?
Case study:“Company B”
Case study:“Company B”
These applications are running on several hosts and
their metrics end to the same point.
You can monitor many different servers bylooking at the same dashboard.
Testing and Monitoring
"measure twice, cut once"-
"Cut it quickly in several pieces and see which fits best (now!)”
You can do both!
Testing: just once during the development Monitoring: it keeps working once the application is
released
Testing and Monitoring
Tests are logical properties of our application. Metrics are not. But Metrics
offer you the possibility to see what is going on once the application/system is in
production
inevitableFailure is not accepted
and detectable!
Monitoring
✗Provide informations
✗Frequent communication
✗Some share decision making
Free!
Dev Ops
Wait... I don't like Graphite Web Interface!
No problem! The world of the interfaces is
In continuous evolution
About 56,100 results
You can't optimize what you can't measure
so monitor and...
Optimize anything,Optimize everything
