Embed Size (px)
Citation preview
@petecheslock
Everything @obfuscurity Taught Me About Monitoring
Monitorama 2016
@petecheslock
“Just use Graphite”- Jason Dixon
@petecheslock
Thank you
@petecheslock
Pete Cheslock @petecheslock Tech Ops & Support at Threat Stack
@petecheslock
get.threatstack.com/monitorama16
@petecheslock
constantly
@petecheslock
It is an AMAZING time to be working in technology
@petecheslock
The “Good ‘ole Days” kinda sucked
@petecheslock
@petecheslock
@petecheslock
@petecheslock
@petecheslock
@petecheslock
@petecheslock
I knew how to cd and ls
@petecheslock
# chown -Rf bob:bob files /
Whoops
@petecheslock
Should've used bcfg2?
<Path name="/home/file" mode="0644" group="bob" owner="bob"> <ACL type="access" scope="user" user="bob" perms="rwx"/></Path>
Won’t exist for 3+ more years…
@petecheslock
@petecheslock
ColdFusion
@petecheslock
* * * * * /scripts/mrtg-netmon
@petecheslock
@petecheslock
@petecheslock
@petecheslock
**PROBLEM Host Alert:Everything Is FUCKED**
@petecheslock
@petecheslock
@petecheslock
Jen: Yo, are you ready to support ~10 beta customers on a completely unautomated, unmonitored env?
Pete: New Phone, who dis?
Jen: 5 9s of availability.
@petecheslock
The Right Way™
@petecheslock
@petecheslock
@petecheslock
@petecheslock
@petecheslock
http://blog.packagecloud.io/eng/2016/04/05/the-definitive-guide-to-linux-system-calls/
@petecheslock
@petecheslock
Premature optimization is the
root of all evil
@petecheslock
@petecheslock
@petecheslock
@petecheslock
collectd -> write_http ->(60 Second)
@petecheslock
@petecheslock
@petecheslock
@petecheslock
@petecheslock
collectd -> write_http ->(60 Second)
statsd plugin
app1 app2 appN
@petecheslock
@petecheslock
• Hosted TSDB is super useful and “just works”
• The opportunity cost of fully replacing it was too high.
@petecheslock
• Use the same tools already deployed
• Devs wrote libraries to send metrics to statsd
• Don’t want to “redeploy the world”
• Limited time to invest
@petecheslock
• Use the same tools already deployed
• Devs wrote libraries to send metrics to statsd
• Don’t want to “redeploy the world”
• Limited time to invest
@petecheslock
collectd -> write_http ->(60 Second)
collectd -> write_graphite ->(10 Second)
statsd plugin
statsd plugin
@petecheslock
@petecheslock
https://github.com/obfuscurity/synthesize
https://github.com/hw-cookbooks/graphite
@petecheslock
https://github.com/obfuscurity/synthesize
https://github.com/hw-cookbooks/graphite
@petecheslock
@petecheslock
Want your Devs to Ops?
Build Consumable Services
@petecheslock
@petecheslock
@petecheslock
@petecheslock
collectd -> write_graphite ->(10 Second)
statsd_plugin
app1 app2 appN
@petecheslock
A lot of people must have suddenlyjust stopped caring
@petecheslock
Community Matters
Relationships Matter
@petecheslock
You are (probably) not Google
@petecheslock
and that is OK(Solve YOUR Problems)
get.threatstack.com/monitorama16
Thank You!
