Upload
steve-feldman
View
777
Download
0
Embed Size (px)
DESCRIPTION
Citation preview
LOGONOMICS: The Hidden Side of Blackboard Logs by steve feldman @PerfForensics
Logging Doesn’t Suck
It’s Like Fishing in the Night…
So Why Don’t We Talk About Logs More OJen?
At least 20% of all people in this room don’t know where to find their logs.
At least 50% of all people in this room don’t look at their logs.
At least 60% of all people in this room don’t visualize their log data.
At least 75% of all people in this room don’t correlate data between logs.
At least 90% of all people in this room don’t standardize the management of
logs to a centralized service.
At least 95% of all people in this room don’t alert IT staff based on a specific
log event.
If a System Doesn’t Output to a Log Do We Assume Nobody is Using it?
If a System ConZnuously Spews Data to a Log Do We Ignore it?
What We Can Do With Our Log Data LOGONOMICS: The Hidden Side of Blackboard Logs
Trending and Intelligence
Service Levels
Threats and VulnerabiliZes
Responsiveness Reliability
Primer Data Points Everyone Should Know
Unique Requests
Time Series of Requests
ConcentraZon of Request Types
Origin of Requests
Quick Averages
Cascading Issues Across Logs
Combining Other Data with Log Data
CorrelaZon
Root Cause
InterpretaZon
CompleZon of Message
Full Picture Sequence and Timelines
Types of Data We Can Get LOGONOMICS: The Hidden Side of Blackboard Logs
Business AnalyZcs: AdopZon and Growth
System Health
Capacity Planning
Security and Threat Analysis
Quality and Experience: MeeZng SLAs
Replay and Benchmarking
Insight into the BbLogs LOGONOMICS: The Hidden Side of Blackboard Logs
Four Horseman of Logs
Bablefield of Other Logs
• AuthenZcaZon • Plugins Directory • NauZlus for events • Monitoring (System Logs) – Syslogs and Rsyslogs (/var/messages) – Windows Event Logs
Is there a Most Important Log?
Access Log
Log Formafng Mabers Log Levels (INFO, WARN, ERROR)
mod_log_forensic
Use %k, %T and %D
Decompose the URI
Log Formafng Mabers
Is there a 2nd Most Important Log?
Tomcat and Java Logs
Stack Traces Startup OpZons
GC Events
GC Pauses and Status
Tools We Should Consider LOGONOMICS: The Hidden Side of Blackboard Logs
It’s All About the Right Fishing Rod
CAT!
GREP!
TAIL!
SED!AWK!
SORT!
GROK!
SomeZmes a Net is Beber to Cast
Log CentralizaZon
Please Take All My Logs
Format Lots of Log Data
Send it Down the River
• amqp • exec • file • gelf • redis • stdin • stomp • syslog • tcp • twiber • xmpp • zeromq
• amqp • elasZcsearch • elasZcsearch_
river • file • ganglia • gelf • graphite • internal • loggly • mongodb • nagios
• date • dns • gelfify • grep • grok • grokdisco
very • json • mulZline • mutate • split
• null • redis • statsd • stdout • stomp • tcp • websocket • xmpp • zabbix • zeromq
Inputs Filters Outputs
Configure Apache for JSON log
• hbp://cookbook.logstash.net/recipes/apache-‐json-‐logs/
Configure Tomcat for MulZ-‐Line Filter
Setup Bb to feed logstash
What We Use Logstash
Log AggregaZon
Non-‐FuncZonal Requirements
Event NoZficaZon
IntegraZon with Zabbix
Kibana Front-‐End Redis Inputs & Outputs
Indexing
Simple Challenge to All
• Setup Logstash architecture (All Single Node) • Start shipping basic log files – Apache 2.X access log or IIS web server log – Tomcat Catalina log file
• Output results to statsD (Etsy Project) – Simple Use Case: IncremenZng HTTP codes (200, 300, 400)
• Visualize statsD data with Graphite
Bonus Challenge to All
• Take the Vagrant VM and integrate Logstash shipper with configuraZon files.
• Add Postgres support (Development Only) • Basic syslog funcZonality for CentOs • Custom Log Interface for a B2
Let’s Add-‐on to the IniZaZve developer.blackboard.com