Cashing In on Logging and Exception Data

Jason TaylorCTO, Stackify

“Logging is hard”

-No one, ever.

3

Logging: not hard, but not perfect

The Easy• Libraries: log4net, nLog, Elmah,

Enterprise Library• Very easy to integrate and use• Great for debug

The Not Easy• Production log file access• Volume of data * Number of

servers• Rotation and retention• Flat and hard to query• Getting enough context• Proactively consuming /

monitoring

4

Today’s topicsLog All The Things

Building a “culture of logging” to always have relevant, contextual logs that don’t add overhead.

Make the most of your logging framework.

Work Smarter, Not Harder

Using proper tooling to consolidate & aggregate all of your logging to be available to everyone that needs it, quickly consume it, and proactively put it to use.

5

What things? • Timestamp• Thread ID• Transaction ID• Server• Environment• User Data• Post Data• Cookies• Querystring• Etc, etc

If you’re going to take the time to log something, log enough to give you a play-by-play picture of events.

When?• For every important business transaction

Start Simple – handle all exceptions

Pros ConsElmah -Easy

-Lots of storage possibilities-Web page to view errors-Limited email & notification possibilities

-Unhandled only-Advanced usage requires a lot of configuration.-If MVC, don’t forget to grab Elmah.MVC-Don’t leave your errors web page exposed in producton!

Global.asax (Application_Error)-Use the framework of your choice-Captures all exceptions-Fully customizable

-You are responsible for returning state back to the method that called (e.g. a controller action) or redirecting

Try/Catch blocks -Try/Catch/Finally generally perceived as a good pattern-You will catch 100% of the exceptions that you “catch”

-You will miss 100% of the exceptions that you don’t “catch” -Can be “error prone” if you are only catching a certain type that doesn’t happen-Time consuming to “retro fit” into an app

Exception Filters (MVC) -Like using global.asax, very customizable -You have to implement everything, including errors that happen globally, how to log, etc

Adding ContextTreat logs like “debug when you can’t attach”

• Use verbosity levels appropriately – allows you to “turn up” and “turn down” the amount of logging as necessary

• Identity key processes and functions that need to be audited and logged• Capture the key steps, measure success / failure• Capture contextual data that helps you solve an issue or answer a question (i.e. logged in

user, Http Request data)

Don’t forget about logging in your client code• Choose a library that is smart enough to bridge server

and client code while keeping context

8

Adding Context - Basic

Bad Slightly Better

9

Adding Context - Advanced• Logged the input parameters• Logged the object created in my

DbContext• Had to use a custom appender for

log4net that knows how to serialize the objects

Alternatively, would have to serialize to a string:

log.Debug(string.Format("Creating a foo: {0}",JsonConvert.SerializeObject(foo)));

Adding Context – Power User!!Wouldn’t it be nice to:• Always capture the user who generated the log or exception?• Always capture basic request details?

This would allow us to have deeper, more meaningful data to search through our logs.

Adding Context – Power User!!Context Properties (log4net)

<log4net> <root> <level value="ALL" /> <appender-ref ref="StackifyAppender" /></root><appender name="StackifyAppender" type="StackifyLib.log4net.StackifyAppender, StackifyLib.log4net"> <logicalThreadContextKeys>User,Request</logicalThreadContextKeys> </appender></log4net></configuration>

Adding Context – Power User!!void MvcApplication_AuthenticateRequest(object sender, EventArgs e) {

try { log4net.LogicalThreadContext.Properties["User"] = User; } catch (Exception ex) { log.Error(ex); } }

The Results: Rich Logs and Errors

Tips & Tricks• Use your verbosity levels• Make sure your logging framework is asynchronous• Log heavily is core / shared services & other high fan-in areas• Always log the “who”• Use patterns like AOP to write enty / exit logging once, apply

everywhere• Make it a part of your development culture (don’t just write the logs –

USE them)

Working SmarterAt Stackify, we set out to make our logging and error monitoring smarter…

…it became a core part of our product offering.

Working SmarterLogs & Errors Tell (Some of) the Story• Logs as “breadcrumb trail” are good for:

• Capturing errors in raw form • Capturing key events (depending on

logging level)• Piecing together the story (if you logged it) on

one server at a time

• Error monitoring is good for:• “After the fact” notification• Seeing just the error in question• Pointing you back at log files for more info

Working SmarterThe Trouble With Your Current Logs:• Working with all those log files is tedious and time consuming• No cross-server / cross-app correlation• No trends / telemetry• File retention & rotation• Intermixed with Exception details

Working SmarterThe Trouble With Your Errors:• Logged errors usually are no more than exception and stack trace• No cross-server/cross-app tracking for errors• Missing/hard to get request details• No trends / telemetry• Need intelligent notifications• In Error Monitoring apps,

disconnected from Logs!

19

Indexing:• Search across apps and servers w/o managing log files• Fully indexed “Context” details• Real-time log tailing• Narrow scope by time range

Telemetry• Visualize trends in logging and exceptions

• Across all levels of granularity• Contextual telemetry – for any app or server, by default

Alerts• Notify when new or “first chance” exceptions occur• Notify on error regression• Notify when error rates are abnormal• Notify on any log query we can think of

Engage• Provide access to those who can help solve issues• Give everyone the same picture, with no extra effort

Architecture• Single platform solution• Relate exceptions to relevant logs for maximum context• Contribute to picture of “application health”

Live demo

Or….

How good is the conference WI-FI?

Search! Quickly filter!

Log tailing!

Context!

Full error details and telemtryFull application health details

23

Full search

Error Rates

Error Details

24

25

Benefits of an integrated logging / error platform• Get real telemetry, know how well your app behaves from release to

release• In context to overall performance• Use in QA environments, look for new exceptions and abnormal rates

before it is released to prod• Proactively hunt down bugs & improve your codebase• Know about an issue before your customers tell you• Quicker, more efficient debugging via logs