SHOWDOWN: Threat Stack

vs. RedHat AuditD

By Jen Andre, Co-Founder Threat Stack

We like magic.

But since magic isn’t real, we have to come up with the next best thing to make our agent the best in its class.

Savvy operations and security people that use our service are blown away by the types of

information the Threat Stack agent can collect, correlate, and analyze

from Linux servers.

“I’ve tried to do this with (Red Hat) audits with little to no success …how do you guys do it?” they say.

The Linux audit subsystem is a very powerful way to collect information about system calls

and other security-relevant activity.

The best part…

No kernel module is required to enable this detailed level of auditing

since it’s built right into Linux!

You can write a log anytime a particular system call happens, whether that be unlink or getpid.

And since the auditing operates at such a low level, the granularity

of information is incredibly useful.

Traditionally, people have used the user land daemon ‘auditd’ built by some good Red Hat folks to collect and consume this data.

However, there are a couple of problems with traditional open source auditd and auditd libraries

that we’ve had to deal with ourselves…

…especially when trying to run it on performance sensitive systems and make sense of the obtuse

data that traditional auditd spits out.

To that effect, we’ve written a custom audit listener from the ground up for the Threat Stack agent

(tsauditd)

This is what makes Threat Stack’s audit listener special…

1. Performance Enhancements

Many people have tried to use traditional Red Hat audits in production to do very detailed auditing of user,

process, and network syscall activity, but have failed due to the performance impact.

We’ve ensured that our agent is responsible with resource utilization

through our unique parsing model.

While benchmarking a web server, we saw auditd consume 120% of

the CPU! !

Threat Stack’s agent CPU consumption

was only 10%!

2. Output Enhancements

This Linux audit system will output many different lines across disparate events into syscall, which you then have to correlate later via your ingestion

engine or your log management system…

The key-value format is also cumbersome to parse,

and values are often encoded into hex randomly…

We’ve decided that all related events should be grouped together and have conveniently

parsed everything correctly for you.

We then transformed that into a JSON output format that is much

simpler to read and parse.

3. Network tooling (“src/dst port”)

Tracking network connections across multiple hosts can be a manual and painful process when

trying to connect across boxes.

To make it easier, our agent adds metadata to network connection events to determine where the connection is

originating from and where it is going.

Our backend then correlates these network events to determine the originating process and potential user activity

that caused that network event, so long as the agent lives on both the source and destination server.

This is especially useful for tracking SSH sessions

across your environment and debugging what servers are speaking to one another and why.

4. User Activity Auditing

Digging around the server logs to see where a user on your system went is not an easy job.

You’d need to manually find the agent and session that a user connected to, yet all the kernel gives us is a nasty hex

encoding string representing the connection address in the traditional auditd logs!

On top of that, most of the information logged by auditd is not really relevant, and

hard for the human eye to parse.

To correct that…

We’ve designed Threat Stack to keep storage of events, activity, and commands associated with a logged in user, and automatically reconstruct this information into a

clean, compact, and readable timeline.

Stay tuned for more engineering feats we are accomplishing at Threat Stack…

To learn more about Threat Stack or to sign up, visit

threatstack.com