OSSEC in the Enterprise - Immutable Security

Michael Starks 2009 Immutable Security http://www.immutablesecurity.com

OSSEC in the Enterprise

Open Source Log Management, Analysis and Intrusion Detection

Rochester Security SummitOctober 29, 2009

Michael Starks, CISSP, CISA, GSNA


Agenda

What is OSSEC?Log AnalysisIntegrity MonitoringRootkit DetectionPolicy MonitoringAlertingActive ResponseOSSEC WebUI

Why OSSEC?Risks & CountermeasuresEnterprise ConsiderationsDemoQuestions


What is OSSEC?

OSSEC is an Open Source Host-based Intrusion Detection System. It performs log

analysis, file integrity checking, policy monitoring, rootkit detection, real-time alerting

and active response.

Source: http://www.ossec.net


What is OSSEC?

Put another way...

OSSEC is security software that looks for bad stuff on the actual host


Multi-Platform

Works on Windows and most Unix-like systems


Centrally Managed

Client/server architecture

Almost everything can be managed from the OSSEC manager

Restart agentsStart integrity checks

Tune rulesBlock attacks


Single Installation

Manager and agent on one machine


Distributed

Centralized manager and distributed agents


Distributed

Multiple managers and multiple agents


Redundant

Fail over to one or more managers


Flexible and Extensible

Easily add support for custom applications

Integrate with commercial SIEMs

Analyze logs on existing syslog servers


Secure by Default

Privilege separated processes

Chroot where possible

Secure programming practices

Encrypted message transport using IP restrictions and replay prevention


Supported

Community

IRC: #OSSEC on Freenode

Mailing lists:

ossec-list

ossec-dev

www.ossec.net

Commercial

Trend Micro

OSSEC Host-Based Intrusion Detection Guide


Fast and Efficient

Analyze millions of events per day

...in real-time

...using commodity hardware


Extensive Application Support

Dozens of decoders and hundreds of rules out of the box

Unix Pam, sshd (OpenSSH), Solaris telnetd, Samba, Su, Sudo, Proftpd, Pure-ftpd, vsftpd, Microsoft FTP server, Solaris ftpd, Imapd, Postfix, Sendmail, vpopmail, Microsoft Exchange, Apache, IIS5, IIS6,

Horde IMP, Iptables, IPF. PF, Netscreen, Cisco PIX/ASA/FWSM, Snort, Cisco IOS, Nmap, Symantec AV, Arpwatch, Named, Squid,

Windows event logs, VMWare


Free

Open source

Budget friendly


Log Analysis

The heart of OSSEC


LIDS

Log-based Intrusion Detection

Not a log management tool

Analyzes (but does not store) every log


A Slight Detour

What if the attacker deletes the logs?

Will you have all the pieces of the puzzle?

Robust log management strategies help OSSEC do its job


Log Management

Corporate policy should define the need for logging


Log Management

Corporate standards should define system audit settings, such as:

What to auditFrequency of log rotationLog formatMethod of communication


Log Management

Logs should, wherever possible, be converted from a proprietary format to a standardized

and normalized format (e.g. syslog)


Log Management

Logs should be centralized and stored on ahardened, purpose-specific server, with nounnecessary or unrelated services running


Log Management

Systems should be synchronized with a common, trusted time source


Log Management

Logs contain sensitive information and should be encrypted in transit wherever possible


Log Management

A copy of each log should be available both locally and centrally

In the event of a compromise, the trusted log server can be compared with the local logs


Log Management

Logs should be maintained online and archived offline according to

regulatory or policy requirements


Log Management

Access to logs should be on a need-to-know and least-privileged basis


Log Management

Access to logs should always be read-only


Log Flow Through OSSEC

Tree-like structure

Alert

Analysis

Decode

Pre-decode

Log enters system


Log Enters System

Secure (encrypted)

Insecure (syslog)

Localhost


Pre-Decoding and Decoding

Extracts individual parts of the log and places them into “buckets”

Useful later on when writing rules

Bob172.16.3.4

528 nsa.gov

user src_ip idurl


SSHd Log Pre-Decoded

Extracts known fields from logs (e.g. time) Compiled in for efficiency

Log comes in as:Apr 14 17:32:06 hostname sshd[1025]:

OSSEC pre-decodes it as:time/date -> Apr 14 17:32:06hostname -> hostnameprogram_name -> sshd

Pre-decoded


SSHd Log Fully Decoded

Log comes in as:Apr 14 17:32:06 hostname sshd[1025]: Accepted password for root from 192.168.2.190 port 1618 ssh2

OSSEC decodes it as:time/date -> Apr 14 17:32:06hostname -> hostnameprogram_name -> sshd

log -> Accepted password for root from 192.168.2.190 port ...srcip -> 192.168.2.190user -> root

Pre-decoded

Decoded


SSHd Log Decoder

<decoder name="sshd"><program_name>^sshd</program_name></decoder>

<decoder name="sshd-success"><parent>sshd</parent><prematch>^Accepted</prematch><regex offset="after_prematch">^ \S+ for (\S+) from (\S+) port </regex><order>user, srcip</order></decoder>

Will there be a test?


Analysis (Rules)

Rules are also called signatures

Simple XML files on the manager

Independent of original log format


Two Types of Rules

Atomic: single event

Bob mistyped his password once

Composite: multiple events across logs

Bob mistyped his password 3,561 times in 3 minutes

on 16 different systems


That Looks Suspicious

I know Bob forgets his password, but...


Rules

Rules pick up where decoders leave off

Instead of writing rules for raw logs, they can be written to normalized data

(e.g. “Bob” is a “user”)

Data flows through the tree until a rule matches or doesn't match


Rules

Severity-based: levels 0 (low) to 15 (high)

Nest multiple rules for granular control

Rule groups further normalize data

●web_scan●firewall_drop●account_changed...


Simplest Rule

If the log was decoded as SSHd, generate rule 111

Not very useful yet

<rule id = "111" level = "5"><decoded_as>sshd</decoded_as><description>Logging every decoded sshd message</description></rule>


Dependent Rule

If rule 111 matched and the log contains “Failed Password”

set the severity (level) to 7 and the group to “authentication_failed”

<rule id=”122” level=”7”><if_sid>111</if_sid><match>^Failed password</match><description>Failed password attempt</description><group>authentication_failed</group></rule>


2nd Dependent Rule

If rule 122 matchedand it's that pesky Bob

Raise the severity (level) to 12

<rule id=”133” level=”12”><if_sid>122</if_sid><user>Bob</user><description>That pesky Bob again</description></rule>


In Other Words

Put another way...

Record all events decoded as SSHd

Alert at level 7 on every authentication failure

If the user is Bob, raise the alert level to 12


Wait a Minute

What if Bob has 3,561 login failures again?


Wait a Minute

What if his login failures aren't just through SSH?


Revised Rule Thoughts

Alert me if Bob has a few authentication failures in a short time, from anywhere,

but don't flood me with alerts


Revised Rule for Bob

Let's try that last rule again

<rule id=”133” level=”12” frequency=”10” timeframe=”300” ignore=”60”><if_matched_group>authentication_failed</if_matched_group><user>Bob</user><description>Bob is acting up</description></rule>


Rule Examples

Other interesting rules


Attack Followed by Account

<group name="syslog,elevation_of_privilege,"> <rule id="40501" level="15" timeframe="300" frequency="2"> <if_group>adduser</if_group> <if_matched_group>attacks</if_matched_group> <description>Attacks followed by the addition of an user.</description> </rule></group>


Really Long URL

<rule id="31115" level="13" maxsize="2900"><if_sid>31100</if_sid><description>URL too long. Higher than allowed on most browsers. Possible attack.</description><group>invalid_access,</group></rule>


Multiple Windows Errors

<rule id="18154" level="10" frequency="$MS_FREQ" timeframe="240"> <if_matched_sid>18103</if_matched_sid> <description>Multiple Windows error events.</description> </rule>


Windows Application Installed

<rule id="18147" level="5"> <if_sid>18101</if_sid> <id>^11707</id> <options>alert_by_email</options> <description>Application Installed.</description> </rule>


Windows Audit Policy Changed

<rule id="18113" level="8"> <if_sid>18104</if_sid> <id>^612|^643|^4719|^4907|^4912</id> <description>Windows Audit Policy changed.</description> <group>policy_changed,</group> </rule>


Virus Found, Not Removed

<rule id="7504" level="12"> <if_sid>7500</if_sid> <regex>$MCAFEE_VIRUS</regex> <group>virus</group> <description>McAfee Windows AV - Virus detected and not removed.</description> </rule>


Integrity Monitoring

Keeping a Known Good State


File Integrity

SHA-1 and MD5 of critical system files and registry keys

Performed in real-time or on a schedule

Auto-ignores files that change too often


File Integrity

Also checks owner, group, permissions

Hashes forwarded to manager for safe keeping (excellent for forensics)

Use the full power of rules to manage alerts(e.g. alert only on changes outside patch window)


World Writable File

OSSEC HIDS Notification.2009 Oct 21 12:02:27

Received From: hostname->syscheckRule: 100018 fired (level 7) -> "World Writable File"Portion of the log(s):

Integrity checksum changed for: '/etc/httpd/conf/httpd.conf'Permissions changed from 'rw-------' to 'rw-r--rw-'

--END OF NOTIFICATION


No Longer World Writable


Received From: hostname->syscheckRule: 552 fired (level 7) -> "Integrity checksum changed again (3rd time)."Portion of the log(s):

Integrity checksum changed for: '/etc/httpd/conf/httpd.conf'Permissions changed from 'rw-r--rw-' to 'rw-------'



Agentless Integrity

Periodic diff of firewalls and routers

Checksum and diff of remote 'nix systems

It's nice to know something changed, but what?Agentless check of /etc/password

shows what changed


Agentless Alerts

OSSEC HIDS Notification.2009 May 14 16:32:20

Received From: (ssh_pixconfig_diff) [email protected]>agentlessRule: 555 fired (level 7) -> "Integrity checksum for agentless device changed."Portion of the log(s):

ossec: agentless: Change detected:206a207> port-object eq 4241556c557

...


Rootkit Detection

Exposing the Hidden


Unix Rootkit Detection

Signature and anomaly-based

Signatures automatically sent to agents

Can be run stand-alone


Signature Method

Signatures for Adore, Knark, LOC, etc

Attempt to stats, fopen and opendir each specified file

Some rootkits don't fully hide themselves


Anomaly Method

Detects known and unknown rootkits

Files in /dev which aren't device files

“Unusual” files(hidden directories, files owned by root

which are world-writable)


Anomaly Method

Running processes hidden from “ps”

Listening ports hidden from “netstat”

Promiscuous interfaces hidden from “ifconfig”


Rootcheck Alert


Received From: XXXX->rootcheckRule: 510 fired (level 7) -> "Host-based anomaly detection event (rootcheck)."Portion of the log(s):

Rootkit 'Suspicious' detected by the presence of file '/var/www/vhosts/YYYY.com/httpdocs/language/lang_english/ /... /.log'.


Source: http://www.void.gr/kargig/blog/2009/10/06/ossec-to-the-rescue/


Windows Rootkit Detection

Not as advanced as Unix-based detection

Alternate data streams

(Files hidden within files)


Policy Monitoring

Detect Insecure Conditions


Policy Monitoring

Is your system configured securely?

Identify situation which can lead to a breach

Benchmark system against CIS standard or create your own


Policy Monitoring

File, registry setting, or process exists or does not exist

Combine values with logical AND/OR

Is anti-virus installed but not running?


Policy Monitoring

Has the host firewall been disabled?

Is LanMan authentication allowed?

*Does not alert by default


Alerting

Getting Notified


Alerting

E-mail, syslog and database output

Built-in e-mail flood protection

Send alerts to different teamsbased on granular rules, severity or group


Alerting

On second thought, maybe it wasn'tBob who tried to login to his account

Someone should get a page if this happens again


Can't Miss the Game

What if it's the weekend and I'm watching the game?


Alerting

That someone should beHenry, the Jr. Security Analyst

What a wonderful opportunityfor “professional development”


Alerting

Create another rule without restricting it to Bob,which will only fire on the weekends

<rule id=”144” level=”12” frequency=”10” timeframe=”300” ignore=”60”><if_matched_group>authentication_failed</if_matched_group><weekday>Saturday,Sunday</weekday><description>Multiple Weekend Authentication Failures</description></rule>


Alerting

Followed by an alert configuration is ossec.conf

<email_alerts> <email_to>[email protected]</email_to> <rule_id>144</rule_id> <format>sms</format></email_alerts>


Alerting

Syslog or database output easilyintegrated with commercial SIEMs

Use OSSEC for the analysis

Use the SIEM GUI for advanced correlation


Rule Examples

Other interesting alerts


Excessive Events


Received From: hostname->/var/log/httpd/error_logRule: 11 fired (level 8) -> "Excessive number of events (above normal)."Portion of the log(s):

The average number of logs between 4:00 and 5:00 is 936. We reached 1218.


First-Time Login


Received From: hostname->/var/log/secureRule: 10100 fired (level 4) -> "First time user logged in."Portion of the log(s):

Oct 22 11:24:33 hostname sshd[2998]: Accepted password for kevin_mitnick from 12.174.169.111 port 52387 ssh2


First Sudo Attempt


Received From: hostname->/var/log/secureRule: 5403 fired (level 4) -> "First time user executed sudo."Portion of the log(s):

Oct 22 11:27:49 hostname sudo: kevin_mitnick : user NOT in sudoers ; TTY=pts/1 ; PWD=/ ; USER=root ; COMMAND=/bin/su -


Active Response

Preventing Breaches


Active Response

Attackers follow common patterns

1. Reconnaissance2. Scan3. Exploit

OSSEC can often prevent breaches by detecting attacks in the early stages


Active Response

Not an IPS, but effective


Active Response

Time-based security implementation

Protection time should be greater than the sum of detection time, plus reaction time

(D+R)>P

This is good!


Active Response

If severity > 6, add the attacker's IPto the host firewall for 10 minutes

Or the perimeter firewall...Or disable an account...Or shut down the system...


Active Response

Execute responses on the manager, one particular agent, a firewall or everywhere

Worldwide?


OSSEC WebUI

A Face to OSSEC


Benefits of GUIs

GUI interfaces allow you to see trends and patterns over time

FTP account gets locked out every day at 4:15 AM

What alerts does OSSEC think aren't worthy of an e-mail?


OSSEC WebUI


OSSEC WebUI


OSSEC WebUI


Other GUI Options

Other options include:

Splunk

OSSIM

Picviz


Why OSSEC?


PCI DSS 1.2

10.5.5 Use file-integrity monitoring or change-detection software on logs to ensure that existing log data cannot be changed without generating alerts (although new data being added shouldnot cause an alert).


PCI DSS 1.2

10.6 Review logs for all system components at least daily...

...Note: Log harvesting, parsing, and alerting tools may be used to meet compliance with Requirement 10.6


Closing the NIDs Circle

Network-based IDS

Only half the picture



Host-based IDS

The other half



Network and Host-based IDS

A new level of insight into your environment



Of course, OSSEC reads NIDs logs


Forensics

Everything is forwarded to the manager for analysis and possible storage

Attackers like to delete logs


Policy Compliance

How do you know your systems are still hardened?

Are admins logging in with unique accounts?

Is anti-virus running?


Keep Employees Honest

Insider threats cost companies millions per year

Employees who know their activitiesare monitored tend to be more honest


Budget

OSSEC can be used for free


Risks & Countermeasures


Mass Deployment

Deploying large amounts of agents is challenging

Each agent uses a unique key

How can a single package be created?


Active Response

Attackers who know Active Response is in use may try to use that to their advantage

IPs can be spoofed, thereby triggering an incorrect response


Alert Flooding

You have 6,972 new messages!

Will you read them all?


Log Injection

Attacker uses poorly written regular expressions to bypass rules

root@slacker:~# ftp 192.168.3.4220 Welcome to labs ossec candy FTP service.Name (192.168.2.3:root): lala] FAIL LOGIN: Client “2.3.4.54″

Normal LogMon Jun 2 21:05:30 2007 [pid 1448] [myuser] FAIL LOGIN: Client “192.168.3.1″Log InjectionMon Jun 2 21:06:02 2007 [pid 1452] [lala] FAIL LOGIN: Client “2.3.4.54″ ] FAIL LOGIN: Client “192.168.3.1″


Risk Countermeasures

E-mail floodingBy default, OSSEC will only send 12 alerts per hour, queuing the rest until the next hour

Active ResponseResponse timeoutIP whitelists

Log InjectionTight regular expressions


Enterprise Considerations


Define the Problem

What problem are you trying to solve?

What are your primary drivers?

What are the obstacles?


Codify in Policy

Explicitly state the need in policy


Set Requirements

Requirements are a measure of success


Define the Scope

Will you monitor all systems?

What is the budget?

What is the time-frame?


Make a Desicion

Is OSSEC a good fit?

Don't design a solutionlooking for a problem!


Plan, Do, Check, Act

Plan your OSSEC rollout

Do the actual rollout

Check the requirements against the rollout

Act on the lessons learned


Demo


Summary

OSSEC can add a new level of insight into your environment

Only use OSSEC if it fits a need

If you do use OSSEC, contribute yourdecoders, rules and lessons learnedback to the community!


Questions?


Acknowledgements

Daniel B. Cid, OSSEC creator

Trend Micro

Rochester Security Summit

OSSEC Aucert presentation


Image CreditsAgenda: http://www.sxc.hu/photo/807162Question mark: http://www.sxc.hu/photo/1147438Tree: http://www.sxc.hu/photo/1195970Vintage Mac: http://www.sxc.hu/photo/1028528Rubber band ball: http://www.sxc.hu/photo/168735Padlock: http://www.sxc.hu/photo/865986Fast car: http://www.sxc.hu/photo/1081680Cardboard box: http://www.sxc.hu/photo/1036068Jumping man: http://www.sxc.hu/photo/1212299Camera lid: http://www.sxc.hu/photo/450946Buckets: http://www.sxc.hu/photo/807354Ruler: http://www.sxc.hu/photo/1010158Bob: http://www.sxc.hu/photo/912662OSSEC WUI: http://www.ossec.net/dcid/?p=29Road sign: http://www.sxc.hu/photo/1157986

The following images were used under fair use provisions of US copyright and trademark law:Logos: Windows, Tux, FreeBSD, PCI and AIXOSSEC WebUI screenshots

http://www.sxc.hu/photo/807162













http://www.ossec.net/dcid/?p=29



Image CreditsFiles in basket: http://www.sxc.hu/photo/456727Potato: http://www.sxc.hu/photo/1132394Paper stack: http://www.sxc.hu/photo/251979Old phone: http://www.sxc.hu/photo/1146563Little guy and stop sign: http://www.sxc.hu/photo/1197499Fence: http://www.sxc.hu/photo/1044635Clock: http://www.sxc.hu/photo/1026820Retro TV: http://www.sxc.hu/photo/981522Sunglasses: http://www.sxc.hu/photo/621374Happy face: http://www.sxc.hu/photo/1147441Thumb print: http://www.sxc.hu/photo/1231735Fist: http://www.sxc.hu/photo/621374Money symbol: http://www.sxc.hu/photo/983478Crowd: http://www.sxc.hu/photo/893433E-mail: http://www.sxc.hu/photo/1102040Red cross: http://www.sxc.hu/photo/971655


















Text Credits

“Attacking Log Analysis Tools,” Daniel B. Cid: http://www.ossec.net/main/attacking-log-analysis-tools

“OSSEC at AusCERT,” Daniel B Cid: http://www.ossec.net/ossec-docs/auscert-2007-dcid.pdf

http://www.ossec.net/main/attacking-log-analysis-tools


Presentation LicenseThis presentation is licensed under the Creative Commons Attribution-Noncommercial-Share Alike 3.0 license. The license does not extend to images, which hold their own copyrights attributed to various authors.

You are free:

to Share — to copy, distribute and transmit the workto Remix — to adapt the work

Under the following conditions:

Attribution — You must attribute the work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work).Noncommercial — You may not use this work for commercial purposes.Share Alike — If you alter, transform, or build upon this work, you may distribute the resulting work only under the same or similar license to this one.

With the understanding that:

Waiver — Any of the above conditions can be waived if you get permission from the copyright holder.Other Rights — In no way are any of the following rights affected by the license:Your fair dealing or fair use rights;Apart from the remix rights granted under this license, the author's moral rights;Rights other persons may have either in the work itself or in how the work is used, such as publicity or privacy rights.Notice — For any reuse or distribution, you must make clear to others the license terms of this work.

Documents

OSSEC in the Enterprise - Immutable Security