Deep Dive Into ArcSight ESM Rules - Micro Focus …...Deep Dive Into ArcSight ESM Rules Rob Block Sr. Software Engineer, Correlation Team September 2009 Overview Swim – Rule Basics

© 2009 ArcSight, Inc. All rights reserved. ArcSight and the ArcSight logo are trademarks of ArcSight, Inc. All other product and company names may be trademarks or registered trademarks of their respective owners.

Deep Dive Into ArcSight ESM Rules

Rob BlockSr. Software Engineer, Correlation TeamSeptember 2009

Overview

Swim– Rule Basics– Rule Matching Overview– Rule Chains

Dive– Rule Partial Matching– Rule Engine Details– Aggregation Engine Details

Deep dive– Excessive Rule Firing– Negated Aliases– Batched Replay and Schedules

www.arcsight.com © 2009 ArcSight Confidentialwww.arcsight.com © 2009 ArcSight Confidential 2

Let’s Swim First

Overview of rules

Overview of rule matching

Rule chains

www.arcsight.com © 2009 ArcSight Confidential 3

ArcSight Rules

Evaluate incoming events for specific conditions and patterns

Correlate information from different events using – Rule correlation – Active lists– Session lists– Threat level calculations

Infer meaning about significance of events, and initiate real-time actions in response


Correlation EventFields• Type = Correlated• Name = Name of the Rule• File Path = URI of the Rule… +

• Fields set using SetEventField Action +

• Fields used in Aggregation

Structure of a Rule

Condition Matching

Events Matches Execute Actions

On Qualifying Triggers

Audit Events For ActionsFields• Type = Action• Name = Type of Action + Success/Failure

• File Path: URI of the Rule• Device Custom String 4: Trigger Involved

GeneratesRule Chain

Aggregation


Condition Matching

Conditions








GeneratesRule Chain

Aggregation

Structure of a Rule


Conditions

Rule can have multiple sets of conditions, each matching one event

Example: rule with two sets of conditions, matching events– Brute force login attempt– Successful login

Join conditions correlate the aliases themselves; both events came from same attacker

Structure of a Rule


Aggregation


On Qualifying TriggersAggregation

Specifies number of matches (threshold) needed in specified time frame

Can aggregate based on identical and/or unique fields

Attack is in progress as long as threshold has been met within latest time frame period

Structure of a Rule

Condition Matching


Triggers

Structure of a Rule

Condition Matching


On Qualifying TriggersAggregation

Trigger typesOn First EventOn Subsequent EventOn Every EventOn First ThresholdOn Subsequent ThresholdOn Every ThresholdOn Time UnitOn Time Window Expiration


Actions

Structure of a Rule

Condition Matching



Action types

Set event fieldSend notificationCreate new caseAdd to caseExecute a commandExecute a connector commandAdd to active listRemove from active listAdd to session listTerminate session

Aggregation


Audit Events and Correlation Events




Condition Matching





GeneratesRule Chain

Aggregation

Structure of a Rule


Rule Chain




Condition Matching





GeneratesRule Chain

Aggregation

Structure of a Rule


Rule Chain

Base events corresponding to the rule trigger

Rule chain is not cumulative – Exception: option for on time window expiration

Example: threshold = 3, on every threshold trigger– 3 base events in rule chain for each OET firing

Structure of a Rule


Rule chain includes the events that contributed to the particular firing of the rule

Consider a rule, with threshold = 3, aggregation time window = 1 minute, and following triggers activated– On first threshold– On time unit (every minute)– On time window expiry

Note: on time unit rule chain only includes events after the first threshold

Rule Chain

12:01 12:02 12:03

First Threshold On Time Unit

12:04

Time Window Expiry

Structure of a Rule

Matching Events


Let’s Dive

Rule partial matching

Rule engine details

Aggregation engine details


Rule Partial Matching

Two types of rules– Filter rule: contains single alias– Join rule: contains multiple aliases

An event matching a rule alias generates either a partial match or a full match for the rule, depending on the number of rule aliases

A single event usually matches one rule alias


Example: Filter Rule

Multiple failed logins on Windows systems

Multiple failed logins on UNIX systems

5 or more failed logins in a minute from same source

on same target

Attempted Brute Force Attack




Example: Join Rule

Correlates two or more different kind of events

Attempted Brute Force Attack + Successful

Login


Successfullogin to Target system



Join Rules and Memory Usage

Join rules usually require more memory than filter rules, due to partial match maintenance

Partial matches– Stored in memory for the specified time window, waiting for

complimentary events– Only minimal event information is stored in the partial match, needed

for join conditions and aggregation

Matching time window for these rules should not be kept too long– Use active lists to correlate information from events spaced far in time



Time Constraints in Rules

Aggregation time: time period to use for aggregation (to wait for specified number of matches)

Alias expiration time: how long partially matched events are kept in memory



Rule Engine

Rule engine matches incoming security events

against the deployed rules

!


Rule Engine Structure

Rule Evaluation

Rules

Working Memory(Events)

Relevant Security Events

Rules Engine

Insert

Matches AggregationEngine

Garbage Collector


Working Memory

Match

Garbage CollectorSuccessful

Login Attempt event2event2

Other Events Matching Filter Rules Eventsevents

Attempted Brute Force Attack Event

event1Insert

AggregationEngine

event1

Rule Evaluation

Rules

Relevant Security Events

Rules Engine

Matches

Rule Engine Example

ExpiredExpiredExpiredExpired


Rule Aggregation

Recognizes patterns involving repetitive events– Example: five failed logins

Has impact on memory as aggregation matches are counted and tracked

Aggregation cell: set of matching events satisfying the aggregation criteria (identical/unique field values)

Large # of aggregation cells increases memory usage

Once time window expires, inactive matches are cleared

One aggregation tracker per rule


insert

Aggregation Engine Structure

Matches from Rules Engine

Rules TriggeredAggregation Cells (set of identical

/unique field values)

Matches are added to an aggregation cell based upon values of the aggregated fields

Thresholds Calculated (per aggregation cell)

Aggregation Cell Identified


AttackerAddress: 192.168.1.5 AttackerZone: InternalZone1

Insert

Matches from Rules Engine

Rules TriggeredAggregation Cells Thresholds Calculated (per aggregation cell)

Aggregation Cell Identified

Aggregation Engine ExampleRule matching 2 failed logins in a minute from same source

AttackerAddress: 192.168.1.2 AttackerZone: InternalZone1

192.168.1.2, InternalZone1

Count = 1


Count = 1

Expire after a minute


Count = 2

Matches are added to an aggregation cell based upon values of the aggregated fields


Aggregate Only When Unique Option

Allows capturing the cases of widespread problems

Example– A rule to identify widespread computer virus in a corporation– Rule may be written to fire when the virus notifications received from at

least 10 different machines– Here we are interested in events from unique machines


Apart from more commonly used aggregation on matching values, aggregation can also be done on unique set of values

Deep Dive

Excessive rule firing—understand and avoid

Negated aliases

Batched replay and schedules


Excessive Rule Firing or Partial Matches

Rule having too relaxed conditions

Triggers not defined judiciously

Single rule recursion– Correlated alerts lead to further firing of the rule

Multiple rule recursion– A set of rules form a recursive cycle

and lead to mutual firing of rules


Excessive Rule Firing

Excessive rule firing and rule recursion is identified and reported to users using audit events

In excessive rule firing– Rule is temporarily deactivated– Activated again after elapse of time equivalent to rule

aggregation time

In case of rule recursion, the events causing recursion are cut off from the recursion loop


Example: Excessive Rule Firing

Denial of service attack

An attacker can flood a server with traffic

There may be a rule defined to identify such attacks

If rule trigger is activated for every event or on every threshold it may lead to excessive rule firing

Can make it difficult looking at other rule firings in the channel


Excessive Rule Firing

On first threshold: will notify start of attackOn time unit: will periodically notify that the attack is still

going onOn time window expiration: will notify end of attack

A solution to handle long running continuous attacks would be to define following triggers


Negated Aliases

Consider a rule R1 with two aliases E1, E2– E1: matches a server console login event, for a server inside the

server room– E2 (negated): matches a badge scan event to enter the server room

Badge scan event should happen before the server console login event, as the person has to be present in the server room to login

Rule will fire if someone logged on to a console in the server room, without scanning the badge to get inside the room


Use of negated aliases in join rules provide ability to take action on missing events

Using Negated Aliases to Find Absence of an Event in the Future

It cannot look for missing event in future

Consider a rule R1 with two aliases E1, E2– E1: matches a server reboot event– E2 (negated): matches a server up event

This rule will always fire on receiving a server reboot event, as server up event hasn’t arrived at that point (server up event will happen in future)


Rules using negated aliases evaluate the absence of the negated event, at the time rule is evaluated

Using Negated Aliases to Find Absence of an Event in the Future

Rule 1 Rule 2 with two aliases E1, E2

• Matches the server reboot event• Trigger defined on (on time window expiration), with suitable aggregation time window (ex: 4 minutes)

• E1: matches Rule 1 event• E2 (negated): matches a server up event

Now Rule 2 will be evaluated when Rule1 fires, which had been time delayed due to on time window expiration trigger

This case can be handled using 2 rules, so as to introduce delay in the event of a matching positive alias


Batched Replay with Rules

Use case: badge reader events sent to manager in batch once a day

Replay task runs in own rules engine

Events queried from DB– Not all event fields are queried– Essential fields plus fields needed to compute conditions, variables,

aggregation and actions– Use event filter to improve performance

Conditions like InActiveList may fail due to expired list entries


Rules can process either real-time events, or historical events in batched replay mode

Scheduled Rules Replay

Schedule rule group to run with specified frequency

Reads events with end time in corresponding time period

Can create multiple schedules for same rule group– May get multiple rule firings for same historical events

Can specify time delay between data cutoff and schedule start time– Server property: rules.batched.time.delay (in ms)

9/14 2:00 9/15 2:00 9/15 2:30

Badge reader events9/14 2:00 – 9/15 2:00

Schedule starts

9/15 3:00

Arrive at Manager

Delay = 60x60x1000 ms

End Time query range

Batched Replay with Rules


Summary

Rule Matching Details

Rule Aggregation Details

Memory Aspects of Rule Matching and Aggregation

Rule Chains

Avoiding Excessive Rule Firings

Use of Negated Aliases

Batched Replay and Schedules


Documents

Deep Dive Into ArcSight ESM Rules - Micro Focus …...Deep Dive Into ArcSight ESM Rules Rob Block Sr. Software Engineer, Correlation Team September 2009 Overview Swim – Rule Basics