Firewalls and Intrusion Detection Systems Vyas Sekar vsekar@andrew.cmu.edu Carnegie Mellon...

Firewalls and Intrusion Detection Systems

Vyas Sekarvsekar@andrew.cmu.edu

Carnegie Mellon University

2

IDS and Firewall GoalsExpressiveness: What kinds of policies can we write?

Effectiveness: How well does it detect attacks while avoiding false positives?

Efficiency: How many resources does it take, and how quickly does it decide?

Ease of use: How much training is necessary? Can a non-security expert use it?

Security: Can the system itself be attacked?

Transparency: How intrusive is it to use?

3

Taxonomy• Approach: Policy vs Anomaly• Location: Network vs. Host• Action: Detect vs. Prevent• Semantics: IP vs TCP vs App

Type Example

Host, Rule, IDS Tripwire

Host, Rule, IPS Personal Firewall

Net, Rule, IDS Snort

Net, Rule, IPS Network firewall

Host, Anomaly, IDS System call monitoring

Net, Anomaly, IDS Working set of connections

Net, Anomaly, IPS

4

FirewallsDimensions:1. Host vs. Network2. Stateless vs. Stateful3. Network Layer

5

Firewall Goals

Provide defense in depth by:1. Blocking attacks against hosts and services2. Control traffic between zones of trust

6

Logical Viewpoint

Inside OutsideFirewall

For each message m, either:• Allow with or without modification• Block by dropping or sending rejection notice• Queue

m

?

7

Placement

Host-based Firewall

Network-Based Firewall

Host Firewall Outside

Firewall OutsideHost B

Host C

Host A

Features:• Faithful to local

configuration• Travels with you

Features:• Protect whole network• Can make decisions on

all of traffic (traffic-based anomaly)

8

Parameters

Types of Firewalls1. Packet Filtering2. Stateful Inspection3. Application proxy

Policies1. Default allow2. Default deny

9

Recall: Protocol Stack

Application(e.g., SSL)

Transport (e.g., TCP, UDP)

Network (e.g., IP)

Link Layer(e.g., ethernet)

Physical

Application message - data

TCP data TCP data TCP data

TCP Header

dataTCPIP

dataTCPIPETH ETH

Link (Ethernet) Header

Link (Ethernet) Trailer

IP Header

10

Stateless FirewallFilter by packet header fields1. IP Field

(e.g., src, dst)2. Protocol

(e.g., TCP, UDP, ...)3. Flags

(e.g., SYN, ACK)

Application

Transport

Network

Link Layer

Firewall

Outside Inside

Example: only allow incoming DNS packets to nameserver A.A.A.A.

Allow UDP port 53 to A.A.A.ADeny UDP port 53 allFail-safe good

practice

e.g., ipchains in Linux 2.2

11

Need to keep state

Inside Outside

Listening

Store SNc, SNs

Wait

SNCrandC

ANC0Syn

SYN/ACK:SNSrandS

ANSSNC

Established

ACK: SNSNC+1ANSNS

Example: TCP HandshakeFirewall

Desired Policy: Every SYN/ACK must have been preceded

by a SYN

12

Stateful Inspection Firewall

Added state (plus obligation to manage)

– Timeouts– Size of table

State

Application

Transport

Network

Link Layer

Outside Inside

e.g., iptables in Linux 2.4

13

Stateful More Expressive

Inside Outside

Listening

Store SNc, SNs

Wait

SNCrandC

ANC0Syn

SYN/ACK:SNSrandS

ANSSNC

Established

ACK: SNSNC+1ANSNS

Example: TCP HandshakeFirewall

Record SNc in table

Verify ANs in table

14

Stateful Firewalls

Pros• More expressive

Cons• State-holding attack• Mismatch between

firewalls understanding of protocol and protected hosts

15

Application Firewall

Check protocol messages directly

Examples:– SMTP virus scanner– Proxies– Application-level

callbacks

State

Application

Transport

Network

Link Layer

Outside Inside

16

Firewall Placement

17

Demilitarized Zone (DMZ)

Inside OutsideFirewall

DMZ

WWW

NNTP

DNS

SMTP

Typical Firewall Configuration

• Internal hosts can access DMZ and Internet

• External hosts can access DMZ only, not Intranet

• DMZ hosts can access Internet only

• Advantages?

• If a service gets compromised in DMZ it cannot affect internal hosts

InternetInternet

IntranetIntranet

DMZDMZ

XX

18

19

Dual Firewall

Inside OutsideHubDMZ

InteriorFirewall

ExteriorFirewall

20

References

Elizabeth D. ZwickySimon Cooper

D. Brent Chapman

William R CheswickSteven M Bellovin

Aviel D Rubin

21

Intrusion Detection and Prevetion Systems

22

Logical Viewpoint

Inside OutsideIDS/IPS

For each message m, either:• Report m (IPS: drop or log)• Allow m• Queue

m

?

23

Overview• Approach: Policy vs Anomaly• Location: Network vs. Host• Action: Detect vs. Prevent• Semantics: “looks deeper”

24

Policy-Based IDSUse pre-determined rules to detect attacks

Examples: Regular expressions (snort), Cryptographic hash (tripwire,

snort)Detect any fragments less than 256 bytesalert tcp any any -> any any (minfrag: 256; msg: "Tiny fragments detected, possible hostile activity";)Detect IMAP buffer overflowalert tcp any any -> 192.168.1.0/24 143 ( content: "|90C8 C0FF FFFF|/bin/sh"; msg: "IMAP buffer overflow!”;)

Example Snort rules

25

Modeling System Calls [wagner&dean 2001]

Entry(f)Entry(g)

Exit(f)Exit(g)

open()

close()

exit()

getuid() geteuid()

f(int x) { if(x){ getuid(); } else{ geteuid();} x++;}g() { fd = open("foo", O_RDONLY); f(0); close(fd); f(1); exit(0);}

Execution inconsistent with automata indicates attack

26

Anomaly Detection

Distribution of “normal” events

IDS

New Event

Attack

Safe

27

Example: Working Sets

Alice

Days 1 to 300

reddit xkcd

slashdot

fark

working setof hosts

Alice

Day 300

outside working set

reddit xkcd

slashdot

fark18487

28

Anomaly Detection

Pros• Does not require pre-

determining policy (an “unknown” threat)

Cons• Requires attacks are

not strongly related to known traffic

• Learning distributions is hard

29

Attacks and Evasion

30

State Holding Attack

Firewall AttackerInside

SynSyn

Syn...

1. SynFlood

2. Exhaust Resources

3. Sneak Packet

Assume stateful TCP policy

31

Fragmentation

Octet 1 Octet 2 Octet 3 Octet 4

Ver IHL TOS Total Length

ID 0DF

MF

Frag ID

...

Data

Frag 1 Frag 2 Frag 3

IP Hdr DF=0 MF=1 ID=0 Frag 1

IP Hdr DF=0 MF=1 ID=n Frag 2

IP Hdr DF=1 MF=0 ID=2n Frag 3

say n bytes

DF : Don’t fragment (0 = May, 1 = Don’t)MF: More fragments(0 = Last, 1 = More)Frag ID = Octet number

32

ReassemblyData

Frag 1 Frag 2 Frag 3

IP Hdr DF=0 MF=1 ID=0 Frag 1

IP Hdr DF=0 MF=1 ID=n Frag 2

IP Hdr DF=1 MF=0 ID=2n Frag 3

Frag 1 Frag 2 Frag 3

0 Byte n Byte 2n

33

Example2,366 byte packet enters a Ethernet network with a default MTU size of 1500

Packet 1: 1500 bytes– 20 bytes for IP header– 24 Bytes for TCP header– 1456 bytes will be data– DF = 0 (May fragment), and MF=1 (More fragments)– Fragment offset = 0

Packet 2: 910 bytes– 20 bytes for IP header– 24 bytes for the TCP header– 866 bytes will be data– DF = 0 (may fragment), MF = 0 (Last fragment)– Fragment offset = 182 (1456 bytes/8)

34

Octet 1 Octet 2 Octet 3 Octet 4

Source Port Destination Port

Sequence Number

....

...DF=

1MF=1 ID=0 ...

1234(src port)

80(dst port)

...Packet 1

Overlapping Fragment Attack

...DF=

1MF=1 ID=2 ... 22 ...Packet 2

1234 8022

Assume Firewall Policy: Incoming Port 80 (HTTP) Incoming Port 22 (SSH)

Bypass policyTCP Hdr(Data!)

Small TTL attack

35

same TCP seq #, “I” has short TTL

Internet

NIDS

Host

End-host sees:Attacker’s data stream

A T T I A C K A T T A C K

A

NIDS sees:

T T I A C K

Fragmentation overlap attack

36

Internet

End-host sees:

NIDS sees:

Attacker’s data stream

NIDS

Host

A T T A I C K

same TCP seq #or same IP frag offset

A T T A I C K

A T T A I C K

Potential Fix: traffic normalizerIntroduce “bump in the wire”: traffic normalizer to evade protocol ambiguities

37

Internet

NIDS

HostNormalizer

Algorithmic complexity attacks

• DoS attacks not only serious for denying service, but can be more severe by using it as a component of an attack

• DoS attack on IDS enables other attacks to remain undetected

• E.g., “Denial of Service via Algorithmic Complexity Attacks” by Crosby and Wallach

38

39

Detection TheoryBase Rate, fallacies, and detection systems

40

Let be the set of all possible events. ΩFor example:

• Audit records produced on a host• Network packets seen

Ω

41

Ω

I

Set of intrusion events I

Intrusion Rate:

Example: IDS Received 1,000,000 packets. 20 of them corresponded to an intrusion.The intrusion rate Pr[I] is:Pr[I] = 20/1,000,000 = .00002

42

Ω

I A

Set of alerts A

Alert Rate:

Defn: Sound

43

Ω

I

A

Defn: Complete

44

Ω

I A

Defn: False PositiveDefn: False Negative

Defn: True Positive

Defn: True Negative

45

Ω

I A

Defn: Detection rate

Think of the detection rate as the set ofintrusions raising an alert normalized by the set of all intrusions.

48

Ω

I A

18 4

2

50

Ω

I A

Think of the Bayesian detection rate as the set of intrusions raising an alert normalized by the set of all alerts. (vs. detection ratewhich normalizes on intrusions.)

Defn: Bayesian Detection rateCrux of IDS usefulness!

52

Ω

I A2

4

18

About 18% of all alerts are false positives!

53

Challenge

We’re often given the detection rate and know the intrusion rate, and want to calculate the Bayesian detection rate

– 99% accurate medical test– 99% accurate IDS– 99% accurate test for deception– ...

54

Fact:

Proof:

55

Calculating Bayesian Detection RateFact:

So to calculate the Bayesian detection rate:

One way is to compute:

56

Example

• 1,000 people in the city

• 1 is a terrorist, and we have their pictures. Thus the base rate of terrorists is 1/1000

• Suppose we have a new terrorist facial recognition system that is 99% accurate.– 99/100 times when someone is a

terrorist there is an alarm– For every 100 good guys, the

alarm only goes off once.

• An alarm went off. Is the suspect really a terrorist?

City

(this times 10)

57

Example

Answer: The facial recognition system is 99% accurate. That means there is only a 1% chance the guy is not the terrorist.

(this times 10)

City

Wrong!

58

Formalization

• 1 is terrorists, and we have their pictures. Thus the base rate of terrorists is 1/1000. P[T] = 0.001

• 99/100 times when someone is a terrorist there is an alarm.P[A|T] = .99

• For every 100 good guys, the alarm only goes off once.P[A | not T] = .01

• Want to know P[T|A]

City

(this times 10)

59

• 1 is terrorists, and we have their pictures. Thus the base rate of terrorists is 1/1000. P[T] = 0.001

• 99/100 times when someone is a terrorist there is an alarm.P[A|T] = .99

• For every 100 good guys, the alarm only goes off once.P[A | not T] = .01

• Want to know P[T|A]

City

(this times 10)

Intuition: Given 999 good guys, we have 999*.01 ≈ 9-10 false alarms

False alarms

60

Unknown

Unknown

61

Recall to get Pr[A]Fact:

Proof:

63

..and to get Pr[A∩ I]Fact:

Proof:

64

✓

✓

=

=

65

=

66

Visualization: ROC(Receiver Operating Characteristics Curve)

Plot true positive vs. false positive for a binary classifier at various threshold settings

67

For IDS

Let – I be an intrusion,

A an alert from the IDS

– 1,000,000 msgs per day processed

– 2 attacks per day– 10 attacks per

message

False positives

70% detection requires

FP < 1/100,000

80% detection generates 40% FP

From Axelsson, RAID 99

68

Why is anomaly detection hard

Think in terms of ROC curves and the Base Rate fallacy.

– Are real things rare? If so, hard to learn– Are real things common? If so, probably ok.

69

Conclusion• Firewalls

– 3 types: Packet filtering, Stateful, and Application– Placement and DMZ

• IDS– Anomaly vs. policy-based detection

• How can we exploit for evasion?– E.g., fragmentation, TCP session reassembly, TTL

• How can we attack the defense infrastructure?– E.g., overload, algorithmic complexity

• Detection theory– Base rate fallacy

70

Questions?

END

72

Thought