Collaborative Intrusion Prevention

Simon P. Chung and Aloysius K. Mok�Department of Computer Sciences

University of Texas at Austin

Austin, TX, 78712, USAfphchung, mokg@cs.utexas.edu

Abstract

Intrusion Prevention Systems (IPSs) have long been pro-

posed as a defense against attacks that propagate too fast

for any manual response to be useful. In an important class

of IPSs, the host-based IPSs, honeypots are used to collect

information about attacks. The collected information will

then be analyzed to generate countermeasures against the

observed attack. Unfortunately, these IPSs can be rendered

useless by techniques that allow the honeypots in a network

to be identified ([1, 9]). In particular, attacks can be de-

signed to avoid targeting the identified honeypots. As a re-

sult, the IPSs will have no information about the attacks,

and thus no countermeasure will ever be generated. The use

of honeypots is also creating other practical issues which

limit the usefulness/feasibility of many host-based IPSs. We

propose to solve these problems by duplicating the detection

and analysis capability on every protected system; i.e., turn-

ing every host into a honeypot. In this paper, we will first

lay out the necessary features of any scheme for such large

scale collaboration in intrusion prevention, then we will

present a framework called Collaborative Intrusion Preven-

tion (CIP) for realizing our idea of turning every host into

a honeypot.

1 Introduction

It is a well recognized fact that the manual generation

and application of patches are too slow to protect vulnerable

systems from attacks, with the most cited demonstration of

this fact being the outbreak of the SQLSlammer worm. In

response, many have proposed intrusion prevention systems

(IPSs) as a defense against this threat from fast-propagating

attacks. One important class of IPSs is the host-based IPSs1�The research reported here is supported partially by a grant from the

Office of Naval Research under contract number N00014-03-1-07051By “host-based IPSs” we refer to all IPSs that analyze how the target

host processes attack traffic.

(e.g. [5, 8, 2, 10, 13, 7]). Many existing host-based IPSs

employ honeypots to detect and collect information about

attacks so that countermeasures against the detected attacks

can be generated2. However, there are a few drawbacks in

these systems:

1. As shown in [1, 9], the addresses of honeypots can be

known to the attackers, allowing them to avoid inter-

action with the honeypots. As a result, the IPSs will

be unable to collect any information about the new at-

tacks, and thus no countermeasures will be generated.

Furthermore, the heavy-weighted analysis commonly

performed in the honeypots also makes them easy tar-

get of DoS attacks. In short, the honeypots have be-

come a single point of failure in the IPSs that use them

to collect information about attacks.

2. The great variety of OSs/applications, along with the

many different versions of the same software running

on different protected hosts is also creating some prac-

tical difficulties; a large number of honeypots may

be needed so that attacks against hosts running rare

OSs/applications, or a specific version of an applica-

tion/library will be covered. This may amount to a

non-trivial cost to set up and manage all these honey-

pots.

3. The passive nature of honeypots also makes them un-

suitable for studying attacks that require some user in-

teraction (e.g. a successful attack on web browsers

usually requires the victim to visit some “contami-

nated” web pages). Even though this problem may be

2For many host-based IPSs, it is infeasible to deploy the compo-

nent for collecting attack information in production systems; the detec-

tion/analysis process either requires a recompilation or extensive instru-

mentation/modification of the underlying system (e.g. address space lay-

out randomization [13, 7], selective transactional emulator [10]), or incur

an extremely high performance overhead (e.g. dynamic tain analysis [5, 8],

close-to-full emulation [2].) . In other word, the systems responsible for

the collection of attack information are effectively honeypots, systems with

no value other than being attacked and compromised

alleviated by techniques like [12], we believe it is not

trivial to generalize this technique.

4. To some system administrators, honeypots are still

hazardous components to be avoided. As a result, the

use of honeypots may harm the deployability of host-

based IPSs. The worries concerning the use of hon-

eypots may be further deepened by the analysis per-

formed in the honeypots after they are attacked: in

many host-based IPSs, buffered attack packets are re-

played in the honeypot in order to observe how a vul-

nerable system behaves before it is compromised.

In this paper, we propose the collaborative intrusion pre-

vention (CIP) framework as a solution to the problems that

spawned from the use of honeypots. Under the proposed

framework, every production system will be equipped to

detect and analyze attacks, and to generate and distribute

countermeasures against them. In other word, every host

under the collaborative intrusion prevention framework will

play both the role of a protected system and that of the hon-

eypot in traditional host-based IPSs.

With every host being effectively a honeypot, we will

have a large number of duplicates for the honeypot. As

a result, the honeypot will no longer be a single point of

failure in the IPS; the attacker can not evade the IPS by

avoiding certain systems in the target network. Attempt to

compromise any protected host may lead to the detection

of the attack, and the production of countermeasure against

the attack. Furthermore, each attempted attack will have

the same chance of being detected. Thus, the more attack

attempts are made, the higher the probability that a coun-

termeasure will be made available to stop the attack. The

threat of DoS against the honeypots also ceases to exist:

a DoS that “blinds” our scheme will also make the target

systems unavailable for being compromised. Finally, with

every host being capable of detecting and generating coun-

termeasures against attacks, the diversity in the software be-

ing run on the protected hosts is a much lesser issue. Even

though rare applications are covered only by the few hosts

running them, they are still not completely unprotected; on

the contrary, in traditional IPSs, if there are no honeypots

dedicated for these applications, attacks against them will

go unchecked.

2 Desirable Features

Before presenting the details about the proposed CIP

framework, we will lay out a set of features necessary for

any IPS that distributes the detection and countermeasure

generation capabilities over all protected systems:

1. Low Cost: In order to have a large number of partici-

pating host (i.e. honeypots for detecting and analyzing

attacks), the detection/analysis components executed

on participating hosts should have minimal installation

and running cost. Furthermore, each participant should

be able to determine the amount of overhead incurred.

2. Egocentric Participants: In addition to having low in-

stallation and operating cost, a good incentive should

also be provided to encourage participation. In par-

ticular, the cost incurred on participating hosts should

produce direct improvement on its security status, in

addition to the indirect benefit of receiving counter-

measures generated by other hosts. Furthermore, the

amount of contribution made by a host to the collabo-

rative scheme as a whole should be proportional to the

direct gain in security obtained by that individual host.

3. Simple Collaboration Scheme: The participating hosts

should collaborate in a loosely-coupled manner that

requires minimal, decentralized management. A

loosely-coupled collaboration will not only reduce the

cost of participation, but also improve the robustness

of the scheme, while decentralized administration of

the collaboration is necessary to avoid introducing a

single point of failure in the scheme.

4. Fault/Traitor Tolerant: We accept that some (hopefully

very few) hosts will be compromised before a coun-

termeasure against a new attack is available. Thus,

the collaborative scheme should be designed so that it

won’t be completely defenseless no matter how many

hosts have been compromised; the remaining hosts

should have a non-zero probability of outputting ef-

fective countermeasures to stop the attack. Further-

more, the scheme should also be robust against the

malicious behavior of the compromised host. In par-

ticular, the “countermeasures” output by compromised

hosts should not have significant adverse effect on the

uncompromised ones, and hosts that output large vol-

ume of bogus “countermeasures” should be identified.

3 Collaborative Intrusion Prevention

At the core of the CIP framework lies the random-

inspection-based intrusion detection proposed in [3]. By

“random-inspection-based intrusion detection”, we refer to

a class of intrusion detection techniques based upon the

monitoring mechanism proposed in [3], and the IDS pre-

sented in [3] is only one instance in the class. As we argue

in the next section, almost all random-inspection-based de-

tection technique are suitable for our proposed framework.

In addition to detecting attacks, the random-inspection-

based intrusion detection is also used in the CIP framework

to “patch” all protected systems. In fact, the main focus of

the countermeasure generation process is to extract infor-

mation that helps other hosts to detect the observed attack.

Before we present the details of this countermeasure gener-

ation process, we will first have a brief review on random-

inspection-based intrusion detection.3.1 Ba kground: Random Inspe tionRandom inspection (proposed in [3]) is a mechanism for

observing the behavior of user processes at random points

during their execution. It is based on the hardware facil-

ity called the performance counter (which can be found on

many CPUs) that generates a “performance-counter inter-

rupt” every k instructions executed in the user space. To

perform truly random inspection, we reset k to some ran-

dom value after each interrupt. In the following discussion,

we assume each thread has a separate performance-counter

value (i.e. the number of instructions before the next perfor-

mance counter interrupt); and this value is saved/restored at

thread context switches.

In a random-inspection-based IDS, we observe the be-

havior of the current process at every performance-counter

interrupt to determine if it has been compromised. We call

each of these occasions where we perform the intrusion de-

tection an inspection point, and the average frequency at

which inspections occur the inspection frequency. In the

prototype system in [3], we simply check at each inspec-

tion point whether the currently executed instruction lies

in code space or data space; any execution of data will be

considered as an intrusion. Obviously, this prototype can

only detect injected code attacks. However, we argue that

other classes of attacks can also be detected with random-

inspection-based IDSs. For example, we are currently ex-

tending our prototype to detect existing-code attacks (a.k.a

“ret-to-libc” attacks) by checking the stack contents at each

inspection.

Note that the non-deterministic manner in which inspec-

tions are performed means intrusion detection in a random-

inspection-based IDS is inherently probabilistic. Even

though some attacks can be detected with certainty, there

is generally a non-zero probability that the IDS will fail to

detect an attack, especially when the IDS is operating under

a low inspection frequency. As shown in [3], if inspections

occur (on average) every k instructions executed, and the

attacked process executes y instructions in an “illegal state”

(a state that allows a positive detection should an inspection

occurs), the probability that the attack will be detected isPd = 1� e� yk .

What makes the random-inspection-based intrusion de-

tection suitable for our CIP framework is its adjustable

performance overhead. The overhead of performing

random-inspection-based intrusion detection can be con-

trolled through the inspection frequency; as shown in the

experiments in [3], the lower the inspection frequency, the

smaller the performance impact will be. The probabilistic

intrusion detection mechanism also provides an elegant way

for hosts to collaborate. By simply performing the random-

inspection-based detection, a host can improve the overall

probability of detecting an attack; if n hosts are participat-

ing in the collaboration, the detection probability with all

hosts considered as a whole becomes 1 � e� yk �n (as op-

posed to the Pd given above).3.2 Countermeasure GenerationAs mentioned before, the main function of countermea-

sures under our framework is to improve the protected

hosts’ ability to detect imminent attacks with little extra

performance overhead. This goal can be easily achieved

by broadcasting the last inspection point before an attack

is detected. In the future, whenever the execution in a pro-

tected host reaches this point, the host will perform random-

inspection-based intrusion detection at a higher frequency

(as we’ll argue in the next section, this will significantly

improve the chance of stopping the attacks targeted by the

countermeasure). If no intrusion is detected after a fixed

number of inspections, the inspection frequency will re-

turn to its normal value. In summary, a host under the CIP

framework works as follows:

1. Detection: Before any attacks detected or countermea-

sures received, the host performs random-inspection-

based intrusion detection at a low frequency f1. After

each inspection, if no intrusion is detected, the “iden-

tity” of the inspection point, as well as the new random

value in the performance counter will be recorded, and

separate records will be kept for each thread in the sys-

tem.

2. Countermeasure Generation: Once an intrusion is de-

tected, the record for the previous inspection point and

the saved performance counter value of the thread in-

volved are retrieved. These are basically all the in-

formation needed for a countermeasure against the de-

tected attack. However, the information retrieved may

be refined to make the countermeasure more portable

among hosts. For example, the virtual address of the

last normal inspection point may be expressed as the

name of the containing module and the offset from the

beginning of the module. After all the necessary re-

finement, the countermeasure will be distributed to all

other hosts.

3. Patching: Upon receiving the countermeasure from an-

other host, the last normal inspection point identified

will be extracted, and a breakpoint will be inserted at

the corresponding instruction so that an exception will

be generated every time it is executed. A record will

also be established to associate the “marked” instruc-

tion with the performance counter value, k, given in

the countermeasure.

4. Stopping Attacks: When a thread executes any in-

struction “marked” in the patching stage, the execution

of the thread will enter the “alert mode”. Random-

inspection-based intrusion detection will then be per-

formed at a higher frequency f2. The execution in alert

mode lasts for k + 1f2 instructions (i.e. it ends after(k � f2 + 1) inspections), where k is the performance

counter value associated with the instruction that trig-

gers the alert mode execution. When the alert mode ex-

ecution ends, the inspection frequency of the involved

thread will return to f1. In addition to performing ran-

dom inspection at higher frequency, the execution in

the alert mode can also take measures to facilitate the

recovery should any attack be detected. For example,

modifications to critical resources can be delayed or

redirected, and only be committed if no intrusion is

detected during the alert mode execution (in [6], these

are called cordoning-in-space and cordoning-in-time

respectively).

One important observation about the proposed counter-

measure generation process is that it can be used with virtu-

ally any random-inspection-based detection technique. As

long as the analysis performed at each inspection point is

self-contained and does not depend on the results of pre-

vious inspections, the last normal inspection point and the

saved performance counter value can be used as an effective

countermeasure to enhance the defense on other systems. In

other word, our framework is not tied to any particular anal-

ysis performed at the random inspection points, but is suit-

able for any detection system that uses random inspection

to monitor processes.3.3 How do the Countermeasures StopAtta ks?Now, let’s consider the effectiveness of the countermea-

sures generated. Essentially, the countermeasures in our

framework identify execution points reached before the ac-

tual attacks occur. Let’s assume for the moment that all ex-

ploitations of a particular vulnerability will bring the victim

process to the “last normal inspection point” given in the

countermeasure before the control hijacking occurs. Fur-

thermore, the way we determine the duration of the alert

mode execution guarantees that the attacked thread will be

executing in “alert mode” when the control hijacking oc-

curs. Thus, it seems the high frequency inspection will very

likely detect the attack concerned.

However, note that we do not know how often the at-

tacked thread will be in an illegal state (a state where the

intrusion is detectable) while being in the alert mode. In

the worst case, if the detection which produced the coun-

termeasure occurs at the very first moment that the attack

executes in an illegal state, and if the execution that follows

quickly leaves this state, the alert mode execution will have

virtually no chance of detecting the attack. However, if we

assume the attacked process will remain in the illegal state

during the first 1f2 (or more) instructions executed following

the control hijacking, detection in the alert mode is guaran-

teed. From our analysis in [3], it seems not uncommon for

attacks to have such an initial period in which they are very

vulnerable to detection. Thus, the generated countermea-

sures should be effective in stopping attacks.

Now let’s revisit the assumption that all exploitations

of the same vulnerability will bring the execution to the

“last normal inspection point”. We believe this assumption

should be valid most of the time, especially if the execu-

tion point concerned is close to the point of control hijack-

ing. Even if the above assumption is not true, there should

be very few execution paths that can lead to a successful

exploitation of any given vulnerability. As a result, a few

countermeasures, each with a “last normal inspection point”

on a different execution path will be sufficient to stop all at-

tacks targeting a particular vulnerability.

4 Evaluation

After giving the details of the proposed framework, we

will evaluate it against the list of desirable features given in

Sect. II:

1. Low Cost: To participate under the CIP framework,

a host needs three pieces of software: the random-

inspection-based IDS, the patching component, and

the facility for broadcasting (and receiving) counter-

measures. While the broadcasting facility is pretty

standard among host-based IPSs, we will focus on the

first two components. The IDS under our framework

is basically a simple device driver to start up the ran-

dom inspection. Due to the self-containedness require-

ment, we expect the code for the analysis at inspection

points to be very simple also. Similarly, the patching

component is a light-weight device driver that looks

for the modules that need to be patched, inserts the

breakpoints, and handles any resulting breakpoint ex-

ceptions. In terms of the running cost, it is dominated

by the performance overhead incurred by the intrusion

detection, and can be limited to any chosen level by

keeping the inspection frequency sufficiently low.

2. Egocentric Participants: Under the CIP framework, the

only altruistic behavior required of a host is the distri-

bution of the generated countermeasure, which only

incurs a minimal cost. For the other activities per-

formed on behalf of the collaboration, the participating

hosts can benefit directly from them; both the intru-

sion detection and the countermeasure generation help

the host better defend itself. In fact, the overhead in-

curred from the intrusion detection (which is the major

running cost for participants) is directly related to the

chance that a host will stop an attack targeting itself.

3. Simple Collaboration Scheme: Except for the distri-

bution of generated countermeasures, hosts participat-

ing in the collaboration do not need to communicate

with each other. All hosts can perform exactly the

same task of random-inspection-based intrusion detec-

tion independently, and yet their contributions will add

up to improve the chance of detecting and stopping

an attack. In other word, the collaboration requires

no management except for maintaining the mechanism

for distributing countermeasures; there is no need to al-

locate/synchronize the work to/from different hosts. In

fact, as far as the countermeasure generation process is

concerned, there is no need for maintaining any mem-

bership information about the collaborating hosts.

4. Fault/Traitor Tolerant: The extremely loosely-coupled

collaboration scheme means that losing a small portion

of hosts will have very minor effect in the effective-

ness on the whole. In fact, the loss of any number of

hosts will not render the remaining hosts completely

defenseless, only the probability of stopping an attack

is affected. Furthermore, if the underlying detection

mechanism has zero false positive rate, bogus coun-

termeasures generated by a compromised host can do

no more harm than incurring extra performance hit on

other hosts. Of course, a large volume of such bogus

countermeasures can still lead to a DoS on all the par-

ticipating hosts. We will leave the defense against the

misbehavior of compromised hosts as future work.

5 Related Work5.1 Appli ation CommunitySimilar to the CIP framework, the Application Commu-

nity scheme in [11] distributes the work of detection and

countermeasure generation over a “community” of hosts

running the same application. As opposed to our loosely-

coupled, minimal-management collaboration scheme, the

Application Community employs a static division of labor;

all functions executed by the protected application are iden-

tified, and each participating host is assigned to monitor a

fixed set of functions. We believe this static work allocation

in [11] makes the Application Community scheme vulner-

able to very similar problems addressed in this work. First

of all, if the attackers can identify the hosts responsible for

monitoring the vulnerable functions exploited by their at-

tack, they can evade the Application Community scheme

just as they evade the traditional host-based IPSs. By mea-

suring the time it takes different hosts to process requests

that utilize different functions, it is not impossible for the

attacker to identify a superset of all hosts monitoring the

target function. Furthermore, since the work discovery is

done on a per-application manner, the Application Commu-

nity may also have poor scalability; multiple communities

have to be set up and managed, and each host has to join

multiple communities so that all its applications are pro-

tected. Finally, the Application Community appears to be

a very altruistic scheme; with vulnerabilities usually found

in the few rarely executed functions, we believe many hosts

under the Application Community scheme are monitoring

functions that will never be the target of any attack. In other

word, very few hosts will directly benefit from the monitor-

ing task performed for the Application Community scheme.

On the contrary, the CIP framework proposed does not have

any of the above problems, as we have shown in the previ-

ous section.5.2 LAIDS/LIDSOur work also borrows heavily from the LAIDS/LIDS

framework in [4]. Both the CIP framework and that in [4]

employs self-contained analysis to perform intrusion detec-

tion, so that the same mechanism can be used for both detec-

tion and patching. As a result of this similarity, the counter-

measure generation algorithms of the two frameworks are

almost the same. However, the detection used in [4] incurs a

high, untunable performance overhead, which makes it im-

possible to duplicate the detection and countermeasure gen-

eration capability on production systems. The LAIDS/LIDS

framework also provides no easy way of dividing the detec-

tion and countermeasure generation task so that the perfor-

mance overhead can be amortized among multiple hosts.

Thus, the major contribution of the CIP framework over

the work in [4] lies in the introduction of the random-

inspection-based IDS, which allows hosts to perform intru-

sion detection at any chosen level of performance overhead,

as well as an easy way of dividing and combining the task

of intrusion prevention among hosts.

6 Conclusions

In this work, we argue that the use of honeypots to col-

lect information about attacks is creating a single point of

failure in traditional host-based IPSs: once the honeypots

are identified, attackers can make the IPSs blind to their

attack by avoiding “interactions” with the honeypot. The

heavyweight analysis usually performed in these honeypots

also make them easy target for DoS attacks. Furthermore,

though dedicated honeypots are suitable for collecting in-

formation about attacks against critical network services, it

appears infeasible to scale up the approach to cover the great

variety of applications running in an entire network.

To overcome the above difficulties, we proposed the

Collaborative Intrusion Prevention (CIP) framework, under

which every host is capable of detecting attacks and gener-

ating countermeasures against them. In other word, every

host under our framework acts both as the honeypot and a

protected system in traditional host-based IPSs. Further-

more, by using random-inspection-based intrusion detec-

tion techniques, the CIP framework allows hosts to collab-

orate in intrusion prevention with minimum management.

Except for the distribution of countermeasures, there is no

need for hosts to communicate with each other, nor is there

the need to keep track of all the participating hosts. Each

host under our framework performs exactly the same intru-

sion detection and countermeasure generation task indepen-

dently, yet, the contribution from each host will automati-

cally add up to improve the system’s overall defense against

new attacks. With this massive duplication of the honeypot

and the simple collaboration scheme, there is no single point

of failure in our system. The system will remain functional

(though less effective) with the failure of any number of

hosts. Finally, the CIP framework also provides very good

incentive for hosts to participate; the performance overhead

suffered from performing random-inspection-based detec-

tion will strengthen the defense of a participant, in addition

to improving the security of all the other hosts.

References

[1] J. Bethencourt, J. Franklin, and M. Vernon. Mapping inter-

net sensors with probe response attacks. In Proceedings of

The 13th USENIX Security Symposium, Aug 2005.

[2] D. Brumley, J. Newsome, D. Song, H. Wang, and S. Jha.

Towards Automatic Generation of Vulnerability-Based Sig-

natures. In Proceedings of The 2006 IEEE Symposium on

Security and Privacy, Oakland, May 2006.

[3] S. P. Chung and A. K. Mok. On random-inspection-based in-

trusion dectection. In Proceedings of 8th International Sym-

posium on Recent Advances in Intrusion Detection (RAID),

Seattle, Sept 2005.

[4] S. P. Chung and A. K. Mok. The laids/lids framework for

systematic ips design. In Proceedings of the Fourth IEEE In-

ternational Information Assurance Workshop (IWIA 2006),

UK, Apr 2006.

[5] M. Costa, J. Crowcroft, M. Castro, A. Rowstron, L. Zhou,

L. Zhang, and P. Barham. Vigilante: End-to-end con-

tainment of internet worms. In Proceedings of 20th ACM

Symposium on Operating Systems Principles, Brighton, Oct

2005.

[6] R. Hu and A. K. Mok. Detecting unknown massive mail-

ing viruses using proactive methods. In Proceedings of 7th

International Symposium on Recent Advances in Intrusion

Detection (RAID), France, Sept 2004.

[7] Z. Liang and R. Sekar. Fast and automated generation of at-

tack signatures: A basis for building self-protecting servers.

In Proceedings of the 12th ACM Conference on Computer

and Communication Security (CCS 05), Virginia, Nov 2005.

[8] J. Newsome, D. Brumley, D. Song, J. Chamcham, and

X. Kovah. Vulnerability-specific execution filtering for ex-

ploit prevention on commodity software. In Proceedings of

13th Annual Network and Distributed System Security Sym-

posium (NDSS 06), San Diego, Feb 2006.

[9] M. Rajab, F. Monrose, and A. Terzis. Fast and evasive at-

tacks: Highlighting the challenges ahead. In Proceedings of

9th International Symposium on Recent Advances in Intru-

sion Detection (RAID), Hamburg, Sept 2006.

[10] S. Sidiroglou, M. Locasto, S. Boyd, and A. Keromytis. Build

a reactive immune system for software services. In Proceed-

ings of the USENIX Annual Technical Conference, 2005,

Apr 2005.

[11] S. Sidiroglou, M. Locasto, and A. Keromytis. Software self-

healing using collaborative application. In Proceedings of

13th Annual Network and Distributed System Security Sym-

posium (NDSS 06), San Diego, Feb 2006.

[12] Y. Wang, D. Beck, X. Jiang, R. Roussev, C. Verbowski,

S. Chen, and S. King. Automated web patrol with strider

honeymonkeys: Finding web sites that exploit browser vul-

nerabilities. In Proceedings of 13th Annual Network and

Distributed System Security Symposium (NDSS 06), Feb

2006.

[13] J. Xu, P. Ning, C. Kil, Y. Zhai, and C. Bookholt. Automatic

diagnosis and response to memory corruption vulnerabili-

ties. In Proceedings of the 12th ACM Conference on Com-

puter and Communication Security (CCS 05), Virginia, Nov

2005.