www.mcafee.com

White Paper April 2005

McAfee System Protection Solutions

System Call Interception

System Call Interception 2

www.mcafee.com

Table of Contents

Introduction 3 What Are System Calls? 3 How Does System Call Interception Work? 4 How Does Entercept Use System Call Intercept To Protect Servers? 5 What is the Performance Impact Associated With System Call Interception? 6 Summary 7

System Call Interception 3

Introduction

System call interception enables many of the McAfee Entercept’s proactive server protection capabilities. This paper addresses the following questions: • What are system calls? • How does system call interception work? • How does Entercept use system call interception to protect servers? • What is the performance impact associated with system call interception? What Are System Calls?

In order to protect the core of the operating system from damage by errant or malicious programs, modern operating system architectures strictly separate code executed by users from code executed by the operating system itself. To achieve this, modern processors include a mode bit that specifies whether the processor is executing kernel-mode code or user-mode code. If the mode bit is set, meaning user-mode code is executing, the processor hardware prevents all access to the kernel memory space. If a user-mode program attempts to access anything in the kernel memory space, the processor generates an illegal access exception. Thus, no user-mode program can access kernel memory directly. User-mode programs need to utilize the functionality provided by the kernel in order to access disk drives, network connections, shared memory, etc. Since the processor prevents direct access to kernel-mode functions, user-mode programs must use system calls, which form the only permitted interface between user-mode and kernel-mode. System calls expose all kernel functionality that user-mode programs require. System calls, such as fopen (which opens a file) are implemented inside the OS using a system call table. The system call table relates each system call to a specific function address within the OS kernel. Conceptually, the structure of a system call table is as follows:

The following C-language program illustrates how system calls are used:

#include <stdio.h> void main() { FILE* handle; handle = fopen("explorer.exe", "w"); }

When the above C program is executed, the processor encounters the fopen instruction, looks-up fopen in the system call table, and transfers control to the kernel-mode function at 0x0000A1F2. Figure 1 graphically illustrates this concept. Each system call has an entry in the system call table, which then points to a corresponding function in the OS kernel:

www.mcafee.com

System Call Interception 4

ProgramA

ProgramB

ProgramC

System Call Table

OS Kernel

fopen

User Mode

Figure 1- How System Calls Work Figure 1- How System Calls Work

How Does System Call Interception Work? How Does System Call Interception Work?

McAfee Entercept adjusts the entries in the system call table, pointing them at Entercept’s kernel-mode driver. This makes the above system call table look like this: McAfee Entercept adjusts the entries in the system call table, pointing them at Entercept’s kernel-mode driver. This makes the above system call table look like this:

This inserts Entercept into the command chain anytime a system call is made. System call interception allows Entercept to intercept and, if necessary, block accesses to any system resources by any program. If Entercept determines that an access should be allowed, the Entercept driver calls the original kernel function.

www.mcafee.com

System Call Interception 5

Program

A Program

B Program

C

System Call Table

fopen

User Mode

OS Kernel

Kernel Mode

Figure 2 - How System Call Interception Works

As shown in Figure 2, Entercept does not modify the kernel. It simply inserts itself into the command execution chain. Several commercial products, including most anti-virus products, use system call interception for various purposes. Entercept applies this well-understood technique to protecting servers from intrusions and misuse.

How Does Entercept Use System Call Intercept To Protect Servers?

Entercept’s behavioral rules determine whether a system call is allowed or blocked. The intricate details of all the rules are beyond the scope of this paper, but in general, Entercept asks three main questions when a system call is made: • What process is making the call? • What user authority is the process running under? • What is the call trying to access? One of Entercept’s many behavioral rules can be summarized as follows:

www.mcafee.com

System Call Interception 6

The following examples illustrate how Entercept enforces this behavioral rule: Case 1: The Web server process attempts to access the Web file “index.html.” Entercept intercepts the call to open the file and determines the following: • Process making the call: inetinfo.exe • User authority: IUSR_<machine> • Resource accessed: index.html. With the above information, Entercept determines that this call involves the Web server, running under the proper user authority, accessing a Web file. Since this matches Rule #1 above, Entercept allows the call. Case 2: An attacker uncovers a new, previously undiscovered Web server security vulnerability. This new vulnerability, like so many before it, allows a remote user to access arbitrary files on the Web server. The attacker exploits this vulnerability, attempting to access the file “credit_cards.mdb,” which contains the credit card numbers of the users of a particular e-commerce site. When the Web server attempts to access credit_cards.mdb,” the system call to open the file is intercepted. Entercept then determines: • Process making the call: inetinfo.exe • User authority: IUSR_<machine> • Resource accessed: credit_cards.mdb. Since “credit_cards.mdb” is NOT a Web file, this violates Rule #1. Entercept blocks the call to open the file and the exploit is prevented. Another Entercept behavioral rule, the converse of Rule #1 is:

The following example illustrates how Entercept enforces this behavioral rule: Case 3: An attacker obtains the Administrator account password to the Web server, using “social engineering.” He or she then logs in to the server as Administrator, opens the company’s homepage in Notepad and attempts to modify it. Entercept intercepts the call to modify the file “company_hompage.html” and determines the following: • Process making the call: notepad.exe • User authority: Administrator • Resource accessed: company_homepage.html. Since “company_homepage.html” is a Web file, but the process and user accessing it are not the Web server process and user, Entercept blocks the call to open the file and the defacement is prevented. What is the Performance Impact Associated With System Call Interception?

System administrators are rightly concerned about any performance impact introduced by security software loaded on their servers. Entercept’s impact on CPU utilization is minimal, and its impact on disk utilization and overall latency is negligible. McAfee Entercept runs entirely in memory, occupying less than 10 MB of RAM. As a result, it can make system call decisions quickly, without accessing the disk. As illustrated in the examples, the amount of information needed by Entercept to decide whether to allow or disallow a system call is small, and, consequently, the number of CPU cycles consumed is also small. In Entercept’s performance testing, and with our customers with the

www.mcafee.com

System Call Interception 7

www.mcafee.com

heaviest use profiles, the percentage of CPU utilized by Entercept has typically been 1-5%. The value of preventing known and unknown attacks far outweighs this minimal impact. Other host-security products use much more of the system resources. Traditional Host-based IPS can easily use 50% of the CPU if all its functionality is enabled. File-integrity monitors, such as Tripwire, can use large amounts of CPU and are constantly accessing the disk, causing poor disk throughput response latency. Entercept generally does not access the disk once it has been loaded. In these days of gigahertz processors, most servers are I/O bound, meaning they spend most of their time waiting on disk and network I/O and have CPU cycles to spare. Web servers, for example, do little processing on the data they serve: they spend most of their time accessing the disk and network resources and transferring data. Since Entercept does not usually access the disk, the overall system latency and response time is unaffected. Summary

Entercept uses system call interception and behavioral rules to protect servers from both known and unknown attacks, and does so without modification to the kernel. System call interception allows Entercept to intercept and, if necessary, block accesses to any system resources by any program. Because all programs running on servers must use system calls to access system resources, system call interception provides an excellent means protect system resources. Additionally, the performance impact of Entercept versus traditional IDS systems is minimal. Entercept blocks attempted attacks before they can compromise the system, which gives maximum security with minimal performance impact.

McAfee, Inc. 3965 Freedom Circle, Santa Clara, CA 95054

McAfee, and/or additional marks herein are registered trademarks or tradermarks of McAfee, Inc. and /or its affiliates in the U.S. and/or other countries. McAfee Red in connection with security is distinctive of McAfee brand products. All other registered and unregulated trademarks herein are the sole property of their respective owners. © 2005 McAfee, Inc. All Rights Reserved. 6-sys-ent-sci-003-0405