Policies and Mechanisms for Operating System Securityvg/talks/os-integrity/osi.pdf · Example 2: Windows Fu rootkit run_list next_task run_list next_task run_list next_task all-tasks:

Policies and Mechanisms for

Operating System Security

Vinod Ganapathy

[email protected]

Associate Professor of Computer Science

Rutgers, The State University of New Jersey

Layered computer system design

Modern computer systems are built using

layers of abstraction

2

Memory I/O devicesCPU

Hardware

Vinod Ganapathy - Policies and Mechanisms for OS Security




3


Hardware

Operating

System

SyscallsProcess

List

Kernel

Code

IDT

…





4


Hardware

Operating

System

SyscallsProcess

List

Kernel

Code

IDT

…

Utilities &

Librariesls, ps, &

bash utilitieslibc gcc …





5

User

app


Hardware

Operating

System

SyscallsProcess

List

Kernel

Code

IDT

…

User

app

Utilities &

Librariesls, ps, &


…


Fundamental principle in security

6

User

app


Hardware

Operating

System

SyscallsProcess

List

Kernel

Code

IDT

…

User

app

Utilities &

Librariesls, ps, &


…

The lower you go, the more control you have

Least control

Most control


7

User

app

Hardware

Operating

System

Utilities &

Libraries

Example: Malware detection


8

User

app

Hardware

Operating

System

Malware

detector

Utilities &

Libraries



9

User

app

Hardware

Operating

System

Malware

detector

Utilities &

Libraries


cat ps ls …TCB

Trusted

Layer


10

User

app

Hardware

Operating

System

Malware

detector

Utilities &

Libraries

But utilities may be compromised!

cat ps ls


11

User

app

Hardware

Operating

System

Malware

detector

Utilities &

Libraries


cat ps ls

1

1 Show me

file contents


12

User

app

Hardware

Operating

System

Malware

detector

Utilities &

Libraries


cat ps ls

2

1 Show me

file contents

2 Fake, benign

content


13

User

app

Hardware

Operating

System

Malware

detector

Utilities &

Libraries

Solution: Query the OS

System call API

1

1

Query with syscall

TCB


14

User

app

Hardware

Operating

System

Malware

detector

Utilities &

Libraries


System call API

1

2

Query with syscall

OS reads file

2

TCB


15

User

app

Hardware

Operating

System

Malware

detector

Utilities &

Libraries


System call API

1

2

3

Query with syscall

OS reads file

Returns true

file content

3

TCB


16

User

app

Hardware

Operating

System

Malware

detector

Utilities &

Libraries

OS detects malicious utilities too

System call API

A

B

cat

cat file

Read file

A Bdiff vs ?

A

B

TCB


17

User

app

Hardware

Operating

System

Malware

detector

Utilities &

Libraries

What if the OS is malicious?

System call API


18

Hardware

Operating

System

Malware

detector

Utilities &

Libraries

Rootkit = Malware that infects OS

System call API

Rootkits hide malware from detectors Long-term stealth

…


How does an OS get infected?

• Exploits of kernel vulnerabilities:

– Injecting malicious code by exploiting a memory

error in the kernel

• Privilege escalation attacks:

– Exploit a root process and use resulting

administrative privileges to update the kernel

• Social engineering attacks:

– Trick user into installing fake kernel updates

• Defeated via signature verification of kernel updates

• Trivial to perform prior to the Windows Vista OS

Vinod Ganapathy - Policies and Mechanisms for OS Security 19

How prevalent are rootkits?

• 2010 Microsoft report: 7% of all infections

from client machines due to rootkits[1]

• 2016 HummingBad Android rootkit:[2]

– Up to 85 million Android devices infected?

– Earns malware authors $300,000 each week

through fraudulent mobile advertisements

• Used in many high-profile incidents:

– Torpig and Storm botnets

– Sony BMG (2005), Greek wiretapping (2004/5)

20

[1] Microsoft Malware Protection Center, “Some Observations on Rootkits,” January 2010,

https://blogs.technet.microsoft.com/mmpc/2010/01/07/some-observations-on-rootkits

[2] CheckPoint Software, “From HummingBad to Worse,” July 2016,

http://blog.checkpoint.com/wp-content/uploads/2016/07/HummingBad-Research-report_FINAL-62916.pdf

21

User

app

Hardware

Operating

System

Malware

detector

Utilities &

Libraries

How can we detect rootkits?

System call API

Hypervisor (a.k.a. Virtual Machine Monitor)

Ask for help from the layers below

TCB


22

User

app

Hardware

Operating

System

Malware

detector

Utilities &

Libraries

How low can we go?

Hypervisor [Bluepill, Subvert]

TCB


23

User

app

Operating

System

Malware

detector

Utilities &

Libraries

How low can we go?

Hardware[Stuxnet, Trojaned ICs]???

TCBVinod Ganapathy - Policies and Mechanisms for OS Security

Today’s talk

• Detecting OS-level rootkit

infections (with some help

from the hardware)

• In two parts:

1. Policies: How do we know

that the OS is infected?

2. Mechanisms: How can the

hardware help us?

24

User

apps

Utilities &

Libraries

Operating

System

HardwareTCB


My contributions to OS security


2008 ---- Detecting rootkits using data structure invariants [ACSAC’08]

2008 ---- Re-architecting device drivers for better isolation [ASPLOS’08]

2009 ---- Securing OSes from malicious device drivers [ACSAC’09]

2010 ---- Exploring rootkits on smartphones [HotMobile’10]

2011 ---- Security/energy tradeoffs in rootkit detection [ACM MobiSys’11]

2012 ---- Rootkit detection on cloud platforms [ACM CCS’12]

2013 ---- Adapting multicore hardware for rootkit detection [TIFS’13]

2016 ---- Rootkit detection with ARM TrustZone [ACM MobiSys’16]

2016 ---- Rootkit detection using 3D-stacked hardware [Submitted]

Covered in today’s talk


2008 ---- Detecting rootkits using data structure invariants [ACSAC’08]

2008 ---- Re-architecting device drivers for better isolation [ASPLOS’08]

2009 ---- Securing OSes from malicious device drivers [ACSAC’09]

2010 ---- Exploring rootkits on smartphones [HotMobile’10]

2011 ---- Security/energy tradeoffs in rootkit detection [ACM MobiSys’11]

2012 ---- Rootkit detection on cloud platforms [ACM CCS’12]

2013 ---- Adapting multicore hardware for rootkit detection [TIFS’13]

2016 ---- Rootkit detection with ARM TrustZone [ACM MobiSys’16]

2016 ---- Rootkit detection using 3D-stacked hardware [Submitted]

Modus operandi


Analysis of memory snapshots obtained from target machine

User app

Physical Memory

Hardware

Operating

System

SyscallProcess

List

Kernel

Code

User app

Utilities &

Libraries

…

Target machinePotentially rootkit infected

TCB

Modus operandi



User app

Physical Memory

Hardware

Operating

System

SyscallProcess

List

Kernel

Code

User app

Utilities &

Libraries

…


Analysis machineTrusted

TCB

Modus operandi



User app

Physical Memory

Hardware

Operating

System

SyscallProcess

List

Kernel

Code

User app

Utilities &

Libraries

…


Analysis machineTrusted

TCB

Snapshot of

memory pages

…

Research questions

• RQ1: What algorithm should we use for memory snapshot analysis?

– Concerns our security policy

– Answer: Formulate rootkit detection problem as one of detecting invariant violations

• RQ2: How can we fetch memory pages without involving the target’s OS?

– Concerns our mechanism

– Answer: Leverage hardware advances


Research questions








Example 1: Linux Adore rootkit

int main()

{

open(…)

...

return(0)

}

sys_open(...)

{

...

}

sys_open

System call table

32

OS kernelUser app



int main()

{

open(…)

...

return(0)

}

sys_open(...)

{

...

}

evil_open(...)

{

malicious();

sys_open(...)

}

evil_open

System call table

33

OS kernelUser app



int main()

{

open(…)

...

return(0)

}

sys_open(...)

{

...

}

evil_open(...)

{

malicious();

sys_open(...)

}

evil_open

System call table

34

OS kernelUser app


Violated: Function pointer values in

system call table should not change

Example 2: Windows Fu rootkit

run_list

next_task

run_list

next_task

run_list

next_task

all-tasks: Used for process accounting

run-list: Used by the scheduler to select processes for execution

Process A Process B Process C

35Vinod Ganapathy - Policies and Mechanisms for OS Security


run_list

next_task

run_list

next_task

run_list

next_task

run_list

next_task



Hidden processProcess A Process B Process C



run_list

next_task

run_list

next_task

run_list

next_task

run_list

next_task



Hidden processProcess A Process B Process C


Violated: run-list ⊆ all-tasks

38

Urandom

Entropy Pool

(128 bytes)

Secondary

Entropy Pool

(128 bytes)Primary

Entropy Pool

(512 bytes)

/dev/random

/dev/urandom

External Entropy

Sources


Example 3: Kernel PRNG corruptor

39

Urandom

Entropy Pool

(128 bytes)

Secondary

Entropy Pool

(128 bytes)Primary

Entropy Pool

(512 bytes)

/dev/random

/dev/urandom

External Entropy

Sources


Example 3: Kernel PRNG corruptor

Attack: Modify coefficients of polynomials used to stir the entropy pools. Weaken quality of random numbers

40

Urandom

Entropy Pool

(128 bytes)

Secondary

Entropy Pool

(128 bytes)Primary

Entropy Pool

(512 bytes)

/dev/random

/dev/urandom

External Entropy

Sources


Example 3: Kernel PRNG corruptorViolated:

• poolinfo.tap1 ∈ {26, 103}




• poolinfo.tap5 == 1

Key technical challenges

• Vast attack surface:

– The kernel has thousands of data structures

• Specifying correctness properties:

– Infeasible to supply properties manually


Key technical challenges

• Vast attack surface:

– The kernel has thousands of data structures

– Solution: Use memory snapshots to analyze all

kernel data structures

• Specifying correctness properties:

– Infeasible to supply properties manually

– Solution: Infer invariants by adapting methods

from dynamic program analysis


Offline training phase


User app

Physical Memory

Hardware

Operating

System

SyscallProcess

List

Kernel

Code

User app

Utilities &

Libraries

…

Clean reference machine Not rootkit infected

Analysis machine

TCB

Snapshot of

memory pages

…

Offline training phase


User app

Physical Memory

Hardware

Operating

System

SyscallProcess

List

Kernel

Code

User app

Utilities &

Libraries

…

Clean reference machine Not rootkit infected

Analysis machine

TCB

Snapshot of

memory pages

…

Inference

Invariant DB

Online enforcement phase


User app

Physical Memory

Hardware

Operating

System

SyscallProcess

List

Kernel

Code

User app

Utilities &

Libraries

…

Target machine Potentially rootkit infected

Analysis machine

TCB

Snapshot of

memory pages

…

Inference

Invariant DB

Compare

Prior work on inferring invariants

• Daikon: Dynamic program analysis tool to

infer data invariants [Ernst et al., 2000]

T1, T2, … , Tn = Traces from execution of a target

program, recording variable values

Values/properties invariant in in T1, T2, … , Tn

(e.g., foo == 5, foo ≤ bar + baz)


Adapting to memory snapshots

S1, S2, … Sn = Snapshots from reference machine

for (i ∈ [1 .. n]) {

Di = Reconstruct kernel data structures in Si

}

D1, D2, … , Dn


Data Structure Invariants

• sys_open == 0x3ee210fb

• run-list ⊆ all-tasks



• …

Reconstructing data structures


(3) Snapshot of

memory pages

…

(2) Entry-points into the kernel(1) Kernel data structure

type definitionsffffe400 init_task

ffffe410 phys_base

ffffe420 loops_per_jiffy

...

struct task_struct {...}

struct list_head {...}

struct siginfo {...}

...



(3) Snapshot of

memory pages

…



ffffe410 phys_base


...




...

struct task_struct {

int state;

int counter;

struct task_struct *next;

...

}

00000001

0034ea23

ac3456bc

...

Data at 0xffffe400Definition of task_struct

init_task.state = 1init_task.counter = 0x34ea23

init_task.next = 0xac3456bc



(3) Snapshot of

memory pages

…



ffffe410 phys_base


...




...

struct task_struct {

int state;

int counter;

struct task_struct *next;

...

}

Data at 0xac3456bcDefinition of task_struct

init_task.next.state = 0init_task.next.counter = 0x56ae71

init_task.next.next = 0xbf6723ae

00000000

0056ae71

bf6723ae

...

Experimental evaluation

① How effective is our approach at detecting

rootkits? i.e., what is the false negative rate?

②What is the quality of automatically-

generated invariants? i.e., what is the false

positive rate?

Target machine ran Linux 2.4.20. Used same

machine as reference machine as well.


Training phase

• Ran LMBench on reference machine:

– Collected 15 complete memory snapshots

(including reboots): took 25 minutes

– Inferred invariants using Daikon in 31 minutes

• Inferred 236,444 invariants across the

memory snapshots


False negative evaluation

• Conducted experiments with 23 Linux

rootkits:

– 14 rootkits from PacketStorm

– 9 advanced rootkits, discussed in the literature

• Installed rootkits one at a time on the target

machine and tested effectiveness of our

approach at detecting the infection


54November 30, 2009

Rootkit name Data structures affected Detected?

1. Adore-0.42 System call table (from PacketStorm)

2. All-root System call table (from PacketStorm)

3. Kbd System call table (from PacketStorm)

4. Kis-0.9 System call table (from PacketStorm)

5. Linspy2 System call table (from PacketStorm)

6. Modhide System call table (from PacketStorm)

7. Phide System call table (from PacketStorm)

8. Rial System call table (from PacketStorm)

9. Rkit-1.01 System call table (from PacketStorm)

10. Shtroj2 System call table (from PacketStorm)

11. Synapsys-0.4 System call table (from PacketStorm)

12. THC Backdoor System call table (from PacketStorm)

13. Adore-ng VFS hooks/UDP recvmsg (from PacketStorm)

14. Knark-2.4.3 System call table, proc hooks (from PacketStorm)

15. Disable Firewall Netfilter hooks (Baliga et al., 2007)

16. Disable PRNG VFS hooks (Baliga et al., 2007)

17. Altering RTC VFS hooks (Baliga et al., 2007)

18. Defeat signature scans VFS hooks (Baliga et al., 2007)

19. Entropy pool struct poolinfo (Baliga et al., 2007)

20. Hidden process Process lists (Petroni et al., 2006)

21. Linux Binfmt Shellcode.com

22. Resource waste struct zone_struct (Baliga et al., 2007)

23. Intrinsic DOS int max_threads (Baliga et al., 2007)

False positive evaluation

• Ran a benign workload for 42 minutes

– Copying Linux kernel source code

– Editing a text document

– Compiling the Linux kernel

– Downloading eight videos from Internet

– Perform file system operations using the IOZone

benchmark

• Only 82 out of 236,444 invariants spuriously

violated during execution

– Can be improved with more training


Current status of this approach

• Adopted widely in community for memory

snapshot-based rootkit detection

• Has led to numerous follow-on projects by

other research groups (200+ citations)

– More accurate data structure reconstruction

– Better ways to express invariants

– Improving accuracy of inferred invariants


Research questions








Tamper

resistance

Performance

isolation

Snapshot

consistency


Snapshot acquisition mechanism

1 2 3

Tamper

resistance

Performance

isolation

Snapshot

consistency


Tamper resistance

Target should not interfere with snapshot acquisition

Tamper

resistance

Performance

isolation

Snapshot

consistency

Virtualization


Tamper resistance


Physical Memory

Virtual Hardware

Operating

System

Hypervisor

• Hypervisor can fetch memory

from virtual machine without

OS involvement

Tamper

resistance

Performance

isolation

Snapshot

consistency

Virtualization

Co-processor


Tamper resistance


Physical Memory

Hardware

Operating

System• Co-processor uses DMA

• OS on target involved in

DMA setup

• Malicious OS can hide

portions of memory with

malicious content

Tamper

resistance

Performance

isolation

Snapshot

consistency

Virtualization

Co-processor


Performance isolation

Do not halt the target during snapshot acquisition

• Necessary for situations where continuous

snapshot acquisition is necessary

• Hypervisor-based acquisition requires pausing

the virtual machine

• Co-processor can operate in concert with target

Tamper

resistance

Performance

isolation

Snapshot

consistency

Virtualization

Co-processor


Snapshot consistency

Snapshot should faithfully represent target’s state at a given instant in time

Physical Memory

Hardware

Operating

System F1 F2…T

F1 F2…T + δ

NULL

CONSISTENT

CONSISTENT

Tamper

resistance

Performance

isolation

Snapshot

consistency

Virtualization

Co-processor


Snapshot consistency

Snapshot should faithfully represent target’s state at a given instant in time

Physical Memory

Hardware

Operating

System F1 F2…

T T + δ

Co-processor cannot pause target.

Snapshot may contain pages

obtained at different instants in time

INCONSISTENT

Tamper

resistance

Performance

isolation

Snapshot

consistency

Virtualization

Co-processor

3D-stacking


Our contribution

• Based on 3D-stacked technology:

– New hardware manufacturing technology that

“stacks” memory/processing logic atop the chip

– Early versions of 3D-stacked hardware already

on market, e.g., AMD Radeon series

3D-stacked chip


Picture courtesy of AMD

CPU and

Memory controller

On-chip memory

(high-speed)

3D-stacked chip


Picture courtesy of AMD

CPU and

Memory controller

On-chip memory

(high-speed)

Memory bus

Traditional (off-chip)

DRAM memory

Our use of 3D-stacking

• On-chip DRAM treated as a page-granularity

cache of off-chip DRAM memory

– Every address accessed by the CPU will result in

the page frame being fetched to on-chip DRAM


Cache of off-chip

DRAM memoryOff-chip DRAM

On-chip DRAM

Memory controller

Crypto logic

CPU

Memory bus

Triggering snapshot acquisition


On-chip DRAM

Memory controller

Crypto logic

CPU

Trigger = Device that communicates to the

CPU to enter snapshot acquisition mode: Physical device attached to South/NorthBridge

that sends a non-maskable interrupt

NIC with Wake-on-LAN-like feature

Off-chip DRAM

Memory bus

Snapshot acquisition mode


Memory controller

Crypto logic

CPU

• Memory controller splits on-chip DRAM into

two parts: Cache of off-chip DRAM memory

Copy-on-Write (CoW) area

CacheCoW

1

Off-chip DRAM

Memory bus



Memory controller

Crypto logic

CPU

• Hardware brings one page frame of off-chip

DRAM at a time to on-chip DRAM cache

CacheCoWFiFi

2

Off-chip DRAM

Memory bus



Memory controller

Crypto logic

CPU

• Crypto logic digitally signs contents of page: Random nonce used to prevent replay attacks

Same nonce used for all pages in snapshot

CacheCoW

+ Page#

+ Rand#Fi

3

Off-chip DRAM

Memory bus



Memory controller

Crypto logic

CPU

• Hardware instructs OS to write signed page

to external medium: Even if OS is infected, it cannot cheat, since

integrity of page is protected by the hardware

CacheCoW

Disk

4

+ Page#

+ Rand#Fi

Off-chip DRAM

Memory bus

• CPU continues to execute concurrently: If it writes to page Fj that has not yet been

copied Memory controller makes a copy of the

original page in the Copy-on-Write area

When hardware ready to snapshot Fj, copy

created from Copy-on-Write area



Memory controller

Crypto logic

CPU

Memory bus

CacheCoW

5

Fj

Off-chip DRAM

Fj

At conclusion of acquisition

• Consistent snapshot of off-chip memory at

instant when acquisition was initiated

• Snapshot is tamper-resistant even to a

corrupted OS

• Obtained without pausing target machine


+ 0

+ RF0

+ 1

+ RF1

+ N

+ RFN

…

Security analysis

Malicious OS cannot:

Corrupt pages in snapshot: Integrity

Hide pages from snapshot: Completeness

Replay old snapshot: Freshness

“Clean” itself during snapshot acquisition

because Copy-on-Write stores original page:

External control


+ 0

+ RF0

+ 1

+ RF1

+ N

+ RFN

…

Evaluation

• Atop 3D-stacked hardware emulator

• Evaluated:

– Impact of 3D-stacked memory available

– Effectiveness of performance-isolation claim

• Used canneal, memcached, graph500, mcf

– Time to procure full snapshot of memory:

• ~10-100 seconds, depending on external medium

– Complexity of hardware modifications:

• Evaluated using CACTI and Aladdin

• Negligible area/energy overheads


Evidence of performance isolation


Applications make progress as long as space available

in on-chip CoW area. Space in CoW area dependent on

speed of external medium that stores snapshot

Other research projects…

• Improving cloud platform security [ACSAC’08, RAID’10, CCS’12a, SOCC’14]

• Security for mobile devices (and other IoT devices)[MobiSys’11, TIFS’13, MobiSys’16]

• Hardware support for software and system security [CCS’08, ECOOP’12a, TIFS’13, MobiSys’16, RU-DCS-TR724]

• Web application and Web browser security [ACSAC’08, ACSAC’09, ECOOP’12a, ECOOP’12b, ECOOP’14, FSE’14]

• Tools for cross-platform mobile app development [ICSE’13, ASE’15]

• Retrofitting legacy software for security [CCS’05, Oakland’06, ASPLOS’06, ICSE’07, CCS’08, CCS’12b]

• Proofs of security for retrofitting transformations[Work in progress]

79

Generic theme: Computer Systems Security


A big thank you to my students

80

Graduated PhDs

• Dr. Mohan Dhawan (IBM Research India)

• Dr. Saman Zarandioon (Amazon.com)

• Dr. Shakeel Butt (NVidia now at Google)

• Dr. Liu Yang (HP Labs now at Baidu)

• Dr. Rezwana Karim (Samsung Research America)

• Dr. Amruta Gokhale (Teradata)

Former Postdocs

• Dr. Arati Baliga (AT&T Security Labs)

Graduated MS students

• Jeffrey Bickford (AT&T Research)

• Yogesh Padmanaban (Microsoft)

Current PhD students

• Jay P. Lim, Hai Nguyen, Daeyoung Kim.


URL: http://www.cs.rutgers.edu/~vinodg

Email: [email protected]

Documents

Policies and Mechanisms for Operating System Securityvg/talks/os-integrity/osi.pdf · Example 2: Windows Fu rootkit run_list next_task run_list next_task run_list next_task all-tasks: