Embed Size (px)
Citation preview
Controlling System Calls and Protecting Application Data
in Virtual Machines
Koichi Onoue* Yoshihiro Oyama** Akinori Yonezawa*
* The University of Tokyo** The University of Electro-Communications
Protection for Applications
• Security systems has been widely applied to provide secure computing environments– Sandboxing systems
– Intrusion detecion/prevension systems (IDSes/IPSes)
– Anti-virus tools
OS kernel
Security system
Applicationcontrol
Confidential data
Security Systems Can Also Be Compromised！
• Security systems and the other applications are running in the same execution space
Application 1
OS kernel
Security system
Application 2
Confidential data
Application 1 was compromised !
control
Advantages of Virtual Machine Monitor(VMM) in Terms of Security
• VMM can provide strong isolation between VMs– VMM prevents a
compromised VM from attacking to the other VMs
• VMM can control access to physical computing resources such as a physical memory and a disk– VMM is running at the
higher privileged level than VMs Hardware
Virtual machine monitor (VMM)
Virtual machine(VM)
OS （Guest OS）
Virtual machine(VM)
OS（Guest OS）
Application Application
Our Goal
• Enhancing application security from outside of VMs– In cooperation with VMM, security systems
control behaviors of application and protect application data
VMM
Control VM Target VM
ApplicationSecuritysystem
Our Approach
• Controlling system calls from outside of target VMs– We confine behaviors of application processes
• Controlling memory and file operations related with target applications in a VMM and a control VM– We prevent non-target programs from leaking target
application data and tampering with them
Our system control only the target application which users specify
We extend a para-virtualization version of Xen
Controlling System Calls fromOutside of target VMs
Comparison between “w/o VMM” and “w/ VMM”
Securitysystems
(“w/o VMM”)
Security system in cooperation with VMM
(“w/ VMM”)
Attack against security systems
×Not hard
○Hard
Execution states obtained by
security systems
○OS-level
×Hardware-level
Securitysystems
Security systems in cooperation with VMM
Attack against security systems
×Not hard
○Hard
Execution states obtained by
security systems
○OS-level
×Hardware-level
Goal for Controlling System Calls
Our goal
Semantic gap
Approach for Controlling System Calls
• Controlling system calls from outside of VMs– Using information on target OSes
• Conforming to security policies
VMM
Control VM Target VM
applicationSecuritysystem
Security policy
Bridging the Semantic Gap
• What a VMM can observe– Events ：Privileged instructions, Interrupts, …
– Execution States：Registers, Memory pages, …
• What security systems require– Events ：System calls, …
– Execution states： Process ID, System call number, …
Semantic gap
Security Policy
• Specifies invoked system calls withpattern matching
...open default: allow
fileEq(“/etc/passwd”)or filePrefixEq(“/etc/cron.d”)
deny(EPERM)...
Controlling Memory and File Operations Related with Application Data
Goal for Protecting Application Data
• Prevent compromised programs from leaking target data and tampering with them– Attackers use ptrace system call and kernel
modules, etc.
VM
Target
OS kernel Application
Confidential data
Compromised program
Leaking
Tampering
Memory
Target
Virtual disk
Approach for Protecting Application Data
• Hiding “real” application data on a memory and a virtual disk from compromised programs– Compromised programs include target OS
• Application data on a memory– Code region, data region, stack region, etc.➔VMM multiplexes target physical pages
• Application data on a virtual disk– Executables, configuration files, etc.➔ Control VM manages them
OS Memory Management
Virtual address space
Physical address space
Application
Application
OS addresstranslation
VMM Memory Management
Target VM virtual address space
（gVA）
Target VM physical address
space（gPA）
Application
application
gVA → hPA
VMM address translation
gPA → hPA
VMM physical address space
（hPA）
Application
17
Approach for Protecting Application Memory Data (1/2)
• According to the operational mode, a VMM switches accessible physical pages
– Overshadow[Chen et al., 2008]– [Rosenblum et al.,2008]
Target VM virtual address space
（gVA）
Target VMphysical address
space（gPA）
VMM physical address space
（hPA）
ApplicationApplication
“Real” data
Dummy data
Accessible page at user-level
Accessible page at kernel-level
Approach for Protecting Application Memory Data (2/2)
• When the operational mode were changed,a VMM switch the page tables– Exception/Interrupt handling
– System call handling
Approach for Protecting Application File Data (1/5)
• Control VM manages “real” target files– Executables, configuration and data base files, etc.
– Security policy specifies target files
VMM
Control VM Target VM
Security system
Security policy
“Real” configuration file
Application
“Real” executable
Dummy configuration
fileDummy
executable
Approach for Protecting Application File Data (2/5)
VMM
Control VM Target VM
Security system
Configuration file
Application
MemoryApplication read a configuration file
Approach for Protecting Application File Data (3/5)
VMM
Control VM Target VM
Security system
Configuration file
Application
Memory
VMM intercepts “read” system call
Approach for Protecting Application File Data (4/5)
VMM
Control VM Target VM
Security system
Configuration file
Application
Security system emulates “read”
Memory
Approach for Protecting Application File Data (5/5)
VMM
Control VM Target VM
Security system
Configuration file
Application
Memory
VMM notifies application of a result of “read”
Conclusion
• We have proposed security system that enhances applications inside target VMs– Controlling of application behaviors
• Controlling of system calls from outside of target VMs
– Protecting application data on a memory anda virtual disk
• Application memory data: VMM multiplexes target physical pages
• Application file data: Control VM manages them
Fin
