Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
Matryoshka: Strengthening Software Protection via Nested Virtual Machines
Sudeep Ghosh (Microsoft Research) Jason D. Hiser (University of Virginia)
Jack W. Davidson (University of Virginia)
The Problem
2
0101110 00111001010 001 01011001000110001110110 0101110110110010111010 101101000101101111111 100010101101100111110 01010111001110010111 1 11111111111111110
• Disassembler • Debuggers • Static Analyzers
• Coverage tools • Simulators
Software is now used to perform critical functionality.
• Banks • Power grids • Medical software • Transportation systems • Internet of Things Researchers must continually upgrade and
enhance software protection approaches
Threat Model
n Software is created in a secure environment. n White-box attack model
n Adversary has access to multiple tools including debuggers, simulators and emulators.
n They can modify the OS to return inaccurate information. n As such, the adversary can inspect, modify or forge any
information. n Given enough time and resources, the adversary can succeed in
manually inspecting and modifying programs. n Boaz Barak , Oded Goldreich , Russell Impagliazzo , Steven Rudich , Amit
Sahai , Salil P. Vadhan , Ke Yang: On the (Im)possibility of Obfuscating Programs, Proceedings of the 21st Annual International Cryptology Conference on Advances in Cryptology, 2001.
n However, most attacks use algorithmic solutions to disable security features in programs.
3
Virtualization
Virtualizing Software (e.g. Xen)
Operating System
Hardware
Applications
System-level VM
Hardware
Process-level VM
Applicationn Application1
Operating System
Virtualizing Software
(e.g. Strata)
Virtualizing Software
(e.g. Strata)
4
Strata
n Infrastructure designed for building process-level virtualization systems
n Designed with extensibility, portability, and application-independence in mind
n Implement new systems by customizing the VM
n Binary only n No source code required n Can be applied to any
application regardless of source language, compiler used, libraries used, etc.
n Provides common services necessary for software dynamic translation
5
Process-level Virtualization
Context Capture
Context Switch Next PC
Translate Decode Fetch
New Fragment
Finished?
Dynamic Translator
Cached? New PC
Application Binary
System Start (first PC)
F$
Direct Conditional branch Trampoline Non-control instruction
PC
7
See: Hiser et al., Fragment Cache Construction Policies for Software Dynamic Translation Systems, VEE 2006.
Strata
0
0.5
1
1.5
2
2.5
Ove
rhea
d N
orm
aliz
ed t
o N
ativ
e R
un
DynamoRIO
PIN
HDTrans
Strata
8
Strata Related Work n SDT
n Evaluating Indirect Branch Handling Mechanisms in Software Dynamic Translation Systems. CGO 2007.
n Evaluating Fragment Construction Policies for Software Dynamic Translation Systems. VEE 2006.
n Retargetable and Reconfigurable Software Dynamic Translation. CGO 2003. n Obfuscation and anti-tamper
n What’s the PointISA? IH 2014. n Replacement Attacks against VM-protected Applications. VEE 2006. n A Secure and Robust Approach to Software Tamper Resistance. IH 2010
n Security n ILR: Where did my gadgets go? Oakland 2012. n Secure and Practical Defense Against Code-injection Attacks Using Software
Dynamic Translation. VEE 2006. n Safe Virtual Execution Using Software Dynamic Translation, ACSAC 2002.
9
Matryoshka: Nested PVMs
10
APPLICATION
PVM2 PVM4
dPVM1 PVM3
G1
G2
G3
G4
Software Protection via Virtualization
Application
Protection Scheme (Guards,
encryption)
Builder (Diablo)
Protected Binary
11
PVMs
PVM1 PVM2 PVM2
Nested PVMs Disk Image
12
F$ (SC1)
Strata1
Strata2
Translated Application Code
Memory
X86 (Hardware)
F$ (SC2)
Translated Application Code
Translated Application Code
Evaluation
F$ Diversity
13
Software Cache Addresses App
Strata2
n Use compression ratio as a proxy for diversity and obfuscation.
n Single PVM: 149; N-PVM: 15.63
Evaluation
Cyclomatic Complexity n Developed by McCabe in 1976 as a measure
of software complexity (TSE Vol. 2, No. 4) n M = E – N + 2P
14
Benchmark CC for PVM CC for N-PVM Increase
176.gcc 1,604 80,109 49X
181.mcf 351 9828 27X
256.perlbmk 803 32,903 40X
179.Art 181 5,130 27X
Evaluation
Run-time Overhead n With a nesting level of two, the base run-time
overhead was 35X. n The problem is trampoline patching (i.e., self-
modifying code), which causes excessive F$ flushes.
15
Nested PVMs Disk Image
17
F$ (SC1)
Strata!
Strata2
Translated Application Code
Memory
X86 (Hardware)
F$ (SC2)
Translated Application Code Translated Application Code
Translated Application Code
Translated Application Code
Super Patching
n When Stratan patches a trampoline, the patch information is sent to Stratan-1
n When a patched (in F$2) target block is translated to F$n-1 by Stratan-1, Stratan-1 patches its F$ (F$n-1), thereby avoid the F$ flush
18
Super Patching Overhead
19
0
0.5
1
1.5
2
2.5
3
3.5
4
164
.gzip
175
.vpr
176
.gcc
181
.mcf
197
.par
ser
253
.per
lbm
k
256
.bzip
2
300.
twol
f
177
.mes
a
179
.art
183
.equ
ake
188
.am
mp AV
G
Strata N-PVM with Super-patching
Perf
orm
ance
Ove
rhea
d (n
orm
alize
d to
nat
ive)
Related Work
n Collberg and Nagra [Pearson 2006] provide an excellent overview of the area
n Anckaert et al [DRM 2006] showed the promise of virtualization for software protection
n Themida [1] and VMProtect [2] use interpreted virtual machines for software protection. Nested VMs apply to them as well.
n Anckaert et al [QoP 2007] discuss metrics for metrics for software protection
20
Summary
n Nested PVMs can significantly increase the complexity of software that is the target of crackers
n More research is needed to determine when and how to apply nested VMs to software to balance run-time performance and the strength of the protection provided
21