Unikernels as ProcessesDan Williams, Ricardo Koller (IBM Research)

Martin Lucina (robur.io/Center for the Cultivation of Technology)Nikhil Prakash (BITS Pilani)

• An application linked with library OS components

• Run on virtual hardware (like) abstraction

• Language-specific

• MirageOS (OCaml)

• IncludeOS (C++)

• Legacy-oriented

• Rumprun (NetBSD-based)

• Can run nginx, redis, node.js, python,etc..

What is a unikernel?

VM

2

Why unikernels?

• Lightweight• Only what the application needs

• Isolated• VM-isolation is the “gold standard”

• Well suited for the cloud• Microservices• Serverless• NFV

3

VM

Virtualization is a mixed bag

• Good for isolation, but…

• Tooling for VMs not designed for lightweight (e.g., lightVM)• How do you debug black-box VMs?• Poor VM performance due to vmexits• Deployment issues on already-virtualized infrastructure

4

Why not run unikernels as processes?

• Unikernels are a single process anyway!

• Many benefits as a process• Better performance• Common tooling (gdb, perf, etc.)• ASLR• Memory sharing• Architecture independence

• Isolation by limiting process interface to host• 98% reduction in accessible kernel functions

5

Process

Outline

• Introduction• Where does unikernel isolation come from?• Unikernels as processes• Isolation evaluation• Performance evaluation• Summary

6

Isolation: definitions and assumptions

• Isolation: no cloud user can read/write state or modify its execution

• Focus on software deficiencies in the host• Code reachable through interface is a metric for

attack surface

• We trust HW isolation (page tables, etc.)

• We do not consider covert channels, timing channels or resource starvation

7

Host Kernel

app

Unikernel architecture

8

Linux

monitor process (e.g., ukvm)

I/O devices

KVM

VT-x

• ukvm unikernelmonitor• Userspace process• Uses Linux/KVM

• Setup and loading• Exit handling

Unikernel architecture

9

Linux

monitor process (e.g., ukvm)

I/O devices

KVM

VT-x

Setup

Set up I/O fds

1

• ukvm unikernelmonitor• Userspace process• Uses Linux/KVM

• Setup and loading• Exit handling

Unikernel architecture

10

Linux

monitor process (e.g., ukvm)

Virtual CPU context

I/O devices

KVM

VT-x

Setup

Set up I/O fds

1

Load unikernel

2

• ukvm unikernel monitor• Userspace process• Uses Linux/KVM

• Setup and loading• Exit handling

Unikernel architecture

11

Linux

monitor process (e.g., ukvm)

I/O devices

KVM

VT-x

Setup

Set up I/O fds

1

Exit handling

Load unikernel

2

I/O

• ukvm unikernelmonitor• Userspace process• Uses Linux/KVM

• Setup and loading• Exit handling

Virtual CPU context

Unikernel isolation comes from the interface

• 10 hypercalls

• 6 for I/O• Network: packet level• Storage: block level

• vs. >350 syscalls

12

Hypercallwalltime

puts

poll

blkinfo

blkwrite

blkread

netinfo

netwrite

netread

halt

Observations

• Unikernels are not kernels!• No page table management after setup• No interrupt handlers: cooperative scheduling and poll

• The ukvm monitor doesn’t “do” anything!• One-to-one mapping between hypercalls and system calls

• Idea: maintain isolation by limiting syscalls available to process

13

Unikernel as process architecture

14

Linux

tender process

I/O devices

• Tender: modified ukvmunikernel monitor• Userspace process• Uses seccomp to

restrict interface

• Setup and loading

Unikernel as process architecture

15

Linux

tender process

I/O devices

Setup

Set up I/O fds

1

• Tender: modified ukvmunikernel monitor• Userspace process• Uses seccomp to

restrict interface

• Setup and loading

Unikernel as process architecture

16

Linux

tender process

I/O devices

Setup

Set up I/O fds

1

Load unikernel2

• Tender: modified ukvmunikernel monitor• Userspace process• Uses seccomp to

restrict interface

• Setup and loading

Unikernel as process architecture

17

Linux

tender process

I/O devices

Setup

Set up I/O fds

1

Load unikernel2

Configure seccomp

3

• Tender: modified ukvmunikernel monitor• Userspace process• Uses seccomp to

restrict interface

• Setup and loading

Unikernel as process architecture

18

Linux

tender process

I/O devices

Setup

Set up I/O fds

1

Exit handling

Load unikernel2

I/O

Configure seccomp

3

• Tender: modified ukvmunikernel monitor• Userspace process• Uses seccomp to

restrict interface

• Setup and loading• “Exit” handling

Unikernel isolation comes from the interface

• 10 hypercalls

19

Hypercallwalltime

puts

poll

blkinfo

blkwrite

blkread

netinfo

netwrite

netread

halt

• 6 for I/O• Network: packet level• Storage: block level

• vs. >350 syscalls

Unikernel isolation comes from the interface

• Direct mapping between 10 hypercalls and system call/resource pairs

20

Hypercallwalltime

puts

poll

blkinfo

blkwrite

blkread

netinfo

netwrite

netread

halt

• 6 for I/O• Network: packet level• Storage: block level

• vs. >350 syscalls

System Call Resourceclock_gettime

write stdout

ppoll net_fd

pwrite64 blk_fd

pread64 blk_fd

write net_fd

read net_fd

exit_group

Implementation: nabla !

21

• Extended Solo5 unikernelecosystem and ukvm• Prototype supports:

• MirageOS• IncludeOS• Rumprun

• https://github.com/solo5/solo5

Measuring isolation: common applications

0 200 400 600 800

1000 1200 1400 1600

nginxnginx-large

node-express

redis-get

redis-set

Uni

que

kern

elfu

nctio

ns a

cces

sed process

ukvmnabla

22

• Code reachable through interface is a metric for attack surface• Used kernel ftrace

• Results:• Processes: 5-6x more • VMs: 2-3x more

Measuring isolation: fuzz testing

23

0

100

200

300

400

500

600

700

0 50 100 150 200 250 300

Uni

que

kern

el fu

nctio

ns

acceptnablablock

0

30

0 10

• Used kernel ftrace• Used trinity

system call fuzzer to try to access more of the kernel

• Results:• Nabla policy reduces

by 98% over a “normal” process

Measuring performance: throughput

80%

100%

120%

140%

160%

180%

200%

py_tornado

py_chameleon

node_fib

mirage_HTTP

py_2to3

node_express

nginx_large

redis_get

redis_set

includeos_TCP

nginx

includeos_UDP

Norm

alize

d th

roug

hput

245

no I/O with I/O

ukvmnablaQEMU/KVM

24

• Applications include:• Web servers• Python benchmarks• Redis• etc.

• Results:• 101%-245% higher

throughput than ukvm

Measuring performance: CPU utilization

0 20 40 60 80

100 120

(a)

CPU

%

0 20 40 60 80

100

(b)

VM

exits

/ms

0

0.5

1

1.5

0 5000 10000 15000 20000

(c)

IPC

(ins

/cyc

le)

Requests/sec

nablaukvm

25

• vmexits have an effect on instructions per cycle

• Experiment with MirageOSweb server• Results:

• 12% reduction in cpuutilization over ukvm

Measuring performance: startup time

0

250

500

750Hello

world

QEMU/KVM

0

10

20

30ukvm

0

10

20

30nabla

0

10

20

30process

0

500

QEMU/KVM

ukvm

nabla

process

2 4 6 8 10 12 14 160

500

1000

1500

HTTP

POST

2 4 6 8 10 12 14 160

50

100

150

200

2 4 6 8 10 12 14 16

Number of cores

0

50

100

150

200

2 4 6 8 10 12 14 160

50

100

150

200

0 2 4 6 8 10 12 14

0

500

1000

1500

26

• Startup time is important for serverless, NFV

• Results:• Ukvm has 30-370% higher

latency than nabla

• Mostly due avoiding KVM overheads

Hel

lo w

orld

HTT

P Po

st

Summary and Next Steps

• Unikernels should run as processes!• Maintain isolation via thin interface• Improve performance, etc.

• Next steps: can unikernels as processes be used to improve container isolation?• Nabla containers • https://nabla-containers.github.io/

27

Process

28