Upload
doanlien
View
235
Download
0
Embed Size (px)
Citation preview
RALLY – PERFORMANCE
BENCHMARKING OF
OPENSTACK
Tips and tools for creating and presenting wide format slides
Openstack – cloud platform
Current Openstack - Ecosysten
OpenStack Basics
Everything is written in python
End users can interact through a common web interface (Horizon) or directly to each service through their API
All services authenticate through a common source
Individual services interact with each other through their public APIs *
Most daemons implemented WSGI middleware (Paste)
Used extensively in OpenStack
Configured through *-paste.ini files
4
Identity (“Keystone”)
Keystone provides a single point of integration for OpenStack policy, catalog, token and authentication.
keystone handles API requests as well as providing configurable catalog, policy, token and identity services.
Standard backends include LDAP or SQL, as well as Key Value Stores (KVS).
Most people will use this as a point of customization for their current authentication services.
OpenStack Identity Service
keystone(ser vice & admin APIs)
ident it y backend
catalog backend
token backend policy backend
5
Dashboard (“Horizon”)
Django application that users can access in their web browser
Communicates with each OpenStack service through their API (and sometimes their admin API)
OpenStackDashboard
Hor izon
HTTP(S)
6
Object Storage (“Swift”)
OpenStack Object Store
swif t-proxy
objectcontaineraccount
objectstore
accountDB
containerDB
memcached
Stores and serves objects (files)
Employs object level replication to safeguard data
Accepts client requests via Objectstore API or HTTP from clients through swift-proxy
Maintains distributed account and container databases
Stores objects according the ring layout on filesystem with extended attributes (XFS, EXT4, etc.)
7
Image Service (“Glance”)
glance-api accepts Image API calls for image discovery, image retrieval and image storage.
glance-registry stores, processes and retrieves metadata about images (size, type, etc.).
Database to store the image metadata.
A storage repository for the actual image files. In many deployments, this is OpenStack Swift
OpenStack Image Ser vice
glance-api
glance-regist r y
glancedatabase
8
Compute (“Nova”)
OpenStack Compute
nova-api(OS, EC2, Admin)
nova-consoleauth
nova-cert/objectstore
nova-console
Queue
nova-compute
nova-scheduler
novadatabase
hyper visor
libvirt, XenAPI, etc.
nova-conductor
nova-api accepts and responds to end user compute API calls.
Supports OpenStack Compute API, Amazon's EC2 API and a special Admin API (for privileged users to perform administrative actions).
Initiates most of the orchestration activities (such as running an instance)
Enforces some policy (mostly quota checks)
Authentication is handled through middleware before getting to this daemon
9
Block Storage (“Cinder”) cinder-api accepts API requests and
routes them to cinder-volume for
action.
cinder-volume acts upon the
requests by reading or writing to the
Cinder database to maintain state,
interacting with other processes (like
cinder-scheduler) through a message
queue and directly upon block
storage providing hardware or
software. It can interact with a
variety of storage providers through a
driver architecture. Currently, there
are drivers for IBM, SolidFire, NetApp,
Nexenta, Zadara, linux iSCSI and
other storage providers.
Much like nova-scheduler, the
cinder-scheduler daemon picks the
optimal block storage provider node
to create the volume on.
OpenStack Block Storage
cinder-api
cinder-volume
cinder-scheduler
cinderdatabase
volume provider
10
Networking (“Quantum”)
OpenStack Net work Ser vice
quant um-ser ver
quant um plugin(s)
net workprovider
Queue
quant umdatabase
quant um agent (s)
quantum-server accepts API requests and then routes them to the appropriate quantum plugin for action.
Quantum ships with plugins and agents for:
Cisco virtual and physical switches
Nicira NVP product
NEC OpenFlow products
Open vSwitch
Linux bridging
Ryu Network Operating System
Midokua
The common agents are L3 (layer 3), DHCP (dynamic host IP addressing) and the specific plug-in agent.
11
Rally - Why ?
"How does OpenStack work at scale?”
Rally Components
Server Providers - provide servers (virtual servers), with ssh access, in one L3
network.
Deploy Engines - deploy OpenStack cloud on servers that are presented by
Server Providers
Verification - component that runs tempest (or another specific set of tests)
against a deployed cloud, collects results & presents them in human readable
form.
Benchmark engine - allows to write parameterized benchmark scenarios & run
them against the cloud.
Rally -Functionality
Rally Use Cases
Installing Rally on Ubuntu.
Prerequisite
• sudo apt-get update
• sudo apt-get install libpq-dev git-core python-dev libevent-dev libssl-dev python-pip libffi-dev
• sudo pip install pbr
Note : We would recommend using a separate machine for Rally. If the machine that
you use to run rally has some Openstack components running, we suggest creating a
virtual environment for running Rally because it may have conflicts with version of the
client python libraries.
Installing Rally on Ubuntu. Installing Rally
Clone
• git clone https://github.com/stackforge/rally.git && cd rally
• python setup.py install
Configure
• sudo mkdir /etc/rally
• sudo cp etc/rally/rally.conf.sample /etc/rally/rally.conf
• sudo vim /etc/rally/rally.conf
• # Change the "connection" parameter,
• # e.g. to connection="sqlite://///home/<your_username>/.venv/rally-db/$sqlite_db" (or any other place)
Create Database
• rally-manage db recreate
Rally : Deploy Engines.
The task of a deploy engine is to control the process of deploying some OpenStack
distribution like DevStack or FUEL before any benchmarking procedures take place.
Every deploy engine should implement the following fairly simple interface:
• constuctor, which takes a deployment entity as its only parameter;
• deploy(), which should deploy the appropriate OpenStack distribution given the cloud
config from the deployment object the engine was initialized with (possibly using one
of available server providers)
• cleanup(), which should clean up the OpenStack deployment (again, possibly using one
of available server providers).
Rally. Deploy Engine . Example If you already have a existing Openstack Deployment that you want to benchmark
Use a json file that looks like one below. (with value specific to your system):
{
"name": "DummyEngine",
"endpoint": {
"auth_url": "http://192.168.122.22:5000/v2.0/",
"username": "admin",
"password": "password",
"tenant_name": "admin"
}
}
For Devstack based deployment {
"name": "DevstackEngine",
"localrc": {
"ADMIN_PASSWORD": "secret",
"NOVA_REPO": "git://example.com/nova/",
...
},
"devstack_repo": "git://example.com/devstack/",
"provider": {
"name": "${PROVIDER_NAME}",
...
}
}
Rally – Server Providers Server providers in Rally are typically used by deploy engines to manage virtual
machines necessary for OpenStack deployment and its following benchmarking.
The key feature of server providers is that they provide a unified interface for
interacting with different virtualization technologies (LXS, Virsh etc.) and cloud
suppliers (like Amazon).
Every server provider should implement the following basic interface:
• constructor, which should take the deployment entity the provider should bind to and a
config dictionary as its parameters;
• create_servers(image_uuid, type_id, amount), which should create the requested number of
virtual machines of the given type using a specific image. The method should also return the
list of created servers wrapped in special Server entities.
• destroy_servers(), which should destroy all virtual machines previously created by the same
server provider.
Rally – Server Providers –Ex DummyProvider – If you already have an Openstack Deployment.
This provider does nothing, but returns endpoints from configuration. This may be
useful if you have specific software/hardware configuration ready to deploy
OpenStack.
{
"name": "ExampleEngine",
"provider": {
"name": "DummyProvider",
"credentials": [{"user": "root", "host": "host1.net"}, {"user": "root", "host": "host2.net"}]
}
}
Rally – How_to_run
Simple
1. Initialize your Deployment
2. Create a json for Supported Benchmarking scenario
3. Run Benchmarking against a deployment above
Rally – Initialize Deployment 1. Create a Deployment configuration (json) file.
If you are running Rally against a existing Openstack Deployment your should look like
{
"name": "DummyEngine",
"endpoint": {
"auth_url": <KEYSTONE_AUTH_URL>,
"username": <ADMIN_USER_NAME>,
"password": <ADMIN_PASSWORD>,
"tenant_name": <ADMIN_TENANT>
}
}
2. Create a deployment using deployment create command
$ rally deployment create --filename=dummy_deployment.json --name=dummy
3. If you want to list deployments
$ rally deployment list
4. Switch to a different Deployment
$ rally use deployment --deploy-id=<Another deployment UUID>
Rally – Set Benchmark scenario. Some sample configurations are provided at “rally/doc/samples/tasks/”.
Lets pick up a scenario boot-and-delete.json from Nova. It looks like
{
"NovaServers.boot_and_delete_server": [
{
"args": {"flavor_id": 1,
"image_id": "73257560-c59b-4275-a1ec-ab140e5b9979"},
"execution": "continuous",
"config": {"times": 10, "active_users": 2, "tenants": 3,
"users_per_tenant": 2}
}
]
}
Modify this to design your test-case. Similarly for all other cases other available json can be modified or you can even write a new one for a custom case.
Rally by default uses the last created deployment. Use the switch deployment commands to run the tests with a different deployment.
Lets dig deeper
Test name : “NovaServers.boot_and_delete_server”
Execution : either continuous / periodic
Times : Number of times the test needs to be run
Active_users : Number of parallel threads (concurrent users)
Tenants : Total number of tenants to be created
Users_per_tenant : Number of users per single tenant
Other parameters to be used only with supported tests
Network : Name of network to be used
Script : If a script is passed as input to the test
Actions : soft_reboot / stop_start
Rally – Run Benchmark. Run your benchmark scenario by pointing at the json you created in the previous step
$ rally --verbose task start --task=my-task.json
You can check the state of the task
$ rally task list
To check a complete task analysis
$ rally task detailed <Task UUID>
Rally Result –Example.
$ rally task detailed <Task UUID>
===========================================================================
=====
Task <Task UUID> is finished. Failed: False
--------------------------------------------------------------------------------
test scenario NovaServers.boot_and_delete_server
args position 0
args values:
{u'args': {u'flavor_id': <Flavor UUID>,
u'image_id': u'<Image UUID>'},
u'config': {u'active_users': 1, u'times': 2}}
+--------------------+---------------+---------------+---------------+
| action | max (sec) | avg (sec) | min (sec) |
+--------------------+---------------+---------------+---------------+
| nova.boot_server | 9.22798299789 | 8.90022659302 | 8.57247018814 |
| nova.delete_server | 4.24928498268 | 3.26377093792 | 2.27825689316 |
+--------------------+---------------+---------------+---------------+
+---------------+---------------+---------------+---------------+-------------+
| max (sec) | avg (sec) | min (sec) | success/total | total times |
+---------------+---------------+---------------+---------------+-------------+
| 13.4775559902 | 12.1641695499 | 10.8507831097 | 1.0 | 2 |
+---------------+---------------+---------------+---------------+-------------+
--------------------------------------------------------------------------------
Detailed performance benchmarking
Docker in OpenStack • Havana
– Nova virt driver which integrates with docker REST API on backend – Glance translator to integrate docker images with Glance
• Icehouse – Heat plugin for docker
• Both options are still under development
9/12/2014 29
nova-docker virt driver docker heat plugin
About This Benchmark • Use case perspective
– As an OpenStack Cloud user I want a Ubuntu based VM with MySQL… Why would I choose docker LXC vs a traditional hypervisor?
• OpenStack “Cloudy” perspective – LXC vs. traditional VM from a Cloudy (OpenStack) perspective – VM operational times (boot, start, stop, snapshot) – Compute node resource usage (per VM penalty); density factor
• Guest runtime perspective – CPU, memory, file I/O, MySQL OLTP
• Why KVM? – Exceptional performance
9/12/2014 30
Benchmark Environment Topology @ SoftLayer
glance api / reg nova api / cond / etc
keystone …
rally
nova api / cond / etc cinder api / sch / vol
docker lxc
dstat
controller compute node
glance api / reg nova api / cond / etc
keystone …
rally
nova api / cond / etc cinder api / sch / vol
KVM
dstat
controller compute node
9/12/2014 31
+ Awesome!
+ Awesome!
Benchmark Specs
9/12/2014 32
Spec Controller Node (4CPU x 8G RAM) Compute Node (16CPU x 96G RAM)
Environment Bare Metal @ SoftLayer Bare Metal @ SoftLayer
Mother Board SuperMicro X8SIE-F Intel Xeon QuadCore SingleProc SATA [1Proc] SuperMicro X8DTU-F_R2 Intel Xeon HexCore DualProc [2Proc]
CPU Intel Xeon-Lynnfield 3470-Quadcore [2.93GHz] (Intel Xeon-Westmere 5620-Quadcore [2.4GHz]) x 2
Memory (Kingston 4GB DDR3 2Rx8 4GB DDR3 2Rx8 [4GB]) x2 (Kingston 16GB DDR3 2Rx4 16GB DDR3 2Rx4 [16GB]) x 6
HDD (LOCAL) Digital WD Caviar RE3 WD5002ABYS [500GB]; SATAII Western Digital WD Caviar RE4 WD5003ABYX [500GB]; SATAII
NIC eth0/eth1 @ 100 Mbps eth0/eth1 @100 Mbps
Operating System Ubuntu 12.04 LTS 64bit Ubuntu 12.04 LTS 64bit
Kernel 3.5.0-48-generic 3.8.0-38-generic
IO Scheduler deadline deadline
Hypervisor tested NA - KVM 1.0 + virtio + KSM (memory deduplication) - docker 0.10.0 + go1.2.1 + commit dc9c28f + AUFS
OpenStack Trunk master via devstack Trunk master via devstack. Libvirt KVM nova driver / nova-docker virt driver
OpenStack Benchmark Client OpenStack project rally NA
Metrics Collection NA dstat
Guest Benchmark Driver NA - Sysbench 0.4.12 - mbw 1.1.1.-2 - iibench (py) - netperf 2.5.0-1
VM Image NA - Scenario 1 (KVM): official ubuntu 12.04 image + mysql snapshotted and exported to qcow2 – 1080 MB
- Scenario 2 (docker): guillermo/mysql -- 334.8 MB
Test Descriptions: Cloudy Benchmarks
9/12/2014 33
Benchmark Benchmark Driver Description
OpenStack Cloudy Benchmarks
Serial VM boot (15 VMs)
OpenStack Rally - Boot VM from image - Wait for ACTIVE state - Repeat the above a total of 15 times - Delete VMs
VM reboot (5 VMs rebooted 5 times each)
OpenStack Rally - Boot VM from image - Wait for ACTIVE state - Soft reboot VM 5 times - Delete VM - Repeat the above a total of 5 times
VM snapshot (1 VM, 1 snapshot)
OpenStack Rally - Boot VM from image - Wait for ACTIVE state - Snapshot VM to glance image - Delete VM
Test Descriptions: Guest Benchmarks
9/12/2014 34
Benchmark Benchmark Driver Description
Guest Runtime Benchmarks
CPU performance Sysbench from within the guest - Run sysbench cpu test - Repeat a total of 10 times - Average results over the 10 times
OLTP (MySQL) performance Sysbench from within the guest - Run sysbench OLTP test - Repeat a total of 10 times - Average results over the 10 times
MySQL Indexed insertion benchmark - Run iibench for a total of 1M inserts printing stats at 100K intervals - Collect data over 5 runs & average
File I/O performance Sysbench from within the guest - Synchronous IO - No-direct
- Run sysbench OLTP test - Repeat a total of 10 times - Average results over the 10 times
Memory performance Mbw from within the guest - Run mbw with array size of 1000 MiB and each test 10 times - Collect average over 10 runs per test
Network performance Netperf - Run netperf server on controller - From guest run netperf client in IPv4 mode - Repeat text 5x - Average results
SERIALLY BOOT 15 VMS OpenStack Cloudy Benchmark
9/12/2014 35
Cloudy Performance: Serial VM Boot
• Benchmark scenario overview – Boot VM via OpenStack nova – Wait for VM to become active – Repeat the above steps for a total of 15 VMs – Delete all VMs
9/12/2014 36
0
5
10
15
20
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Act
ive
VM
s
Time
Benchmark Visualization
VMs
Cloudy Performance: Serial VM Boot
9/12/2014 37
3.900927941
5.884197426
0
1
2
3
4
5
6
7
docker KVM
Tim
e in
Se
con
ds
Average Server Boot Time
docker
KVM
Docker 1.5x faster
Cloudy Performance: Serial VM Boot
9/12/2014 38
0
20
40
60
80
100
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75 77 79 81 83 85 87 89CP
U U
sage
In
Pe
rce
nt
Time
Docker: Compute Node CPU
usr
sys
0
20
40
60
80
100
1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101105109113117121125129133137CP
U U
sage
In
Pe
rce
nt
Time
KVM: Compute Node CPU
usr
sys
Averages
– 1.14
– 0.44
Averages
– 12.6
– 2.08
Cloudy Performance: Serial VM Boot
9/12/2014 39
0.00E+00
1.00E+09
2.00E+09
3.00E+09
4.00E+09
5.00E+09
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75 77 79 81 83 85 87 89
Me
mo
ry U
sed
Time
Docker: Compute Node Used Memory
Memory
0.00E+00
1.00E+09
2.00E+09
3.00E+09
4.00E+09
5.00E+09
1 5 9
13
17
21
25
29
33
37
41
45
49
53
57
61
65
69
73
77
81
85
89
93
97
10
1
10
5
10
9
11
3
11
7
12
1
12
5
12
9
13
3
13
7
Me
mo
ry U
sed
Time
KVM: Compute Node Used Memory
Memory
Delta 687 MB Per VM 45.8 MB
Delta 2775 MB Per VM 185 MB
Cloudy Performance: Serial VM Boot
9/12/2014 40
0
5
10
15
20
25
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75 77 79 81 83 85 87 89
1 M
inu
te L
oad
Ave
rage
Time
Docker: Compute Node 1m Load Average
1m
0
5
10
15
20
25
1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101105109113117121125129133137
1 M
inu
te L
oad
Ave
rage
Time
KVM: Compute Node 1m Load Average
1m
Average 0.09 %
Average 9.94 %
SERIAL VM SOFT REBOOT OpenStack Cloudy Benchmark
9/12/2014 41
Cloudy Performance: Serial VM Reboot • Benchmark scenario overview
– Boot a VM
• Wait for it to become active
– Soft reboot the VM
• Wait for it to become active
• Repeat soft reboot a total of 5 times
– Delete VM
– Repeat the above for a total of 5 VMs
9/12/2014 42
0
1
2
3
4
5
6
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55
Act
ive
VM
s
Time
Benchmark Visualization
Active VMs
Cloudy Performance: Serial VM Reboot
9/12/2014 43
6.591313448
124.4525079
0
20
40
60
80
100
120
140
docker KVM
Tim
e In
Se
con
ds
Average Server Reboot Time
docker
KVM
Docker 18.9x faster
Cloudy Performance: Serial VM Reboot
9/12/2014 44
7.857602167
4.451871514
0
1
2
3
4
5
6
7
8
9
docker KVM
Tim
e In
Se
con
ds
Average Server Delete Time
docker
KVM
KVM 1.75x faster
Cloudy Performance: Serial VM Reboot
• After investigating docker delete times – Docker sends SIGTERM to container process to stop
– Bash is immune to SIGTERM
– Docker waits for X seconds before stop
– Default container image (init) command was using bash
– See: https://github.com/dotcloud/docker/issues/3766
• Rebuild the docker mysql image – Don’t use bash for container image command
– No change to docker (hypervisor); image config only change
• Rerun rally reboot tests; Cloudy reboot round 2
9/12/2014 45
Cloudy Performance: Serial VM Reboot
9/12/2014 46
2.541945305
124.4525079
0
20
40
60
80
100
120
140
docker KVM
Tim
e In
Se
con
ds
Average Server Reboot Time (Round 2)
docker
KVM
Docker 48.99x faster
Cloudy Performance: Serial VM Reboot
9/12/2014 47
4.093254519
4.451871514
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
docker KVM
Tim
e In
Se
con
ds
Average Server Delete Time (Round 2)
docker
KVM
Docker 1.09x faster
Cloudy Performance: Serial VM Reboot
9/12/2014 48
0
2
4
6
8
10
1
74
14
7
22
0
29
3
36
6
43
9
51
2
58
5
65
8
73
1
80
4
87
7
95
0
10
23
10
96
11
69
12
42
13
15
13
88
14
61
15
34
16
07
16
80
17
53
18
26
18
99
19
72
20
45
21
18
21
91
22
64
23
37
24
10
24
83
25
56
26
29
27
02
27
75
28
48
29
21
29
94
30
67
31
40
CP
U U
sage
In
Pe
rce
nt
Time
KVM: Compute Node CPU
usr
sys
0
2
4
6
8
101 7
13
19
25
31
37
43
49
55
61
67
73
79
85
91
97
10
3
10
9
11
5
12
1
12
7
13
3
13
9
14
5
15
1
15
7
16
3
16
9
17
5
18
1
18
7
19
3
19
9
20
5
21
1
21
7
22
3
22
9
23
5
24
1
CP
U U
sage
In
Pe
rce
nt
Time
Docker: Compute Node CPU
usr
sys
Averages
– 0.34
– 0.11
Averages
– 0.82
– 0.19
Cloudy Performance: Serial VM Reboot
9/12/2014 49
0.00E+00
5.00E+08
1.00E+09
1.50E+09
2.00E+09
2.50E+09
1 7
13
19
25
31
37
43
49
55
61
67
73
79
85
91
97
10
3
10
9
11
5
12
1
12
7
13
3
13
9
14
5
15
1
15
7
16
3
16
9
17
5
18
1
18
7
19
3
19
9
20
5
21
1
21
7
22
3
22
9
23
5
24
1
Me
mo
ry U
sed
Time
Docker: Compute Node Used Memory
Memory
0.00E+00
5.00E+08
1.00E+09
1.50E+09
2.00E+09
2.50E+09
17
91
57
23
53
13
39
14
69
54
76
25
70
37
81
85
99
37
10
15
10
93
11
71
12
49
13
27
14
05
14
83
15
61
16
39
17
17
17
95
18
73
19
51
20
29
21
07
21
85
22
63
23
41
24
19
24
97
25
75
26
53
27
31
28
09
28
87
29
65
30
43
31
21
Me
mo
ry U
sed
Time
KVM: Compute Node Used Memory
Memory
Delta 57 MB
Delta 467 MB
Cloudy Performance: Serial VM Reboot
9/12/2014 50
0
1
2
3
4
17
11
41
21
12
81
35
14
21
49
15
61
63
17
01
77
18
41
91
19
81
10
51
11
21
11
91
12
61
13
31
14
01
14
71
15
41
16
11
16
81
17
51
18
21
18
91
19
61
20
31
21
01
21
71
22
41
23
11
23
81
24
51
25
21
25
91
26
61
27
31
28
01
28
71
29
41
30
11
30
81
31
51
1 M
inu
te L
oad
Ave
rage
Time
KVM: Compute Node 1m Load Average
1m
-0.5
0.5
1.5
2.5
3.51 7
13
19
25
31
37
43
49
55
61
67
73
79
85
91
97
10
3
10
9
11
5
12
1
12
7
13
3
13
9
14
5
15
1
15
7
16
3
16
9
17
5
18
1
18
7
19
3
19
9
20
5
21
1
21
7
22
3
22
9
23
5
24
1
1 M
inu
te L
oad
Ave
rage
Time
Docker: Compute Node 1m Load Average
1m
Average 0.05 %
Average 0.35 %
SNAPSHOT VM TO IMAGE OpenStack Cloudy Benchmark
9/12/2014 51
Cloudy Performance: Snapshot VM To Image
• Benchmark scenario overview
– Boot a VM
– Wait for it to become active
– Snapshot the VM
– Wait for image to become active
– Delete VM
9/12/2014 52
Cloudy Performance: Snapshot VM To Image
9/12/2014 53
26.39477992
42.92771101
0
5
10
15
20
25
30
35
40
45
50
docker KVM
Tim
e (
Seco
nd
s)
Average Snapshot Server Time
docker
KVM
Docker 1.62x faster
Cloudy Performance: Snapshot VM To Image
9/12/2014 54
0
2
4
6
8
1 4 7
10
13
16
19
22
25
28
31
34
37
40
43
46
49
52
55
58
61
64
67
70
73
76
79
82
85
88
91
94
97
10
0
10
3
10
6
10
9
11
2
CP
U U
sage
In
Pe
rce
nt
Time
KVM: Compute Node CPU
usr
sys
-1
1
3
5
7
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67CP
U U
sage
In
Pe
rce
nt
Time
Docker: Compute Node CPU
usr
sys
Averages
– 0.4
– 0.11
Averages
– 1.58
– 1.07
Cloudy Performance: Snapshot VM To Image
9/12/2014 55
1.65E+09
1.7E+09
1.75E+09
1.8E+09
1.85E+09
1 4 7
10
13
16
19
22
25
28
31
34
37
40
43
46
49
52
55
58
61
64
67
70
73
76
79
82
85
88
91
94
97
10
0
10
3
10
6
10
9
11
2
Me
mo
ry U
sed
Time
KVM: Compute Node Used Memory
Memory
1.82E+091.84E+091.86E+091.88E+09
1.9E+091.92E+091.94E+09
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67
Me
mo
ry U
sed
Time
Docker: Compute Node Used Memory
Memory
Delta 48 MB
Delta 114 MB
Cloudy Performance: Snapshot VM To Image
9/12/2014 56
0
0.2
0.4
0.6
1 4 7
10
13
16
19
22
25
28
31
34
37
40
43
46
49
52
55
58
61
64
67
70
73
76
79
82
85
88
91
94
97
10
0
10
3
10
6
10
9
11
2
1 M
inu
te L
oad
Ave
rage
Time
KVM: Compute Node 1m Load Average
1m
0
0.2
0.4
0.6
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67
1 M
inu
te L
oad
Ave
rage
Time
Docker: Compute Node 1m Load Average
1m
Average 0.1 %
Average 0.37 %
GUEST PERFORMANCE BENCHMARKS
Guest VM Benchmark
9/12/2014 57
Guest Performance: CPU
• Linux sysbench 0.4.12 cpu test
• Calculate prime numbers up to 20000
• 2 threads
• Instance size
– 4G RAM
– 2 CPU cores
– 20G disk 9/12/2014 58
Guest Performance: CPU
9/12/2014 59
15.11 15.08 15.03
0
2
4
6
8
10
12
14
16
Docker KVM Bare Metal
Seconds
Calculate Primes
Docker
KVM
Bare Metal
Guest Performance: Memory
• Linux mbw 1.1.1-2
• Instance size
– 2 CPU
– 4G memory
• Execution options
– 10 runs; average
– 1000 MiB 9/12/2014 60
Guest Performance: Memory
9/12/2014 61
0
2000
4000
6000
8000
10000
12000
14000
MEMCPY DUMB MCBLOCK
MiB
/s
Memory Tests
Memory Benchmark Performance
BareMetal
docker
KVM
Guest Performance: Network
• Netperf 2.5.0-1 – Netserver running on controller
– Netperf on guest
– Run netperf 5 times & average results
• Instance size – 2 CPU
– 4G memory
• Execution options – IPv4 / TCP
9/12/2014 62
Guest Performance: Network
9/12/2014 63
0
100
200
300
400
500
600
700
800
900
1000
docker KVM
Thro
ugh
pu
t In
10
^6 b
its/
seco
nd
Network Throughput
docker
KVM
Guest Performance: File I/O • Linux sysbench 0.4.12 fileio test
– Synchronous IO – Random read / write – Total file size of 150G – 16K block size
• Thread variations: 1, 8, 16, 32 • Instance size
– 4G RAM – 2 CPU cores – 200G disk
• KVM specs – Disk cache set to none – Virtio – Deadline scheduler (host & guest)
• Docker specs – AUFS storage driver – Deadline scheduler
9/12/2014 64
Guest Performance: File I/O
9/12/2014 65
0
100
200
300
1 8 16 32
Mb
Threads
File I/O: Read
Docker
KVM
0
50
100
150
200
1 8 16 32
Mb
Threads
File I/O: Write
Docker
KVM
Guest Performance: File I/O – Read / Write
9/12/2014 66
0
200
400
600
800
1000
1200
1400
1600
1 8 16 32
Kb
/se
c
Threads
File I/O: Transfer Rate
Docker
KVM
Guest Performance: MySQL OLTP • Linux sysbench 0.4.12 oltp test
– Table size of 2,000,000 – MySQL 5.5 (installed on Ubuntu 12.04 LTS with apt-get)
• Variations – Number of threads – Read only & read / write
• Instance size – 4G RAM – 2 CPU cores – 20G disk
9/12/2014 67
Guest Performance: MySQL OLTP (Read)
9/12/2014 68
0
20000
40000
60000
2 8 16
Tota
l Tra
nsa
ctio
ns
Threads
MySQL OLTP Read Transactions
Docker
KVM
0
200000
400000
600000
800000
2 8 16
Re
qu
est
s
Threads
MySQL OLTP Read Requests
Docker
KVM
Guest Performance: MySQL OLTP (Read / Write)
9/12/2014 69
0
5000
10000
15000
20000
25000
2 16 32 64
Tota
l Tra
nsa
ctio
ns
Threads
MySQL OLTP Read/Write Transactions
docker
KVM
0
100000
200000
300000
400000
2 16 32 64
Re
qu
est
s
Threads
MySQL OLTP Read/Write Requests
Docker
KVM
Guest Performance: MySQL Indexed Insertion
• Indexed insertion benchmark (iibench python script) – A total of 1,000,000 insertions
– Print stats at 100K intervals
– Collect stats over 5 runs
– Average
• Instance size – 4G RAM
– 2 CPU cores
– 20G disk
9/12/2014 70
Guest Performance: MySQL Indexed Insertion
9/12/2014 71
0
20
40
60
80
100
120
140
1 2 3 4 5 6 7 8 9 10
Seco
nd
s P
er
10
0K
In
sert
ion
Bat
ch
Table Size In 100K Increments
MySQL Indexed Insertion @ 100K Intervals
docker
kvm
BENCHMARK OBSERVATIONS
9/12/2014 72
Docker LXC CPU Growth 26x Lower Than VM
9/12/2014 73
y = 0.0091x + 0.7349
y = 0.237x + 2.2993
0
5
10
15
20
25
30
351 4 7
10
13
16
19
22
25
28
31
34
37
40
43
46
49
52
55
58
61
64
67
70
73
76
79
82
85
88
91
94
97
10
0
10
3
CP
U U
sage
In
Pe
rce
nt
User CPU Growth Trend
docker
KVM
Linear (docker)
Linear (KVM)
Slope
– 0.0091
– 0.237
Docker LXC Memory Growth 3x Lower Than VM
9/12/2014 74
y = 1E+07x + 1E+09
y = 3E+07x + 1E+09
0.00E+00
5.00E+08
1.00E+09
1.50E+09
2.00E+09
2.50E+09
3.00E+09
3.50E+09
4.00E+09
4.50E+09
5.00E+09
1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88 91 94
Me
mo
ry U
sed
Memory Usage Growth Trend
docker
KVM
Linear (docker)
Linear (KVM)
Slope
– 1E+07
– 3E+07
Observations • Cloudy operations with Docker LXC outperform VM
– 48x server reboot, 1.5x server boot, 1.62x server snapshot, etc.
• Docker LXC density potential compared to VMs – 3x memory savings – 26x CPU savings – 3.22x smaller images in this test (note – image sizes can vary based on required packages)
• Docker LXC containers run on bare metal – Greater / equivalent “in the VM” performance at the micro benchmark level
• Micro benchmark results do not always reflect macro performance – Always benchmark your “real” workload
• Docker image (init) command can impact performance – Bash ignores SIGTERM
• Nova-docker virt driver and docker-registry components still under dev – Nice work, but room for improvement (python anyone?)
• Real performance of Docker LXC ops capped my Cloud manager – Can start the SQL image from docker CLI in 0.191s
9/12/2014 75
REFERENCES
9/12/2014 76
References & Related Links • http://www.slideshare.net/BodenRussell/realizing-linux-containerslxc • https://www.docker.io/ • http://sysbench.sourceforge.net/ • http://dag.wiee.rs/home-made/dstat/ • http://www.openstack.org/ • https://wiki.openstack.org/wiki/Rally • https://wiki.openstack.org/wiki/Docker • http://devstack.org/ • http://www.linux-kvm.org/page/Main_Page • https://github.com/stackforge/nova-docker • https://github.com/dotcloud/docker-registry • http://www.netperf.org/netperf/ • http://www.tokutek.com/products/iibench/ • http://www.brendangregg.com/activebenchmarking.html
9/12/2014 77
REFERENCE
9/12/2014 78
Rally Boot 15 VM Configuration
9/12/2014 79
• Rally config: { "NovaServers.boot_server": [ { "args": { "flavor_id": 2, "image_id": “IMAGE_ID" }, "runner": { "type": "constant", "times": 15, "active_users": 1 }, "context": { "users": { "tenants": 1, "users_per_tenant": 1 } } } ] }
• Flavor +-----+-----------+-----------+------+-----------+------+-------+-------------+-----------+ | ID | Name | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public | +-----+-----------+-----------+------+-----------+------+-------+-------------+-----------+ | 1 | m1.tiny | 512 | 1 | 0 | | 1 | 1.0 | True | | 2 | m1.small | 2048 | 20 | 0 | | 1 | 1.0 | True | | 3 | m1.medium | 4096 | 40 | 0 | | 2 | 1.0 | True | | 4 | m1.large | 8192 | 80 | 0 | | 4 | 1.0 | True | | 42 | m1.nano | 64 | 0 | 0 | | 1 | 1.0 | True | | 451 | m1.heat | 1024 | 0 | 0 | | 2 | 1.0 | True | | 5 | m1.xlarge | 16384 | 160 | 0 | | 8 | 1.0 | True | | 6 | m1.perf | 4096 | 200 | 0 | | 2 | 1.0 | True | | 7 | m1.sql | 512 | 5 | 0 | | 1 | 1.0 | True | | 8 | m1.bench | 512 | 10 | 0 | | 1 | 1.0 | True | | 84 | m1.micro | 128 | 0 | 0 | | 1 | 1.0 | True | | 9 | m1.mini | 512 | 20 | 0 | | 1 | 1.0 | True | +-----+-----------+-----------+------+-----------+------+-------+-------------+-----------+
Cloudy Benchmark: Serially Boot 15 VMs
• Docker +------------------+-------+--------------+-------------+---------------+---------------+---------------+
| action | count | max (sec) | avg (sec) | min (sec) | 90 percentile | 95 percentile |
+------------------+-------+--------------+-------------+---------------+---------------+---------------+
| nova.boot_server | 15 | 4.8055100441 | 3.900927941 | 3.64957404137 | 4.56917948723 | 4.80114896297 |
+------------------+-------+--------------+-------------+---------------+---------------+---------------+
+---------------+---------------+--------------+---------------+---------------+---------------+-------------+
| max (sec) | avg (sec) | min (sec) | 90 pecentile | 95 percentile | success/total | total times |
+---------------+---------------+--------------+---------------+---------------+---------------+-------------+
| 5.02777004242 | 4.11018741926 | 3.8767888546 | 4.77419886589 | 5.02389762402 | 1.0 | 15 |
+---------------+---------------+--------------+---------------+---------------+---------------+-------------+
• KVM +------------------+-------+---------------+---------------+---------------+---------------+---------------+
| action | count | max (sec) | avg (sec) | min (sec) | 90 percentile | 95 percentile |
+------------------+-------+---------------+---------------+---------------+---------------+---------------+
| nova.boot_server | 15 | 7.00468301773 | 5.88419742584 | 4.84723997116 | 6.13021831512 | 6.48467411995 |
+------------------+-------+---------------+---------------+---------------+---------------+---------------+
+---------------+--------------+---------------+--------------+---------------+---------------+-------------+
| max (sec) | avg (sec) | min (sec) | 90 pecentile | 95 percentile | success/total | total times |
+---------------+--------------+---------------+--------------+---------------+---------------+-------------+
| 7.22686600685 | 6.1018067201 | 5.05242800713 | 6.332523489 | 6.68914074898 | 1.0 | 15 |
+---------------+--------------+---------------+--------------+---------------+---------------+-------------+
9/12/2014 80
Rally Reboot 5x5 Configuration
9/12/2014 81
• Rally config: {
"NovaServers.boot_and_bounce_server": [
{
"args": {
"flavor_id": 2,
"image_id": “ID",
"actions": [
{"soft_reboot": 5}
]
},
"runner": {
"type": "constant",
"times": 5,
"active_users": 1
},
"context": {
"users": {
"tenants": 1,
"users_per_tenant": 1
}
}
}
]
}
Cloudy Performance: Serial VM Reboot • Docker +--------------------+-------+---------------+---------------+---------------+---------------+---------------+ | action | count | max (sec) | avg (sec) | min (sec) | 90 percentile | 95 percentile | +--------------------+-------+---------------+---------------+---------------+---------------+---------------+ | nova.reboot_server | 25 | 6.85025811195 | 6.59131344795 | 6.50809788704 | 6.65774655342 | 6.72213015556 | | nova.boot_server | 5 | 4.96266412735 | 4.13350987434 | 3.82891011238 | 4.61819009781 | 4.79042711258 | | nova.delete_server | 5 | 8.74263191223 | 7.85760216713 | 6.61050701141 | 8.72944793701 | 8.73603992462 | +--------------------+-------+---------------+---------------+---------------+---------------+---------------+ +---------------+---------------+---------------+---------------+---------------+---------------+-------------+ | max (sec) | avg (sec) | min (sec) | 90 pecentile | 95 percentile | success/total | total times | +---------------+---------------+---------------+---------------+---------------+---------------+-------------+ | 45.8827331066 | 44.9664116383 | 43.5658369064 | 45.7298994541 | 45.8063162804 | 1.0 | 5 | +---------------+---------------+---------------+---------------+---------------+---------------+-------------+
• KVM +--------------------+-------+---------------+---------------+---------------+---------------+---------------+ | action | count | max (sec) | avg (sec) | min (sec) | 90 percentile | 95 percentile | +--------------------+-------+---------------+---------------+---------------+---------------+---------------+ | nova.reboot_server | 10 | 124.951740026 | 124.452507925 | 123.876129866 | 124.950992012 | 124.951366019 | | nova.boot_server | 2 | 6.1228749752 | 5.58395600319 | 5.04503703117 | 6.0150911808 | 6.068983078 | | nova.delete_server | 2 | 4.45799517632 | 4.45187151432 | 4.44574785233 | 4.45677044392 | 4.45738281012 | +--------------------+-------+---------------+---------------+---------------+---------------+---------------+ +---------------+---------------+---------------+--------------+---------------+---------------+-------------+ | max (sec) | avg (sec) | min (sec) | 90 pecentile | 95 percentile | success/total | total times | +---------------+---------------+---------------+--------------+---------------+---------------+-------------+ | 632.718183994 | 632.315432072 | 631.912680149 | 632.63763361 | 632.677908802 | 0.4 | 5 | +---------------+---------------+---------------+--------------+---------------+---------------+-------------+
9/12/2014 82
Rally Snapshot Configuration
9/12/2014 83
Rally config: {
"NovaServers.boot_and_bounce_server": [
{
"args": {
"flavor_id": 1,
"image_id": “IMAGE",
"actions": [
{"soft_reboot": 5}
]
},
"runner": {
"type": "constant",
"times": 5,
"active_users": 1
},
"context": {
"users": {
"tenants": 1,
"users_per_tenant": 1
}
}
}
]
}
Cloudy Performance: Snapshot VM To Image
• Docker (note -- defect deleting image) +--------------------+-------+---------------+---------------+---------------+---------------+---------------+ | action | count | max (sec) | avg (sec) | min (sec) | 90 percentile | 95 percentile | +--------------------+-------+---------------+---------------+---------------+---------------+---------------+ | nova.create_image | 1 | 26.3947799206 | 26.3947799206 | 26.3947799206 | 26.3947799206 | 26.3947799206 | | nova.boot_server | 2 | 4.120429039 | 3.85578501225 | 3.59114098549 | 4.06750023365 | 4.09396463633 | | nova.delete_server | 2 | 8.61395692825 | 7.64416801929 | 6.67437911034 | 8.41999914646 | 8.51697803736 | +--------------------+-------+---------------+---------------+---------------+---------------+---------------+ +-----------+-----------+-----------+--------------+---------------+---------------+-------------+ | max (sec) | avg (sec) | min (sec) | 90 pecentile | 95 percentile | success/total | total times | +-----------+-----------+-----------+--------------+---------------+---------------+-------------+ | n/a | n/a | n/a | n/a | n/a | 0 | 1 | +-----------+-----------+-----------+--------------+---------------+---------------+-------------+ • KVM +--------------------+-------+----------------+----------------+----------------+----------------+----------------+ | action | count | max (sec) | avg (sec) | min (sec) | 90 percentile | 95 percentile | +--------------------+-------+----------------+----------------+----------------+----------------+----------------+ | nova.delete_image | 1 | 0.700344085693 | 0.700344085693 | 0.700344085693 | 0.700344085693 | 0.700344085693 | | nova.create_image | 1 | 42.92771101 | 42.92771101 | 42.92771101 | 42.92771101 | 42.92771101 | | nova.boot_server | 2 | 40.9650099277 | 22.9950100183 | 5.02501010895 | 37.3710099459 | 39.1680099368 | | nova.delete_server | 2 | 4.47270512581 | 4.46178817749 | 4.45087122917 | 4.47052173615 | 4.47161343098 | +--------------------+-------+----------------+----------------+----------------+----------------+----------------+ +--------------+--------------+--------------+--------------+---------------+---------------+-------------+ | max (sec) | avg (sec) | min (sec) | 90 pecentile | 95 percentile | success/total | total times | +--------------+--------------+--------------+--------------+---------------+---------------+-------------+ | 98.541975975 | 98.541975975 | 98.541975975 | 98.541975975 | 98.541975975 | 1.0 | 1 | +--------------+--------------+--------------+--------------+---------------+---------------+-------------+
9/12/2014 84
Configuring Docker LXC Container For 2CPU x 4G RAM
• Pin container to 2 CPUs / Mems – Create cpuset cgroup – Pin group to cpuset.mems to 0,1 – Pin group to cpuset.cpus to 0,1 – Add container root proc to tasks
• Limit container memory to 4G – Create memory cgroup – Set memory.limit_in_bytes to 4G – Add container root proc to tasks
9/12/2014 85
Docker Command Line
• Start docker image from docker CLI root@devstack-compute:~/devstack$ time docker run -d guillermo/mysql
859f2a88adeb7190eb855eb060172fc2f137bff609f2f3be8ad8ee069d7e88f4
real 0m0.191s
user 0m0.004s
sys 0m0.004s
9/12/2014 86