Operating your OpenStack Private Cloud.pdf

October 12, 2012

Operating your OpenStack Private Cloud

Ryan RichardOpenStack Engineer

[email protected]@rackninja

Thursday, October 18, 12

mailto:[email protected]

mailto:[email protected]

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

Monitoring and Reporting

Where we were -‐ April 2012

Basic CDM





Basic CDMNow





Basic CDMNow



Tools

There is no good way to get the following info:

I need a list of instances on a host and their IPs

I need to gracefully start/stop all instances on a host

Some tools needs hostname, some need id (decimal or hex), some need uuid



Tools

There is no good way to get the following info:

I need a list of instances on a host and their IPs

I need to gracefully start/stop all instances on a host

Some tools needs hostname, some need id (decimal or hex), some need uuid

SELECT instances.id,instances.hostname,instances.project_id,fixed_ips.address as fixed_address,floating_ips.address as floating_address FROM instances LEFT JOIN fixed_ips ON instances.id=fixed_ips.instance_id LEFT JOIN floating_ips ON floating_ips.fixed_ip_id=fixed_ips.id WHERE instances.deleted="NULL" AND instances.host="<hostname of physical machine>" ORDER BY instances.id;


RACKSPACE® HOSTING | WWW.RACKSPACE.COM 4

Tools

WE NEED BETTER OPS TOOLS!



Tools

Pulsar

https://github.com/rsoprivatecloud/pulsar

“nova swiss army knife”

requires direct nova database access



https://github.com/JCallicoat/pulsar





Tools

Pulsar

https://github.com/rsoprivatecloud/pulsar

“nova swiss army knife”

requires direct nova database access








Tools

Holland (opensource database backup framework)

Written by Rackspace DBAs

http://wiki.hollandbackup.org/





Tools

dsh

dsh -Mcg compute uname-a



Tools

dsh




Tools

dsh


bashfoo

for i in `knife node list | grep cpu`; do knife node run_list add $i "role[single-‐compute]"; done

for k in `seq 1 20`; do for i in {compute,network}; do nova-‐manage service disable computevm0$k nova-‐$i; done; done



Performance and Scale Considerations

Disk IO

For high performance use remote block storage

For “local” disk IO, raw image type is only slightly faster than qcow2

IO will degrade while Glance copies images between machines

scheduler=cfq, KVM cache=none




Disk IO

For high performance use remote block storage

For “local” disk IO, raw image type is only slightly faster than qcow2

IO will degrade while Glance copies images between machines

scheduler=cfq, KVM cache=none

0" 200" 400" 600" 800" 1000" 1200" 1400" 1600"

compute/host"

compute/host"(no"ht)"

compute/host"(deadline)"

Rs/speed/test13"(deadline"cache=none)"

Rs/speed/test13"(cfq,"cache=none,"no"ht)"

Rs/speed/test12"(cfq"cache=none)"

Rs/speed/test12"(noop"cache=none)"

rs/speed/test13"(cfq,"cache=writeback)"

Rs/speed/test13"(noop,"cache=writeback)"

rs/speed/test12"(cfq,"host"deadline,"cache=none)"

Async&Random&IO&

randW"(direct)"

randR"(direct)"

randW"

randR"

0"

2000"

4000"

6000"

8000"

10000"

12000"

14000"

randR" randW" randR"(direct)"

randW"(direct)"

seqR" seqW" seqR"(direct)"

seqw"(direct)"

Host&vs.&Instance&

compute/host"

Rs/speed/test12"(cfq"cache=none)"




Glance chunk Size

200Mb chunk size

1GB chunk size

5GB chunk size




Glance chunk Size

200Mb chunk size

1GB chunk size

5GB chunk size




Glance chunk Size

200Mb chunk size

1GB chunk size

5GB chunk size




Glance chunk Size

200Mb chunk size

1GB chunk size

5GB chunk size




Swift disk usage with different chunk sizes

5 zones -‐ 4 x 1TB disks per zone

20TB raw -‐ 6.67TB usable




Swift disk usage with different chunk sizes




Glance chunk size

Too high and swift can become unbalanced

What are the downsides to being too low?



Glance

Disk Tuning (swift)

read ahead on your block device(s) -‐ no noticeable gain

deadline scheduler -‐ no noticeable gain

Best thing for glance performance -‐ Caching




Glance

Disk Tuning (swift)




Image Size Not Cached Cached

1.4GB 20secs 1sec

16.4GB 2min 21secs 1sec




Glance

Disk Tuning (swift)




Image Size Not Cached Cached

1.4GB 20secs 1sec

16.4GB 2min 21secs 1sec

*times from “creating image” to “qemu-‐img

create”





Scheduler

What we use by default:

scheduler tasks are not processed in parallel

Adding additional schedulers helps provide HA but they don’t speed up overall times to complete requests



Automated Config Management

Chef: http://github.com/rcbops/chef-‐cookbooks

time to stand up

controller -‐ less than 20 minutes

compute node -‐ less than 2 min


http://github.com/rcbops/chef-cookbooks

http://github.com/rcbops/chef-cookbooks


Day to Day tasks

Dealing with new issues

resize -‐ all nova-‐compute processes need to be able to log into all other compute nodes via ssh keys

Hardware failures

We’re still managing infrastructure, failures happen



Lessons Learned

We need better Operations tools!

Network Design can be confusing for people used to “the old way”

OpenStack is still relatively new, help your organization understand it.

It’s easy to forget we’re working with Linux machines

It’s not you, it’s a bug :)



But....

But this is a design summit also

Open to discussions/thoughts/questions


RACKSPACE® HOSTING | 5000 WALZEM ROAD | SAN ANTONIO, TX 78218US SALES: 1-800-961-2888 | US SUPPORT: 1-800-961-4454 | WWW.RACKSPACE.COM

RACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN THE UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.COM

Rackspace is hiring

www.rackertalent.com


Documents

Operating your OpenStack Private Cloud.pdf