28
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution.* * All unlicensed or borrowed works retain their original licenses. Pimp My Cloud Nova Configuration Hints and Tricks Joe Gordon 1 Wednesday, October 17, 12

OpenStack Summit :: Pimp My Cloud

Embed Size (px)

DESCRIPTION

Pimp My Cloud / Nova Configuration Hints and Tricks: presentation by Joe Gordon given at OpenStack Summit fall 2012.

Citation preview

Page 1: OpenStack Summit :: Pimp My Cloud

CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution.** All unlicensed or borrowed works retain their original licenses.

Pimp My CloudNova Configuration Hints and Tricks

Joe Gordon

1

Wednesday, October 17, 12

Page 2: OpenStack Summit :: Pimp My Cloud

CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution.** All unlicensed or borrowed works retain their original licenses.

About Me

• Engineer at Cloudscaling• Contributor• Deployer• Folsom Contributions

o Top 10 developer (by commits)

o Mostly in Nova

http://bitergia.com/public/reports/openstack/2012_09_folsom/

2

Wednesday, October 17, 12

Page 3: OpenStack Summit :: Pimp My Cloud

CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution.** All unlicensed or borrowed works retain their original licenses.

By the Numbers: Nova Folsom

• 190+ Contributors •

• Code churn:

Release Python Lines Other Lines Python Files Other Files

Folsom 186,738 242,721 666 788

Essex 150,894 221,109 593 302

Diablo 110,581 110,393 427 389

Code churn generated with git log --numstat --pretty="%H" $A..$B| grep \.py$ | awk 'NF==3 {plus+=$1; minus+=$2} END {printf("+%d, -%d\n", plus, minus)}'

Release Lines Insertions Lines Deletions Insertions/LoC %

Folsom 110,308 71,911 59.0%

Essex 182,298 138,346 120.8%

3

Wednesday, October 17, 12

Page 4: OpenStack Summit :: Pimp My Cloud

CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution.** All unlicensed or borrowed works retain their original licenses.

Who Wrote Nova Folsom?

4

1 138 Russell Bryant <[email protected]> 2 112 Johannes Erdfelt <[email protected]> 3 97 Dan Prince <[email protected]> 4 88 Vishvananda Ishaya <[email protected]> 5 81 Joe Gordon <[email protected]> 6 63 Michael Still <[email protected]> 7 59 Mark McLoughlin <[email protected]> 8 58 Rick Harris <[email protected]> 9 50 Yun Mao <[email protected]> 10 45 Daniel P. Berrange <[email protected]> 11 36 Chris Behrens <[email protected]> 12 31 Eoghan Glynn <[email protected]> 13 29 Brian Waldon <[email protected]> 14 26 Pádraig Brady <[email protected]> 15 25 Chuck Short <[email protected]> 16 23 Sean Dague <[email protected]> 17 21 Alex Meade <[email protected]> 18 18 Kevin L. Mitchell <[email protected]> 19 17 Brian Elliott <[email protected]> 20 17 Zhongyue Luo <[email protected]> 21 16 John Griffith <[email protected]> 22 13 Dan Smith <[email protected]> 23 13 Andrew Bogott <[email protected]> 24 12 Renuka Apte <[email protected]> 25 12 Thierry Carrez <[email protected]> 26 12 Monty Taylor <[email protected]> 27 10 MotoKen <[email protected]>

git shortlog -sne --since="Tue Mar 20 08:17:40 2012 +0100" --no-merges | cat -n

Wednesday, October 17, 12

Page 5: OpenStack Summit :: Pimp My Cloud

CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution.** All unlicensed or borrowed works retain their original licenses.

OpenStack Architecture

5

Wednesday, October 17, 12

Page 6: OpenStack Summit :: Pimp My Cloud

CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution.** All unlicensed or borrowed works retain their original licenses.

Nova Architecture

6

Wednesday, October 17, 12

Page 7: OpenStack Summit :: Pimp My Cloud

CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution.** All unlicensed or borrowed works retain their original licenses.

Vanilla Nova

• Devstack• All in One• RabbitMQ• KVM• MySQL• Default configuration options

7

Wednesday, October 17, 12

Page 8: OpenStack Summit :: Pimp My Cloud

CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution.** All unlicensed or borrowed works retain their original licenses.

Customize Nova

• Choose ao Hypervisoro Databaseo Message Queueo Network Modelo Scheduler

• Configurationo Any of the 500+ options

• Custom Serviceso Any service can be

swapped out for one you write

8

Wednesday, October 17, 12

Page 9: OpenStack Summit :: Pimp My Cloud

CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution.** All unlicensed or borrowed works retain their original licenses.

Customize Nova

• Choose ao Hypervisoro Databaseo Message Queueo Network Modelo Scheduler

• Configurationo Any of the 500+ options

• Custom Serviceso Any service can be

swapped out for one you write

9

Wednesday, October 17, 12

Page 10: OpenStack Summit :: Pimp My Cloud

CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution.** All unlicensed or borrowed works retain their original licenses.

What Makes Openstack Open

• 3 RPC backends• 3+ DB backends• 6+ Virtualization backends• 500+ configuration options

10

Wednesday, October 17, 12

Page 11: OpenStack Summit :: Pimp My Cloud

CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution.** All unlicensed or borrowed works retain their original licenses.

New Options for Folsom

• 1 RPC backendso ZeroMQ

• 2 virt backendso Hyper-Vo PowerVM

• 100+ configuration options

11

Wednesday, October 17, 12

Page 12: OpenStack Summit :: Pimp My Cloud

CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution.** All unlicensed or borrowed works retain their original licenses.

New Features for Folsom by Blueprint

• General host aggregates (operator)• Disable API extensions• Rootwrap pluggable filters (operator)• Multi-scheduler support (operator)• Flavor extra-specs• LVM ephemeral disk images (operator)• Project specific flavors• Multi-process API services (operator)

12

Wednesday, October 17, 12

Page 13: OpenStack Summit :: Pimp My Cloud

CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution.** All unlicensed or borrowed works retain their original licenses.

Backend Choices

13

Wednesday, October 17, 12

Page 14: OpenStack Summit :: Pimp My Cloud

CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution.** All unlicensed or borrowed works retain their original licenses.

RPC Comparisons

RPC Backend Protocol Company Affiliations Brokerless Language

kombu(rabbitMQ) AMQP 0-9-1 Erlang

qpid AMQP 0-10 Java, C++

ZeroMQ Python, C++

14

Wednesday, October 17, 12

Page 15: OpenStack Summit :: Pimp My Cloud

Database Simultaneous writes High Availability Native python client

?

CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution.** All unlicensed or borrowed works retain their original licenses.

Database Comparisons

15

Wednesday, October 17, 12

Page 16: OpenStack Summit :: Pimp My Cloud

CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution.** All unlicensed or borrowed works retain their original licenses.

Virtualization Comparisons

16

Backend maintained by Virtualization type Fast Open Source

Baremetal N/A N/A

UML paravirt only

LXC container

Xen full/paravirt

KVM full

Qemu qemu.org full

Hyper-V full

PowerVM full

VMware ESX full

Wednesday, October 17, 12

Page 17: OpenStack Summit :: Pimp My Cloud

CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution.** All unlicensed or borrowed works retain their original licenses.

Configuration Options

17

Wednesday, October 17, 12

Page 18: OpenStack Summit :: Pimp My Cloud

CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution.** All unlicensed or borrowed works retain their original licenses.

Configuration Options by Type

18

57%

1%4%

23%

1%

13%

Boolean 68Floats 6Ints 119Lists 18MultiString 6String 290

Wednesday, October 17, 12

Page 19: OpenStack Summit :: Pimp My Cloud

CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution.** All unlicensed or borrowed works retain their original licenses.

New Config Options in Folsomallowed_rpc_exception_modulesapi_urlauth_blobbackdoor_portbase_dir_namecinder_*claim_timeout_secondsconfig_drive_formatconfig_drive_skip_versionsconfig_drive_tempdirdefault_scheduler_driverdefer_iptables_applyec2_strict_validationec2_workersfatal_deprecationsforce_config_driveglance_api_insecurehyperv_attaching_volume_retry_counthyperv_wait_between_attach_retryimage_info_filename_patterninstance_build_timeoutinstance_uuid_formatlibvirt_cpu_modelibvirt_cpu_modellibvirt_images_typelibvirt_images_volume_grouplibvirt_inject_keylibvirt_inject_partitionlibvirt_snapshots_directorylibvirt_sparse_logical_volumeslimit_cpu_featuresmatchmaker_ringfile

max_agenetapp_storage_service_prefixnfs_mount_point_basenon_inheritable_image_propertiesnotify_on_any_changenotify_on_state_changeosapi_max_request_body_sizeosapi_volume_workersperiodic_fuzzy_delayportpowervm_*quantum_*quota_driverquota_key_pairsquota_security_group_rulesquota_security_groupsrbd_*reservation_expirerootwrap_configrpc_cast_timeoutrpc_zmq_*serverserver_ca_filesnapshot_same_hostsql_connection_debugsql_connection_tracestorwize_*syslog_log_facilityuntil_refreshvswitch_namexenapi_check_hostxenapi_num_vbd_unplug_retries

19

Wednesday, October 17, 12

Page 20: OpenStack Summit :: Pimp My Cloud

CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution.** All unlicensed or borrowed works retain their original licenses.

Virtualization options1.pick a compute RPC topic scheduler_topic=compute

2.pick a compute managercompute_manager=nova.compute.manager.ComputeManager

3.compute manager optionsinstances_path=$state_path/instancesbase_dir_name=_basecompute_driver=libvirt.LibvirtDriverconsole_host=socket.getfqdn()live_migration_retry_count=30reboot_timeout=0instance_build_timeout=0rescue_timeout=0resize_confirm_window=0*_interval=running_deleted_instance_timeout=0running_deleted_instance_action=loginstance_usage_audit=false

4.compute driver optionsrescue_*_id=libvirt_type=kvmlibvirt_inject_password=falselibvirt_inject_key=true*_migration_=libvirt_cpu_mode=libvirt_cpu_model=...

20

Wednesday, October 17, 12

Page 21: OpenStack Summit :: Pimp My Cloud

CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution.** All unlicensed or borrowed works retain their original licenses.

compute.manager options    cfg.IntOpt('live_migration_retry_count',

               default=30,               help="Number of 1 second retries needed in live_migration"),    cfg.IntOpt("reboot_timeout",               default=0,               help="Automatically hard reboot an instance if it has been "                    "stuck in a rebooting state longer than N seconds. "                    "Set to 0 to disable."),    cfg.IntOpt("instance_build_timeout",               default=0,               help="Amount of time in seconds an instance can be in BUILD "                    "before going into ERROR status."                    "Set to 0 to disable."),    cfg.IntOpt("rescue_timeout",               default=0,               help="Automatically unrescue an instance after N seconds. "                    "Set to 0 to disable."),    cfg.IntOpt("resize_confirm_window",               default=0,               help="Automatically confirm resizes after N seconds. "                    "Set to 0 to disable."),  ...    cfg.StrOpt("running_deleted_instance_action",               default="log",               help="Action to take if a running deleted instance is detected."                    "Valid options are 'noop', 'log' and 'reap'. "                    "Set to 'noop' to disable."),

21

Wednesday, October 17, 12

Page 22: OpenStack Summit :: Pimp My Cloud

CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution.** All unlicensed or borrowed works retain their original licenses.

reap running deleted instances    cfg.IntOpt("running_deleted_instance_timeout",

               default=0,               help="Number of seconds after being deleted when a running "                    "instance should be considered eligible for cleanup."),    cfg.IntOpt("running_deleted_instance_interval",               default=30,               help="Number of periodic scheduler ticks to wait between runs of "                    "the cleanup task."),    cfg.StrOpt("running_deleted_instance_action",               default="noop",               help="Action to take if a running deleted instance is detected."                    "Valid options are 'noop', 'log' and 'reap'. "                    "Set to 'noop' to disable."),

nova.conf settings

running_deleted_instance_timeout=60running_deleted_instance_action=reap

22

Wednesday, October 17, 12

Page 23: OpenStack Summit :: Pimp My Cloud

CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution.** All unlicensed or borrowed works retain their original licenses.

Scheduler options1.pick a scheduler RPC topic scheduler_topic=scheduler

2.pick a scheduler managerscheduler_manager=nova.scheduler.manager.SchedulerManager

3.pick a scheduler driverscheduler_driver=nova.scheduler.multi.MultiScheduler

4.set up scheduler.driver optionsscheduler_host_manager=nova.scheduler.host_manager.HostManagerscheduler_max_attempts=3

5.multi scheduler optionscompute_scheduler_driver=nova.scheduler.filter_scheduler.FilterSchedulervolume_scheduler_driver=nova.scheduler.chance.ChanceSchedulerdefault_scheduler_driver=nova.scheduler.chance.ChanceScheduler

6.host_manager optionsscheduler_available_filters=nova.scheduler.filters.standard_filtersscheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter

7.filter optionsram_allocation_ratio=1.5

8.resource_tracker optionsreserved_host_disk_mb=0reserved_host_memory_mb=512claim_timeout_seconds=600compute_stats_class=nova.compute.stats.Stats

23

Wednesday, October 17, 12

Page 24: OpenStack Summit :: Pimp My Cloud

CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution.** All unlicensed or borrowed works retain their original licenses.

Add custom filtercfg.MultiStrOpt('scheduler_available_filters',

            default=['nova.scheduler.filters.standard_filters'],            help='Filter classes available to the scheduler which may '                    'be specified more than once. An entry of '                    '"nova.scheduler.filters.standard_filters" '                    'maps to all filters included with nova.'),     cfg.ListOpt('scheduler_default_filters',                default=[                  'RetryFilter',                  'AvailabilityZoneFilter',                  'RamFilter',                  'ComputeFilter',                  'ComputeCapabilitiesFilter',                  'ImagePropertiesFilter'                  ],                help='Which filter class names to use for filtering hosts '                      'when not specified in the request.'),

nova.conf settings

scheduler_available_filters=myfilter.CustomFilterscheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,ComputeFilter, ComputeCapabilitiesFilter,ImagePropertiesFilter,CustomFilter

24

Wednesday, October 17, 12

Page 25: OpenStack Summit :: Pimp My Cloud

CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution.** All unlicensed or borrowed works retain their original licenses.

*_driver *_manager

• network• quota• compute• console• db• linuxnet_interface• scheduler

o compute_schedulero volume_schedulero default_scheduler

• baremetal• firewall• libvirt_vif• volume_driver

• compute• console• cert• instance_dns• floating_ip_dns• network• volume• scheduler• metadata• image_cache• consoleauth• scheduler_host

25

Wednesday, October 17, 12

Page 26: OpenStack Summit :: Pimp My Cloud

CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution.** All unlicensed or borrowed works retain their original licenses.

Nova Architecture

26

Driver

manager

Wednesday, October 17, 12

Page 27: OpenStack Summit :: Pimp My Cloud

CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution.** All unlicensed or borrowed works retain their original licenses.

ResourcesBlueprints: https://blueprints.launchpad.net/nova/folsomSource: https://github.com/openstack/nova/tree/stable/folsomDocumentation: http://docs.openstack.org Release notes: http://wiki.openstack.org/ReleaseNotes/Folsom

27

Wednesday, October 17, 12

Page 28: OpenStack Summit :: Pimp My Cloud

CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution.** All unlicensed or borrowed works retain their original licenses.

Questions?

28

Wednesday, October 17, 12