View
6
Download
0
Category
Preview:
Citation preview
BUILDING AND RUNNING OPENSTACK ONPOWER8
LANCE ALBERTSONOregon State University Open Source Lab
@ramereth
0
SUMMARYPOWER8 OverviewPOWER at OSUOSLBuilding a RHEL-based P8 platform with OpenstackArchitecture porting issuesProblems we have encountered with OpenStackOpenstack deployment with ChefOSL Wrapper cookbookNext Steps
0
DISCLAIMER
POWER8 OVERVIEW
POWER8 OVERVIEW
DESIGNDesigned to be a massively multithreaded chipDesigns are available for licensing under the OpenPOWER FoundationLittle-Endian & Big-EndianSeveral non-IBM companies building P8 hardware
Tyan, Rackspace (OpenCompute-based) & Google
OPENPOWER ABSTRACTION LAYER (OPAL)OPAL is the new Open Source firmware for POWER8Acts as an on-system HMCEnables the machine to boot similar to PC serversLinux Kernel and loads the boot loader PetitbootPetitboot provides a shell environment for debugging and setupPetitboot will use kexec and boot into the system kernel
POWER AT OSUOSL
POWER AT OSUOSL
HISTORYProviding PPC64 compute resources since 2005Close collaboration with IBM LTCPOWER5, POWER7 and now POWER8OSL managed LPAR deployment to make it easier on projectsPre-P8 Projects:
Debian, Gentoo, Fedora, PostgreSQLLinux Foundation, Haskell, GoLangMozilla, OpenSUSE, LLVM, GCC
POWER8 AT OSUOSLGoal is to provide on-demand PPC64/PPC64LE compute resources to FOSSprojectsAssist with ppc64/ppc64le porting & testingExpose OSU students to OpenStack and POWER8Collaboration with IBM engineers on architecture issuesCreate a vanilla Openstack cluster for FOSS projects
PROJECTS RUNNING ON OUR P8 CLUSTERCloudFoundry, Docker, CentOS, CouchDBHaskell, Glibc, JXcore, LLVM, NodeJSOpenJDK, GoLang, oVirt, libjpeg-turboBLCR, Gentoo
BUILDING A RHEL-BASED P8 PLATFORM WITH OPENSTACK
BUILDING A RHEL-BASED P8 PLATFORM WITH OPENSTACK
SUPPORTED OS PLATFORMSPowerKVM
Ubuntu
RHEL
DECISION TO USE RHELLittle community support at the time and opportunity to help the communityWe use CentOS internally as our primary OS & more familiar with the RHELeco-systemRHEL has the RDO OpenStack distribution that is well supportedChef support with OpenStack needed some helpI love challenges!
OPENSTACK ARCHITECTURE (OLD)Started in 2014IcehouseController node
Runs all public API services, dashboardDB hosted on a shared bare-metal systemX86_64 CentOS 6 VM running on Ganeti+KVM
Compute node(s)Nova compute and networkingFlat networkingPPC64 Fedora
OPENSTACK ARCHITECTURE (NEW)Deployed 2016 (deployed last week)MitakaController node
Runs all public API services, dashboardDB hosted on a shared bare-metal systemX86_64 CentOS 7 VM running on Ganeti+KVM
Compute node(s)Nova ComputeNeutron Networking
LinuxbridgeProvider and Tenant networking using VXLAN
PPC64LE CentOS 7.2
COMPUTE NODESDid initial development on Fedora 19Fedora 20 PPC64 base system (old)Fedora 21 versions of a few packagesCentOS 7.2 PPC64LE base system (new)
ARCHITECTURE PORTING ISSUES
ARCHITECTURE PORTING ISSUES
CHEFNo PPC64/PPC64LE Chef clientNeeded to build our own chef-clientOmnibus
Bootstrap build envBuild dependency issuesArchitecture configuration issues in Omnibus
Chef has stable ppc64/ppc64le builds today
PACKAGE SUPPORTSupport for P8 was bleeding edge and new features were added weeklyBuilt versions of latest packages from Fedora rawhide packages:
qemulibvirtkernel
Internal repo for these custom packages:
Kernel required a few custom options to be enabledRuntime setup: Disable SMT
http://�p.osuosl.org/pub/osl/repos/yum/openpower/centos-7/ppc64le/
GUEST OS IMAGESFew OS supported ppc64/ppc64le or provided guest images pre-builtVariety of tools which are platform specificMissing support for cloud-initInitially started creating images manually with qemu directly
PACKER -- MULTI PLATFORM SUPPORTWe needed Go to use PackerGoLang support was literally in the worksFinally built our own packer binary last Nov!
WIP Packer Templates:http://�p.osuosl.org/pub/osl/openpower/rpms/
https://github.com/osuosl/bento/tree/ramereth/ppc64
ARCHITECTURE ISSUESOPAL firmware bugspre-P8 machines were very buggyIPMI console would sometimes stop workingRandom lockupsIncluded HW RAID, but no cached write-back support
PROBLEMS WE HAVE ENCOUNTERED WITH OPENSTACK
PROBLEMS WE HAVE ENCOUNTERED WITH OPENSTACK
LEARNING AND UNDERSTANDING OPENSTACKLots of moving piecesNeutron networking is complex and a moving targetDeciding on the proper design architecture for our use case
BUGS AND "FEATURES"Interaction between libvirt and nova-compute was buggy at timesSome bugs were just Icehouse itself, others were architecture specificLearning how to deploy Openstack and making (gasp) mistakes!Iptables issues between Chef and OpenstackProvider networks configures dnsmasq as an open resolverSSL API endpoints
STABILITYRabbitmq would constantly need to be restartednova-compute services would randomly stop workingRunning Fedora on compute and CentOS on controller made things ...interesting
CENTOS 7 PPC64LE ON OPENSTACK
CENTOS 7 PPC64LE ON OPENSTACK
RHEL / CENTOS SUPPORTIntroduced in 7.1 and fully supported in 7.2CentOS community was still bootstrapping and testingWe built our own pre-release CentOS 7.2 for testingUsing ppc64le on compute nodes
RDOCommunity for deploying Openstack on CentOS, Fedora and RHELRepositories built against each PlatformEach release of OpenStack separated
RHEV (RED HAT ENTERPRISE VIRTUALIZATION)Updated KVM packagingPart of the Virt SIG of CentOSUsed SRPMs to build ppc64le versions in a location repoOne patch needed to work around bug
OPENSTACK DEPLOYMENT WITH CHEF
OPENSTACK DEPLOYMENT WITH CHEF
WHY CHEF?Primary CM tool used at the OSLProvides a lot of testing capability on deploymentCan use the full power of the Ruby language for configuring the cluster
CHEF OPENSTACKSet of cookbooks that will deploy the various services of OpenstackPart of the OpenStack umbrellaCommunity drivenDid a major refactor of the code for Mitaka release
OSL OPENSTACKCreated a wrapper cookbook (osl-openstack)
OSL site specific configurationSplit recipes out by upstream cookbook nameContains ppc64le specific changesCurrently only tested on CentOS 7
https://github.com/osuosl-cookbooks/osl-openstack
OSL WRAPPER COOKBOOK
OSL WRAPPER COOKBOOK
RECIPES/DEFAULT.RB
Default configuration for clusterInclude local yum reposInclude command clientsLogic around endpoints
recipes/default.rb
RECIPES/IDENTITY.RB
Just includes recipesSome wrapper, some upstreamAllows us to test just Keystone by itself
recipes/identity.rb
RECIPES/CONTROLLER.RB
Pulls in all wrapper recipes needed to build a controllerAllows for us to split things out eventually if we want to
recipes/controller.rb
TESTING AND DEVELOPMENTUnit Testing
ChefSpecRSpec
Integration TestingTest KitchenServerSpec
Chef ProvisioningDeploy VMs as controller/computeDeploy on bare-metal for a test cluster
UNIT TESTINGEnsure the Chef code is doing what it’s supposed to doEasily test Architecture-specific logicVerify configuration files contain proper settingsExamples:
spec/default_spec.rbspec/compute_controller.rbspec/linuxbridge_spec.rb
UNIT TESTING (OUTPUT)$ rspec spec/default_spec.rb
oslopenstack::default includes cookbook base::ifconfig includes cookbook selinux::permissive includes cookbook yumqemuev includes cookbook openstackcommon includes cookbook openstackcommon::logging includes cookbook openstackcommon::sysctl includes cookbook openstackidentity::openrc includes cookbook openstackcommon::client includes cookbook openstacktelemetry::client setting arch to x86_64 does not add OSLOpenpower repository on x86_64 setting arch to ppc64 add OSLopenpoweropenstack repository on ppc64 setting arch to ppc64le
TEST KITCHEN & SERVERSPECTest Kitchen
Test CLI tool which allows you to execute the configured code on one ormore platformsIntegrates with testing frameworksMust have tool for Chef usersConfigured via
ServerSpecRSpec tests for configured serversIntegration testsEnsures things actually happen on the systemExample:
.kitchen.yml
test/integration/default/serverspec/default_spec.rb
TEST KITCHEN (LIST)$ kitchen listInstance Driver Provisioner Verifier Transport Last Actiondefaultcentos72 Openstack ChefZero Busser Rsync Not Createdmoncentos72 Openstack ChefSolo Busser Rsync Not Createdmoncontrollercentos72 Openstack ChefSolo Busser Rsync Not Createdopsmessagingcentos72 Openstack ChefZero Busser Rsync Not Createdidentitycentos72 Openstack ChefZero Busser Rsync Not Createdimagecentos72 Openstack ChefZero Busser Rsync Not Creatednetworkcentos72 Openstack ChefZero Busser Rsync Not Createdlinuxbridgecentos72 Openstack ChefZero Busser Rsync Not Createdcomputecontrollercentos72 Openstack ChefZero Busser Rsync Not Createdcomputecentos72 Openstack ChefZero Busser Rsync Not Createddashboardcentos72 Openstack ChefZero Busser Rsync Not Createdblockstoragecentos72 Openstack ChefZero Busser Rsync Not Createdblockstoragecontrollercentos72 Openstack ChefZero Busser Rsync Not Createdtelemetrycentos72 Openstack ChefZero Busser Rsync Not Createdcontrollercentos72 Openstack ChefZero Busser Rsync Not Created
TEST KITCHEN (TEST)$ kitchen test default> Starting Kitchen (v1.8.0)> Cleaning up any prior instances of <defaultcentos72>> Destroying <defaultcentos72>... Finished destroying <defaultcentos72> (0m0.00s).> Testing <defaultcentos72>> Creating <defaultcentos72>... OpenStack instance with ID of <a25fa4105caf4f96bddb1e6daddd06d9> is ready....
Chef Client finished, 115/198 resources updated in 03 minutes 12 secondsFinished converging <defaultcentos72> (3m41.31s).> Setting up <defaultcentos72>...Finished setting up <defaultcentos72> (0m0.00s).> Verifying <defaultcentos72>...Preparing files for transfer...
NEXT STEPS
NEXT STEPS
INFRASTRUCTURE NEXT STEPSAdd Nagios checks (DONE!)Continue to fix bugs and other issues as they come upRebuild old Icehouse cluster as Mitaka (no upgrade)Add support for object storageUpdate documentationAdd support for non-live migrationMellanox networking
PROJECT EXPERIENCEImprove and streamline on boarding processExpand cluster’s disk storage capacityImprove stability of the clusterAdd more projects!Submit your request:
http://osuosl.org/services/powerdev/request_hosting
QUESTIONS?Lance Albertson
lance@osuosl.org
@ramereth
– http://osuosl.org http://lancealbertson.com
Links:
http://github.com/ramereth/presentation-openstack-power8https://github.com/osuosl-cookbooks/osl-openstackhttp://osuosl.org/services/powerdev/request_hostinghttp://�p.osuosl.org/pub/osl/repos/yum/openpower/centos-7/ppc64le/http://�p.osuosl.org/pub/osl/openpower/
Attribution-ShareAlike CC BY-SA ©2016
Recommended