Table of Contents - docs.pivotal.iodocs.pivotal.io/archives/mysql-docs-1.7.pdf... , and Service...

23262730313435394045535557

TableofContents

TableofContentsMySQLforPivotalCloudFoundryReleaseNotesKnownIssuesFrequentlyAskedQuestionsClusterScaling,NodeFailure,andQuorumClusterConfigurationProxyforMySQLforPivotalCloudFoundryCreatingApplicationSecurityGroupsforMySQLMonitoringtheMySQLServiceDeterminingClusterStateBootstrappingaGaleraClusterBackingUpMySQLforPivotalCloudFoundryScalingDownMySQLRotatingMySQLforPCFCredentialsRunningmysql-diag

©CopyrightPivotalSoftwareInc,2013-2017 1of59 1.7

MySQL for Pivotal Cloud FoundryThisisdocumentationfortheMySQLforPivotalCloudFoundry (PCF)tile.

ProductSnapshotCurrentMySQLforPCFDetails

Version:v1.7.32

Release Date:August11,2017

Software component versions:MariaDBv10.1.18,Galerav25.3.17

Compatible Ops Manager version(s):v1.6.xthroughv1.10.x

Compatible Elastic Runtime version(s):v1.6.xthroughv1.10.x

vSphere support?Yes

AWS support?Yes

OpenStack support?Yes

IPsec support?Yes

UpgradingtotheLatestVersionConsiderthefollowingcompatibilityinformationbeforeupgradingMySQLforPCF.

Formoreinformation,seethefullProductCompatibilityMatrix .

Ops Manager VersionSupported Upgrades from Imported MySQL InstallationFrom To

v1.3.xv1.2 v1.3

v1.3.2 v1.4.0

v1.4.x and v1.5.x v1.3.2v1.4.0

v1.5.0

v1.4.x - v1.10.x

v1.4.0 v1.5.0

v1.5.0 v1.6.1–v1.6.26

v1.6.1–v1.6.25Nextv1.6.xrelease–v1.6.26

v1.7.0–v1.7.32

v1.7.0–v1.7.31 Nextv1.7.xrelease–v1.7.32

(*)Notethereisaknownissueupgradingsomereleasesofp-mysqlv1.6andv1.7onOpsManagerv1.6.

ReleaseNotesForinformationaboutchangesbetweenversionsofthisproduct,seetheReleaseNotes.

OverviewTheMySQLforPCFproductdeliversafullymanaged,“DatabaseasaService”toCloudFoundryusers.Wheninstalled,thetiledeploysandmaintainsasingleorthree-nodeclusterrunningarecentreleaseofMariaDB ,SQLProxiesforsuper-fastfailover,andServiceBrokersforCloudFoundryintegration.Weworkhardtoshiptheserviceconfiguredwithsanedefaults,followingtheprincipleofleastsurpriseforageneral-userelationaldatabaseservice.

MySQL for PCF 1.7 is no longer supported.Thesupportperiodforversion1.7hasexpired.Tostayuptodatewiththelatestsoftwareandsecurityupdates,pleaseplantoupdatetomorerecentreleasesofMySQLforPCF.

Wheninstalled,developerscanattachadatabasetotheirapplicationsinaslittleastwocommands, cfcreate-service

and cfbind-service

.Connection

credentialsareautomaticallyprovidedinthestandardmanner .Developerscanselectfromamenuofserviceplansoptions,whichareconfiguredbytheplatformoperator.

Twoconfigurationsaresupported:

Single Highly Available

MySQL 1node 3-nodecluster

SQL Proxy 1node 2nodes

Service Broker 1node 2nodes

HighAvailability - Yes

Multi-AZSupport - Yes*

RollingUpgrades - Yes

AutomatedBackups Yes Yes

CustomizablePlans Yes Yes

CustomizableVMInstances Yes Yes

PlanMigrations Yes Yes

EncryptedCommunication Yes✝ Yes✝

EncryptedDataat-rest - -

Long-livedCanaries - -

(*)vSphereonly,v1.7andearlier(✝)RequiresIPSECBOSHplug-in

LimitationsSingleandthree-nodeclustersaretheonlysupportedtopologies.OpsManagerwillallowtheOperatortosetthenumberofinstancestoothervalues,onlyoneandthreeareadvised.Formoreinformation,seeAvoidanevennumberofnodesintheClusterScaling,NodeFailure,andQuorumtopic.

AlthoughtwoProxyinstancesaredeployedbydefault,thereisnoautomationtodirectclientsfromonetotheother.Toaddressthis,configurealoadbalancerasdescribedintheProxysection.

OnlytheInnoDBstorageengineissupported;itisthedefaultstorageenginefornewtables.Useofotherstorageengines(includingMyISAM)mayresultindataloss.

Alldatabasesaremanagedbyshared,multi-tenantserverprocesses.Althoughdataissecurelyisolatedbetweentenantsusinguniquecredentials,applicationperformancemaybeimpactedbynoisyneighbors.

Round-triplatencybetweendatabasenodesmustbelessthanfiveseconds;ifthelatencyishigherthanthis,nodeswillbecomepartitioned.Ifmorethanhalfofclusternodesarepartitioned,theclusterwilllosequorumandbecomeunusableuntilmanuallybootstrapped.

SeealsothelistofKnownLimitations inMariaDBcluster.

KnownIssuesForinformationaboutissuesincurrentreleasesofMySQLforPCF,seeKnownIssues.

Installation1. DownloadtheproductfilefromPivotalNetwork .

2. NavigatetotheOpsManagerInstallationDashboard.

3. ClickImport a ProducttouploadtheproductfiletoyourOpsManagerinstallation.

4. ClickAddnexttotheuploadedproductdescriptionintheAvailableProductsviewtoaddthisproducttoyourstagingarea.

5. ClickthenewlyaddedtiletoreviewconfigurableSettings.

6. ClickApply Changestodeploytheservice.

Settings

ServicePlanAsingleserviceplanenforcesquotasof100megabytesofstorageperdatabaseand40concurrentconnectionsperuserbydefault.UsersofOperationsManagercanconfiguretheseplanquotas.Changestoquotaswillapplytoallexistingdatabaseinstancesaswellasnewinstances.Incalculatingstorageutilization,indexesareincludedalongwithrawtabulardata.

Thenameoftheplanis100mb-devbydefaultandisautomaticallyupdatedifthestoragequotaismodified.Thus,ifthestoragequotaischangedto1024megabytes,thenewdefaultplannamewillbe1024mb-dev.

Note:Afterchangingaplan’sdefinition,allinstancesoftheplanmustbeupdated.Foreachplan,eithertheoperatorortheusermustruncfupdate-serviceSERVICE_INSTANCE-pNEW_PLAN_NAME

onthecommandline.

Further Note:ThisfeaturedoesnotworkproperlyinversionsofMySQLforPCFv1.6.3andearlier.SeetheentryinKnownIssuesfortherecommendedworkaround.

ProvisioningaserviceinstancefromthisplancreatesaMySQLdatabaseonamulti-tenantserver,suitablefordevelopmentworkloads.Bindingapplicationstotheinstancecreatesuniquecredentialsforeachapplicationtoaccessthedatabase.

ProxyTheproxytierisresponsibleforroutingconnectionsfromapplicationstohealthyMariaDBclusternodes,evenintheeventofnodefailure.

ApplicationsareprovidedwithahostnameorIPaddresstoreachadatabasemanagedbytheservice.Formoreinformation,seeApplicationBinding .Bydefault,theMySQLservicewillprovideboundapplicationswiththeIPofthefirstinstanceintheproxytier.Evenifadditionalproxyinstancesaredeployed,clientconnectionswillnotberoutedthroughthem.Thismeansthefirstproxyinstanceisasinglepointoffailure.

In order to eliminate the first proxy instance as a single point of failure, operators must configure a load balancer to route clientconnections to all proxy IPs, and configure the MySQL service to give bound applications a hostname or IP address that resolves tothe load balancer.

ConfiguringaLoadBalancer

Inolderversionsoftheproduct,applicationsweregiventheIPofthesingleMySQLserverinbindcredentials.Whenupgradingtov1.5.0,existingapplicationswillcontinuetofunction,but,totakeadvantageofhighavailabilityfeatures,theymustbereboundtoreceiveeithertheIPofthefirstproxyinstanceortheIP/hostnameofaloadbalancer.

InordertoconfigurealoadbalancerwiththeIPsoftheproxytierbeforev1.5.0isdeployedandpreventapplicationsfromobtainingtheIPofthefirstproxyinstance,theproductenablesanoperatortoconfiguretheIPsthatwillbeassignedtoproxyinstances.ThefollowinginstructionsappliestotheProxysettingspagefortheMySQLproductinOperationManager.

IntheProxy IPsfield,enteralistofIPaddressesthatshouldbeassignedtotheproxyinstances.TheseIPaddressesmustbeintheCIDRrangeconfiguredintheDirectortileandnotbecurrentlyallocatedtoanotherVM.LookattheStatuspagesofothertilestoseewhatIPaddressesareinuse.

IntheBinding Credentials Hostnamefield,enterthehostnameorIPaddressthatshouldbegiventoboundapplicationsforconnectingto

databasesmanagedbytheservice.ThishostnameorIPaddressshouldresolvetoyourloadbalancerandbeconsideredlong-lived.Whenthisfieldismodified,applicationsmustbereboundtoreceiveupdatedcredentials.

ConfigureyourloadbalancertorouteconnectionsforahostnameorIPtotheproxyIPs.Asproxyinstancesarenotsynchronized,werecommendconfiguringyourloadbalancertosendalltraffictooneproxyinstanceatatimeuntilitfails,thenfailovertoanotherproxyinstance.Formoreinformation,seeKnownIssues.

Important:Toconfigureyourloadbalancerwithahealthcheckormonitor,useTCPagainstport1936.Unauthenticatedhealthchecksagainstport3306willcausetheservicetobecomeunavailable,andwillrequiremanualinterventiontofix.

AddingaLoadBalancerafteranInitialDeploy

Ifv1.5.0isinitiallydeployedwithoutaloadbalancerandwithoutproxyIPsconfigured,aloadbalancercanbesetuplatertoremovetheproxyasasinglepointoffailure.However,thereareseveralimplicationstoconsider:

ApplicationswillhavetobereboundtoreceivethehostnameorIPthatresolvestotheloadbalancer.Torebind:unbindyourapplicationfromtheserviceinstance,binditagain,thenrestageyourapplication.Formoreinformation,seeManagingServiceInstanceswiththeCLI .Inordertoavoidunnecessaryrebinding,werecommendconfiguringaloadbalancerbeforedeployingv1.5.0.

InsteadofconfiguringtheproxyIPsinOperationsmanager,usetheIPsthatweredynamicallyassignedbylookingattheStatuspage.ConfigurationofproxyIPsaftertheproductisdeployedwithdynamicallyassignedIPsisnotwellsupported;seeKnownIssues.

ApplicationServiceGroupsYoumustcreateappropriateApplicationSecurityGroups (ASGs)fortheMySQLforPCFinorderforapplicationstohaveaccesstotheservice.

SeeCreatingApplicationSecurityGroupsforMySQLforinstructions.

LifecycleErrandsTwolifecycleerrandsarerunbydefault:thebroker registrarandthesmoke test.ThebrokerregistrarerrandregistersthebrokerwiththeCloudControllerandmakestheserviceplanpublic.Thesmoketesterrandrunsbasicteststovalidatethatserviceinstancescanbecreatedanddeleted,andthatapplicationspushedtoElasticRuntimecanbeboundandwritetoMySQLserviceinstances.BotherrandscanbeturnedonoroffontheLifecycleErrandspageundertheSettingstab.

ResourceConfig

InstanceCapacity

Anoperatorcanconfigurehowmanydatabaseinstancescanbeprovisioned(instancecapacity)byconfiguringtheamountofpersistentdiskallocatedtotheMySQLservernodes.Thebrokerwillprovisionarequesteddatabaseifthereissufficientunreservedpersistentdisk.ThiscanbemanagedusingthePersistentDiskfieldfortheMySQLServerjobintheResourceConfigsettingpageinOperationsManager.Notallpersistentdiskwillbeavailableforinstancecapacity;about2-3GBisreservedforserviceoperation.Addingnodestotheclusterincreasesdurability,notcapacity.Multiplebackendclusters,toincreasecapacityorforisolation,arenotyetsupported.

Indetermininghowmuchpersistentdisktomakeavailablefordatabases,operatorsshouldalsoconsiderthatMariaDBserversrequiresufficientCPU,RAM,andIOPStopromptlyrespondtoclientrequestsforalldatabases.

ProvisioningandBindingviaCloudFoundry

Note:WithoutASGs,theservicewillnotbeusable.

Note:Youmightalsonoticeabroker-deregistrarerrand.Do not run this errand unless instructed to do so by Support.Broker-deregistrarisapartoftheautomationusedbyOpsManagerwhiledeletingatile.Runningthiserrandunderanyothercircumstancewilldeleteuserdata.

AspartofinstallationtheproductisautomaticallyregisteredwithPivotalCloudFoundry ElasticRuntime(seeLifecycleErrands).Onsuccessfulinstallation,theMySQLserviceisavailabletoapplicationdevelopersintheServicesMarketplace,viatheweb-basedDeveloperConsoleor cf

marketplace.

Developerscanthenprovisioninstancesoftheserviceandbindthemtotheirapplications:

$cfcreate-servicep-mysql100mb-devmydb$cfbind-servicemyappmydb$cfrestartmyapp

Formoreinformationabouttheuseofservices,seetheServicesOverview .

ExampleApplicationTohelpapplicationdevelopersgetstartedwithMySQLforPCF,wehaveprovidedanexampleapplication,whichcanbedownloadedhere.InstructionscanbefoundintheincludedREADME.

ServiceInstanceDashboardCloudFoundryuserscanaccessadashboardforeachMySQLserviceinstancesviaSSOfromAppsManager.Thedashboarddisplayscurrentstorageutilizationofthedatabaseandtheplanquotaforstorage.OntheSpacepageinAppsManager,userswiththeSpaceDeveloperrolewillfindaManagelinknexttotheinstance.ClickingthislinkwilllogusersintotheservicedashboardviaSSO.

ConnecttoyourDatabasewiththeMySQLPluginYoucanusetheCloudFoundryCommandLineInterface(cfCLI)MySQLplugintoconnecttotheMySQLdatabasesusedbyyourCloudFoundryapps.Thepluginsupportsthefollowingactions:

Inspectingdatabasesfordebuggingpurposes.

Manuallyadjustingdatabaseschemaorcontentsindevelopmentenvironments.

Dumpingandrestoringdatabases.

Formoreinformation,seethecf-mysql-plugin repository.

ProxyDashboardTheserviceprovidesadashboardwhereadministratorscanobservehealthandmetricsforeachinstanceintheproxytier.Metricsincludethenumberofclientconnectionsroutedtoeachbackenddatabaseclusternode.

Thedashboardforeachproxyinstancecanbefoundat: http://proxy-<jobindex>.p-mysql.<system-domain>

.Jobindexstartsat0soifyouhavetwoproxy

instancesdeployedandyoursystem-domainis example.com ,dashboardswouldbeaccessibleat http://proxy-0.p-mysql.example.com andhttp://proxy-1.p-mysql.example.com .

Basicauthcredentialsarerequiredtoaccessthedashboard.ThesecanbefoundintheCredentialstaboftheMySQLproductinOperationsManager.

FormoreinformationaboutSwitchBoard,readtheproxydocumentation.

SeeAlsoClusterConfiguration

BackingUpMySQLNote:ForinformationaboutbackingupyourPCFinstallation,seeBackingUpandRestoringPivotalCloudFoundry .

DeterminingClusterState

ClusterScaling,NodeFailure,andQuorum

BootstrappingaCluster

ScalingDownMySQL

Release Notes

v1.7.32ReleaseDate:11August,2017

Change the Interruptor’s default setting to OFF.Forayear ,MySQLforPCFhasincludedtheInterruptor.It’saprotectivemechanismwhichstopsanodefromautomaticallyrejoiningtheclusterifdoingsomaydeleteapplicationdata.WealsoupgradedtoMariaDB10.1 andprovidedtheReplicationCanarytofurtherprotectapplicationdata.TherehavebeenzeroinstanceswheretheInterruptorhasbeenneededtoprotectapplicationdata.

Inthisrelease,wearedisablingtheInterruptorbecauseitisdisruptivetonormalclusterfunction,andrequiresmanualOperatoractiontorestoreavailability.WefeelconfidentthatdisablingtheInterruptorinallbutthemostcriticalenvironmentsisasafeandconvenientchoice.

IfyouwishtocontinueusingtheInterruptor,makesurethat“Preventnodeautore-join”ischeckedinthe“AdvancedOptions”configurationpane,thenhitApply Changes.

Upgradesseveraldependenciesincluding nokogiri1.8.0 , golang1.8.3 , xtrabackup2.4.5 , boost1.59.0 ,and python2.7.13

Updatedstemcellto3312.32.Thissecurityupgraderesolvesthefollowing:

USN-3265-2

Formoreinformation,seepivotal.io/security .

v1.7.31ReleaseDate:June22,2017

Newconfigurationpaneforsyslog:Previously,MySQLforPCFusedthesameconfigurationsettingsasElasticRuntime.However,someuserswanttosendMySQLforPCFlogstodestinationsotherthanElasticRuntimelogs.Thus,MySQLforPCFnowhasseparateconfiguration,similartoRabbitMQforPCFandRedisforPCF.Actionrequired:DuringinstallationorupgradeofMySQLforPCF,youmustconfigureordisablesysloggingintheSyslogsettingspane.

USN-3334-1 .

v1.7.30ReleaseDate:June2,2017

USN-3291-3

v1.7.29ReleaseDate:May19,2017

BugfixestoaddressissueswithMySQLforPCFwhentheIPsecadd-on isalsoinstalled:

Bug fix:WhileinstallingMySQLforPCFwithIPsecinstalled,theproductmightfailtodeploy.ThismaybeduetoanissuewherethedefaultprobetimeoutistoolongwhilerunningunderIPsec,andshouldbereduced.Version1.7.29ofMySQLforPCFallowsyoutoreducetheNewClusterProbe

This is the last planned release of MySQL for PCF 1.7.Thesupportperiodforversion1.7hasexpired.Tostayuptodatewiththelatestsoftwareandsecurityupdates,pleaseplantoupgradetomorerecentreleasesofMySQLforPCF.

TimeoutintheMySQLserverconfigurationpage.Formoreinformation,seeOptionsandFeatures inthev1.8documentation.Bug fix:WealsomadeasmallchangeinthewaythatMySQLnodesshutdown,whichshouldbetterallownodestoleavetheclustergracefullywhileIPsecisinstalled.

v1.7.28ReleaseDate:April27,2017

USN-3265-2

Bug fix:AddressedanissuewherebackupswereunabletostorebackupsonAWSS3regionsthatrequirethev4signature.

Bug fix:Addressedanissuewherenodesmayfailtorejointheclusterafterrestart.SeetheRejoinUnsafeFailsKnownIssueformoredetails.

Note:Thetitleofthetilenowappearsas“MySQLforPCF,”notsimply“MySQL.”

Additionalinformationcanbefoundathttps://pivotal.io/security

v1.7.27ReleaseDate:2017April3

Seebelow,sameupdateasv1.6.26

v1.6.26ReleaseDate:2017April3

Updatednokogiritov1.7.1.Thisisasecurityupgradethatresolvesthefollowing:

USN-3235-1

Updatedstemcellto3263.22.Thisisasecurityupgradethatresolvesthefollowing:

USN-3249-2

v1.7.26ReleaseDate:2017March23

UpdateddependencyofGolanguagetov1.7.Thischangedoesnotimpactfunctionality,itisdoneonlytokeepuptodatewithsupportedversionsofGo.

Bug fix:Changedthevalueof wsrep_max_ws_rows to0topreventMariaDBbugMDEV-11817 fromaffectingDDLs.

Updatedstemcellto3263.21.Thisisasecurityupgradethatresolvesthefollowing:

USN-3220-2

v1.7.24ReleaseDate:2017February24

Updatedstemcelltov3263.20toresolvethefollowing:

USN-3208-2

v1.7.23ReleaseDate:26January2017

Updatedstemcelltov3263.17,whichisaroutinepatchupdatetoaddressmediumandlowsecurityvulnerabilities.

v1.7.22ReleaseDate:21December2016

Resolvesanissuewhichpreventsupgradingpreviousinstallationsofp-mysqltov1.7.19throughv1.7.22onOpsManagerv1.6andearlier.

Updatedstemcelltov3263.14.

Resolvesanissuewhichpreventsupgradingpreviousinstallationsofp-mysqltov1.6.20andv1.6.21onOpsManagerv1.6andearlier.

USN-3156-1

USN-3151-2

v1.7.19ReleaseDate:14November2016

Updatedstemcelltov3233.4toaddressstandardsecurityupdates.

UpdatedMariaDBtov10.1.18toresolveavarietyofunspecifiedsecurityvulnerabilities.

Updatedmysql-backup-releasetov1.27.3.Whenbackupsaredisabled,thebackupserveroneachnodeisalsodisabled.

v1.7.18ReleaseDate:26October2016

Updatedstemcelltov3233.3,sameupdateasv1.6.19.

Security:Updatetheservicebrokertopreventloggingofservicecredentials.

UpdatedMariaDBtov10.1.18toresolveavarietyofunspecifiedsecurityvulnerabilities.

Updatedstemcelltov3233.3.Thisisasecurityupgradethatresolvesthefollowing:

USN-3106-2

USN-3099-2

Includesstabilityandbugfixes.

USN-3087-2

UpgradestheLinuxv3.19kerneltov4.4.

v1.7.14ReleaseDate:28September2016

Seebelow,sameupdateasv1.6.16.

USN-3087-2

Note:UpdatedMariaDBtov10.1.17.Theupgradeisautomatic,andifdeployedinHAconfigurationwillnotcausedowntimeforapplications.

Updatedstemcelltov3232.19.Thisisaroutinesecurityupgradethatresolveslowandmediumvulnerabilities.

UpdatedRubyandRailssoftwaretoadditionalresolvesecurityvulnerabilities.

Bug fix:Introducedafixtothereplicationcanarywhichreducesthepossibilityoffalsepositives.

Updatedstemcelltov3232.19.Thisisaroutinesecurityupgradethatresolveslowandmediumvulnerabilities.

UpdatedRubyandRailssoftwaretoadditionalresolvesecurityvulnerabilities.

v1.7.12Seebelow,sameupdateasv1.6.14.

Bug fix:AddressesabuginOpsManagerv1.7.0,inwhichupgradingtoarecentversionofPivotalMySQLcausesOpsManagertoissueaninternalservererror.

v1.6.14ReleaseDate:24August2016

USN-3064-1

USN-3048-1

USN-3060-1

USN-3061-1

USN-3065-1

v1.7.11ReleaseDate:27July2016

We’vediscoveredarareconditionwhereaMySQLclusterexperiencesafaultinreplicationthatcanresultinsomedataloss.Whenthisoccurs,previousreleasesdonotlogtherootcauseofthebug.Inordertobestaddressthisissue,v1.7.11containssignificantadditionaltelemetryandseveraldefensivefeatureswhichwillaccountforthefailureconditionandpreventdataloss.

Introducing the Replication CanaryWe’veincludedanewlong-runningmonitor,theReplication Canary.TheReplicationCanarycontinuallymonitorstheMySQLcluster,watchingforinstancesinwhichcross-clusterreplicationhasfailed.Itisenabledbydefault,andrequiresane-mailaddressintheAdvancedOptionsconfigurationpane.

If any of these protections activate, it is critical that you contact Pivotal support immediately.Supportwillworkwithyoutodeterminethenatureofthecluster’sfailure,andadviseasuggestedresolution.Additionally,contactingSupportwillprovideuswithevidencethatwillenableustoidentifyandaddresstherootcauseinthefuture.

Intheeventthatreplicationhasfailed,theCanaryperformstwoactions:

E-mailtheOperator:PartoftheReplicationCanary’sconfigurationisane-mailaddress,whichcanbedirectedtoanyOperatore-mailaddress,oranescalationsystemsimilartoPagerDuty.DenyAccess:Whenreplicationhasfailed,theReplicationCanarywillautomaticallydisableuserandapplications’abilitytoaccesstheclusterviatheProxies.

Youmustsetthe Monitoring jobto1intheResourceConfigpane,ortheReplicationCanarywillnotbeenabled,regardlessofconfiguration.YoumustalsoconfirmthattheElasticRuntimetileisproperlyconfiguredtosende-mail.ThesesettingsarenecessaryforanystandardCloudFoundryconfiguration.

Ensurethatthe Notifications errandhasbeenenabled.Ensurethat SMTP Config hasbeenproperlyconfigured.

Ifeitherofthesearenotset,configureandApply Changesbeforedeployingv1.7.11.

FormoreinformationabouttheReplicationCanary,seethemonitoringdocumentation.

Introducing the InterruptorTheMySQLnodeshavenewlogicthat,whenenabled,willpreventanodefromre-joiningaclusterundercertainconditions.Thisisasecondlevelofprotectionagainstthepossibilityofdataloss.

FormoreinformationabouttheInterruptor,seethemonitoringdocumentation.

New feature:BackingupallnodesIntheBackupsconfigurationpane,there’snowanoptiontotakebackupsfromallMySQLnodes.Thisfeatureprotectsyourusersfromdatalossinthecasethatsomenodeshavedifferentdatathantheothers.

LoggingChanges

MySQLjoblogsarekeptlocalontheVM,inadditiontosenttosyslogifconfigured.Binarylogsarenowenabledandrotatedautomaticallybythesystem.Plusahostofdebuglogchangeshavebeenaddedtoaidindiagnosisefforts.

XATransactionsarenowdisallowed.

XATransactionsarenotcompatiblewithourHAtechnology.

QuotaEnforcerisnowconfigurable.

Maximumopenfiledescriptorsnowdefaultto65Kforlargedatabases.

Security fix:NowincludesMariaDB10.0.23

Avoidsapossiblecredentialleak.

v1.7.10Seebelow,sameupdateasv1.6.13

USN-3020-1

Additionalinformationcanbefoundathttps://pivotal.io/security .

v1.7.9

Note:Duetotheseriousnatureofafailureinreplication,bothbehaviorsareenabledbydefault.Duringconfiguration,youmayelecttosettheReplicationCanarytonotify-onlymode,butthisisnotrecommended.

v1.6.12ReleaseDate:16June2016

USN-3001-1

v1.6.11ReleaseDate:18May2016

Updatedstemcelltov3232.4Thisisasecurityupgradethatresolvesthefollowing:

USN-2977-1

v1.6.10ReleaseDate:06May2016

Updatedstemcelltov3146.11Thisisasecurityupgradethatresolvesthefollowing:

USN-2959-1

Bug fix:UpdatedacceptanceteststopassonPCFv1.7.

Bug fix:Update broker-registrar toavoidrunawayCPUconditiononbrokerVMs.

USN-2929-1

USN-2910-1

USN-2900-1 ,acriticalGNUClib(glibc)CVEUSN-2897-1

USN-2896-1

USN-2882-1

USN-2879-1

USN-2875-1

USN-2874-1

USN-2871-1

USN-2868-1

USN-2865-1

USN-2861-1

v1.6.5

ReleaseDate:18January2016

USN-2869-1

CVE-2016-0715 .

Updatedstemcelltov3146.2.ThisisasecurityupgradethatresolvesthefollowingUbuntuSecurityNotices:

USN-2857-1,USN-2842-1,USN-2842-2,USN-2836-1,USN-2834-1,USN-2830-1,andUSN-2829-1

v1.7.0.4Seebelow,sameupdateasv1.6.3.4

v1.6.3.4ReleaseDate:04December2015

Addressesanissuewherechangingthemaximumnumberofallowedconnectionsintheserviceplandoesnotaffectthemaximumnumberofallowedconnectionsinserviceinstances,neworexisting.NotethattheKnownIssueforChangingServicePlanDefinitionstillapplies;you’llstillneedtorunthemanualworkaroundforexistinginstances.PleaselookforimprovementsinafuturereleaseofMySQLforPivotalCloudFoundry(PCF),wearesorryfortheinconvenience.

Updatedstemcelltov3146.ThisisasecurityupgradethatresolvesthefollowingUbuntuSecurityNotices:

[USN-2821-1 ]GnuTLSvulnerability

v1.6.3.3ReleaseDate:02December2015

Updatedstemcelltov3144.ThisisaregularsecurityupgradethatresolvesthefollowingUbuntuSecurityNotices:

[USN-2815-1 ]libpngvulnerabilities[USN-2812-1 ]libxml2vulnerabilities[USN-2810-1 ]Kerberosvulnerabilities

v1.6.3.2ReleaseDate:11November2015

Updatedstemcelltov3130.Thisisaregularsecurityupgradethatresolvesthefollowingissues:

[USN-2806-1 ]Linuxkernel(VividHWE)vulnerability[USN-2798-1 ]Linuxkernel(VividHWE)vulnerabilities

v1.6.3.1ReleaseDate:03November2015

[USN-2778-1 ]Linuxkernel(VividHWE)vulnerabilities

New Feature:AutomatedOperator-configureddatabasebackupsforDisasterRecovery.

UpdatedMariaDBtov10.0.21 whichalsoincludesupdatesfromMariaDBv10.0.20 .

Updatedstemcelltov3100.

Security:FixesforCVE-2015-3900,aman-in-the-middlerubygemsvulnerability.

Bugfix:SwitchboardfailstofindrecreatedmysqlnodewhenARPcachelockedbyhangingSYN_SENT.

Bugfix:EveryinstanceofSwitchboardregistrarstheroute proxy-0.p-mysql. ratherthanchangingbasedonAZindex.

Bugfix:MySQLforPCFnowsupportstheElasticRuntimesettingtorestrictHAProxytraffictoHTTPSonly.UsersofMySQLforPCFv1.6andearliermustupgradetoMySQLforPCFv1.7orlaterinordertousethisfeature.

NowhonorsOpenIDConnectIDtokenswheninteractingwithUAA.

Thiswillpreventblankpageswhenclicking‘Manage’inAppsManagerwithmorerecentversionsofElasticRuntime.

[USN-2765-1 ]Linuxkernel(VividHWE)vulnerability

[USN-2694-1]PCREvulnerabilities[USN-2698-1]SQLitevulnerabilities[USN-2710-1]OpenSSHvulnerabilities[USN-2710-2]OpenSSHregression

[USN-2718-1]Linuxkernel(VividHWE)vulnerability

KnownIssuesExperimentalfeatureHTTPStraffictoHAProxydoesnotwork;itwillbefixedinanupcomingrelease.

Updatedstemcelltov3026toresolveCVE-2015-3290

v1.6.0NowincludesMariaDBv10.0.19andGalerav5.5.43(releasenotes )

IncludesseveraldefaultconfigurationchangestobettermanageMariaDB’smemoryanddiskusageduringperiodsofheavyuse.

Improved stability:Thisversionincludesanall-newQuotaEnforcerforenhancedstabilityandinpreparationfornewfeaturesinfuturereleases.

Improved stability:Nowprovidesgreaterstabilityduringclusterrecoverybyusingthextrabackup-v2replicationmechanism.

UpdatestobothServiceandProxydashboardstosupporttheexperimentalHTTPS-onlyfeatureinElasticRuntimev1.5

NowusestheMariaDBconnectorratherthanadditionallyincludingtheMySQLconnector.

Security:TheMySQLdeploymentnowrunsasuservcap,notroot.

Security:UpgradedRubyandRailscomponentstoaddressvariousCVEs.

Bug fix:Onceoverquota,writeprivilegesarenotrestoredbydroppingalltables.

Bug fix:Thebroker-deregistrarerrandnowsucceedsevenwhenaMySQLserviceisbroken.

Bug fix:ServiceBrokerdashboardshouldnotreturn500ifOAuthaccesstokenexpires.

Upgrade support:Thisproductcanbeautomaticallyupgradedfromv1.5.0

Documentationnowincludesseveralnewsections:

NotesonclusterconfigurationDeterminingMySQLclusterstateBackgroundonClusterScaling,NodeFailure,andQuorumBootstrappinganailingMySQLcluster

Note:BOSHStemcellv3026isrequired;thisstemcellisprovidedbyOpsManagerv1.5.1.

AWS support:TheclustereddatabaseservicecannowbedeployedonAmazonWebServicesfromtheOperationsManagerWebUI.

DeploymentislimitedtoasingleAvailabilityZone.Lookformulti-AZinfuturereleases.SingleavailabilityzoneisalimitationonAWS.OperationsManageronvSpherecontinuestosupportdeploymenttomultipleavailabilityzones.ThedefaultinstancetypefortheclusternodesonAWSism3.large.AlljobsaredeployedwithSSDforephemeralandpersistentdisk.

IaaS agnostic

ThesameproductcanbedeployedtobothAWSandvSpherePrecompiledpackagesarenolongerincludedMySQLforPCFv1.5.0requiresOpsManagerv1.4.0

New proxy tier

Improvedavailability:Wehaveentirelyre-writtentheproxytoeliminatesituationswhereclientscouldhangwhenaclusternodewasunhealthy.

Adashboardthatclearlydisplaysnodehealthinrealtime

Upgrade support:Thisproductcanbeautomaticallyupgradedfromv1.3.2orv1.4.0

Cluster node resources increased for vSphere:Thedefaultresourcesarenow4GBRAM,2CPU,10GBpersistentdisk

Faster compilation:DefaultresourceforthecompilationjobsonvSpherearenow4GBRAM,4CPU,20GBpersistentdisk

Bug fix:Fixbroker-deregistrarerrandtosucceedevenwhenMySQLserviceisbroken

Bug fix:Quotaenforcercouldfailwhenbrokerhasn’tfinishedinitializing

Known issues:

OnAWS,thisversionsupportsdeploymentsintheUS-Eastregion.Multi-regionsupportiscominginafuturerelease.

TheexperimentalHTTPS-onlyfeatureinElasticRuntimev1.5maycauseissueswiththisversionoftheproduct.FullsupportforHTTPS-onlytrafficiscominginafuturerelease.

Note:BOSHStemcellv2865.1isrequiredforinstallationonOpsManagerv1.5.xandabove.

High Availability:databaseserverisnowclusteredandsynchronouslyreplicatedusingMariaDBGaleraCluster.Acopyofeachdatabaseresidesonallclusternodes,andwritestoanydatabasearereplicatedtoallcopies.Allclientconnectionsareroutedtoaprimaryclusternode,andintheeventofanodefailuretheproxytiermanagesfailover,routingclientconnectionstoahealthyclusternode.MySQLserver,proxy,andbrokerjobscanallbescaledouthorizontallyforincreasedavailability,eliminatingsinglepointsoffailure.

Improved logging and monitoring:route-registrationonthebrokerisnowanindependentprocess

Bug fix:calculationofstorageutilizationforthepurposesofquotaenforcementwhenmultipleappsarebound

Bug fix:formatofjdbcUrlconnectionstring(foundinVCAP_SERVICESonbind)

NotesonHighAvailabilityWhenupgradingfromanolderversion,applicationsmustbereboundtotakeadvantageofhighavailabilityfeatures.Torebind:unbindyourapplicationfromtheserviceinstance,binditagain,thenrestageyourapplication.FormoreinformationseeManagingServiceInstanceswiththeCLI.

Eliminationoftheproxyasasinglepointoffailurerequiresconfigurationofanexternalloadbalancertorouteconnectionstoproxyinstances.Fordetails,seeProxySettings.

SeeKnownIssues.

Updated stemcell addresses bash-shellshock vulnerabilities:resolvesCVEsdiscussedhere andhere .

v1.3.0Syslog forwarding:SyslogsarenowstreamedtothesamehostandportconfiguredinElasticRuntimesettings

Dynamic instance capacity management:Previouslyoperatorshadtomanuallyconfigurethemaximumnumberofserviceinstancespermittedbytheserver.Thisrequiredmanualcalculationandaknowledgeofrequiredsystemheadroom.Adminscannowmanageinstancecapacitysimplybyadjustingpersistentdiskallocatedtomysqlnodes.Remaininginstancecapacityisdetermineddynamicallybysubtractingasafeestimateforsystemheadroomandreservedstorageforprovisionedinstances.

Trusty stemcell:ServerandbrokerarenowdeployedonUbuntu“Trusty”v14.04LTSstemcells,providingimprovedsecurity,performance,andasmallerresourcefootprint.

Least necessary privileges:TheMySQLservicedashboardusesanew,limitedpermissionOAuthscopetodeterminewhetherausercurrentlyhasaccesstoaserviceinstance.Thedashboardnolongerhasfullreadaccesstoauser’saccount.

Precompiled packages:Mostpackageshavebeenprecompiledforthetargetedstemcell.Thiswilllowerinitialdeploymenttimes,atthecostofalargerdownload.

v1.2.0Productrenamedto'MySQLforPivotalCF’

Planattributesareconfigurable:maxstorageperdatabase,maxconcurrentconnectionsperuser,andmaxdatabases

Plannameisdetermineddynamicallybasedonconfiguredstoragequota

Planfeaturesincludedisclaimerthattheserviceisnotforproductionuse

DeveloperscanSSOtoaservicedashboardthatdisplaysstorageutilization

SecurityfixesincludingupdatestoRails

ServicebrokerisregisteredbyURL(ratherthanbyIP).Typicallyhastheformat https://p-mysql.<cf-domain> .

Lifecycleerrandsareusedtoregisterthebrokerandrunteststhatverifythedeployment.

Improvedlogginginservicebroker

Thefollowingcomponentswillbere-deployed:cf-mysql-broker

Newcomponents:broker-registrar

broker-deregistrar

acceptance-tests

v1.1.0Updatedtheformatofmetadatafieldsinthebrokercatalogendpointandaddedadditionalfields.Formoreinformation,seeCatalogMetadata.

UpdatedRubytov2.0.0p353tofixavulnerabilityinv1.9.3p448.

Requeststodeleteaserviceinstanceorbindingnowgeta200responsewithanemptyJSONbodyinsteadofa204.

Thebrokernowreturnsaclearerrorwhenthereisnomorecapacityforadditionalinstancesduringaprovisionrequest.Theresponsehasstatuscode507 .Theuser-facingerrormessageis“Serviceplancapacityhasbeenreached.”

Thefollowingcomponentswillbere-deployed:cf-mysql-broker

Known Issues

UnabletoUpgradefromPreviousVersionsonOpsManagerv1.6MySQLforPivotalCloudFoundry(PCF)v1.6.20throughv1.6.21andv1.7.19throughv1.7.21arenotabletoupgradefrompreviousdeploymentsofMySQLonOpsManagerv1.6andearlier.Thishasbeencorrectedinv1.6.22andv1.7.22.

ThisisasampleerrorfromOpsManagerwhenattemptingtoupgradetooneoftheaffectedversions:

Product'MySQLforPivotalCloudFoundry'couldnotbeupgradedfrom'1.7.11'to'1.7.21'.PleasecontactyourPivotalrepresentative.

MySQLBackupstoAWSS3LimitedtoStandardRegionInMySQLforPCFv1.7,backupsareonlysenttoAWSS3bucketsthathavebeencreatedintheUSStandard region,“us-east-1.”Thislimitationhasbeenresolvedinv1.8.0-Edge.2andlater.

ElasticRuntimeHTTPS-onlyFeatureSupportfortheExperimentalHTTPS-onlyfeatureisbrokeninMySQLforPCFv1.6.xandearlier.TheHTTPS-onlyfeatureworksasdesignedinMySQLforPCFv1.7.0andlater.

AccidentalDeletionofaServicePlanIfandonlyiftheOperatordoesallofthesestepsinsequence,aplanwillbecome“unrecoverable”:

1. Clickthetrash-caniconintheServicePlanscreen.

2. Enteraplanwiththeexactsamename.

3. ClickSaveonthesamescreen.

4. ReturntotheOpsManagertop-level,andclickApply Changes.

AfterclickingApply Changes,thedeploywilleventuallyfailwiththeerror:

Servererror,statuscode:502,errorcode:270012,message:Servicebrokercatalogisinvalid:Plannamesmustbeuniquewithinaservice

Thisunfortunatesituationisunavoidable;aftertheOperatorhascommittedwithApply Changes,theoriginalplancannotberecovered.Foraslongasserviceinstancesofthatplanexist,youmaynotenteranewplanofthesamename.Atthispoint,theonlyworkaroundistocreateanewplanwiththesamespecifications,butspecifyadifferentname.Existinginstanceswillcontinuetoappearundertheoldplanname,butnewinstanceswillneedtobecreatedusingthenewplanname.

Ifyouhavecommittedsteps1and2,butnot4,noproblem.Donothitthe‘Save’button.SimplyreturntotheInstallationDashboard.Anyaccidentalchangeswillbediscarded.

Ifyouhavecommittedsteps1,2and3,donotclick'ApplyChanges.’Instead,returntotheInstallationDashboardandclickthe’Revert’button.Anyaccidentalchangeswillbediscarded.

ChangingServicePlanDefinitionInMySQLforPCFv1.7.0andearlier,thereisonlyoneserviceplan.Changingthedefinitionofthatplan,thenumberofmegabytes,numberofconnections,orboth,willmakeitsothatanynewserviceinstanceswillhavethosecharacteristics.

ThereisabuginMySQLforPCFv1.7andearlier.Changingtheplandoesnotchangeexistingserviceinstances.Existingplanswillcontinuetobegovernedbytheplanconstraintseffectiveatthetimetheywerecreated.Thisistrueregardlessofwhetherornotanoperatorruns cfupdate-service .

Thereisaworkaroundforthisbug,whichwillberesolvedinfuturereleasesofMySQLforPCF.Inorderforthechangetobeeffectiveforexistingplans,youmusttriggerthisbyinteractingdirectlywiththeservicebroker:curl-v-k-XPATCHhttps://BROKER_CREDS_USERNAME:BROKER_CREDS_PASSWORD@p-mysql.SYSTEM.DOMAIN.example.com/v2/service_instances/SERVICE_INSTANCE_ID?plan_id=UNIQUE_ID

SYSTEM.DOMAINisdefinedinOpsManager,underElasticRuntime’sSettingstab,inthe Cloud Controller entry.

BROKER_CREDS_USERNAMEandBROKER_CREDS_PASSWORDaredefinedinOpsManager,underMySQLforPCF’sCredentialstab,intheBroker Auth Credentials entry.

TogettheUNIQUE_IDfortheserviceplan:1. Run cf curl /v2/services2. Intheoutput,findthedataobjectwiththe entity > label and description fortheservicewiththechangedplan,andrecordits

metadata > guid value.3. Run cf curl /v2/services/SERVICE-GUID/service_plans withtheserviceGUIDfromthelaststep.4. Intheoutput,findthedataobjectwiththe entity > label and description forthechangedserviceplan,andrecordits unique_id value.

ThisistheUNIQUE_IDvaluetopasstothe plan_id argumentabove;donotusetheplan’sGUID.

TogeteachSERVICE_INSTANCE_ID,run cf service INSTANCE --guid .Youshouldseeoutputlikethisexample:

$cfserviceacceptDB--guid4cae3a5e-66b1-4c9a-8536-feaff25237bf

Runthis curl commandforeachserviceinstancetobeupdated.

Furthermore,ifyouhavechangedthemaxnumberofconnectionsconstraint,thenyouneedtoupdateeachboundapplication’ssettingdirectlyfromtheMySQLconsole.Followthesesteps:

1. SSHintoyourOpsManagerDirectorusingtheseinstructions .

2. Run bosh deployments todiscoverthenameofyourMySQLforPCFdeployment.

3. Run bosh ssh usingyourMySQLforPCF’sdeploymentname.Example: bosh ssh mysql-partition-9d32f5601988152e869b/0

4. Run /var/vcap/packages/mariadb/bin/mysql -u root -p .Therootuser’spasswordisdefinedinOpsManager,underMySQLforPCF’sCredentialstab.

5. IssuethisMySQLcommand:UPDATE mysql.user SET mysql.user.max_user_connections=NEW_MAX_CONN_VALUE WHERE mysql.user.User NOT LIKE '%root%' ;

Makesuretochange NEW_MAX_CONN_VALUE towhatevernewsettingyou’vechosen.

6. exit;

ProxiesMayWritetoDifferentMySQLMastersAllproxyinstancesusethesamemethodtodetermineclusterhealth.However,certainconditionsmaycausetheproxyinstancestoroutetodifferentnodes,forexampleafterbriefclusternodefailures.

Thiscouldbeanissuefortablesthatreceivemanyconcurrentwrites.Multipleclientswritingtothesametablecouldobtainlocksonthesamerow,resultinginadeadlock.Onecommitwillsucceedandallotherswillfailandmustberetried.Thiscanbepreventedbyconfiguringyourloadbalancertorouteconnectionstoonly one proxy instance at a time.

NumberofProxyInstancesCannotbeReducedAftertheproductisdeployedwithoperator-configuredproxyIPs,thenumberofproxyinstancescannotbereduced,norcantheconfiguredIPsberemovedfromtheProxy IPsfield.IfinsteadtheproductisinitiallydeployedwithoutproxyIPs,IPsaddedtotheProxy IPsfieldwillonlybeusedwhenaddingadditionalproxyinstances,scalingdownisunpredictablypermitted,andthefirstproxyinstancecanneverbeassignedanoperator-configuredIP.

BackupsMetadataInMySQLforPCFv1.7.0,both compressed and encrypted showas N inthebackupmetadatafile.ThisisduetothefactthatMySQLforPCFimplementscompressionandencryptionoutsideofthetoolusedtogeneratethefile.Thisisaknowndefect,andwillbecorrectedinfuturereleases.

MyISAMTablesTheclusteringpluginusedinthisrelease(Galera)doesnotsupportreplicationofMyISAMTables.However,theservicedoesnotpreventthecreationofMyISAMtables.WhenMyISAMtablesarecreated,thetableswillbecreatedoneverynode(DDLstatementsarereplicated),butdatawrittentoanodewon’tbereplicated.Ifthepersistentdiskislostonthenodewheredataiswrittento(forMyISAMtablesonly),datawillbelost.TochangeatablefromMyISAMtoInnoDB,followthisguide .

MaxUserConnectionsWhenupdatingthe max_user_connections propertyforanexistingplan,theconnectionscurrentlyopenwillnotbeaffected.Forexample,ifyouhavedecreasedfrom20to40,userswith40openconnectionswillkeepthemopen.Toforcethechangesuponuserswithopenconnections,anoperatorcanrestarttheproxyjob.Thiswillcausetheconnectionstoreconnectandstaywithinthelimit.Otherwise,ifanyconnectionabovethelimitisreset,itwon’tbeabletoreconnect,sothenumberofconnectionswilleventuallyconvergeonthenewlimit.

LongSSTTransfersWeprovidea database_startup_timeout inourmanifestwhichspecifieshowlongtowaitfortheinitialSSTtocomplete(defaultis150seconds).IftheSSTtakeslongerthanthisamountoftime,thejobwillreportasfailing.Versionsbefore cf-mysql-releasev23 haveaflawinourstartupscriptwhereitdoesnotkillthemysqldprocessinthiscase.Whenmonitrestartsthisprocess,itseesthatmysqlisstillrunningandexitswithoutwritinganewpidfile.Thismeansthejobwillcontinuetoreportasfailing.TheonlywaytofixthisistoSSHontothefailingnode,killthemysqldprocess,andre-run monitstart

mariadb_ctrl.

LoadBalancerTimeoutInterruptsLong-RunningQueriesTosomeloadbalancers,aconnectionthatwaitsforresultsappearstobeanidleconnection.Theselong-runningqueriesmaybeinterruptediftheyexceedtheidletimeoutoftheloadbalancer.Thefollowingerroristypicalofsuchaninterruption:

LostconnectiontoMySQLserverduringquery

Forexample,theAWSElasticLoadBalancer hasadefaultidletimeoutof60seconds.Ifaquerytakeslongerthanthisduration,theELBseverstheMySQLconnectionandreturnsanerror.

Topreventthesetimeouts,increasetheidletimeoutdurationaccordingly.

Frequently Asked Questions

ManyreplicationerrorsinthelogsIseelotsofreplicationerrorsinmylogs!Istheclusterbroken?

UnlesstheGRAfilesshowaclearexecutionerror(e.g.,outofdiskspace)thisisanormalbehavior,andit’snothingtoworryabout.Wewillbeworkingonmoreadvancedmonitoringtodetectthefailurecase,andalertOperatorsinthefuture.

Occasionally,you’llseereplicationerrorsintheMySQLlogsthatwilllooksomethinglikethis:1603189:25:16[Warning]WSREP:RBRevent1Queryapplywarning:1,169924561603189:25:16[Warning]WSREP:IgnoringerrorforTOisolatedaction:source:abcd1234-abcd-1234-abcd-1234abcd1234version:3local:0state:APPLYINGflags:65conn_id:246804trx_id:-1seqnos(l:865022,g:16992456,s:16992455,d:16992455,ts:2530660989030983)1603189:25:16[ERROR]SlaveSQL:Error'Duplicatecolumnname'number''onquery.Defaultdatabase:'cf_0123456_1234_abcd_1234_abcd1234abcd'.Query:'ALTERTABLE...'

Whatthisissayingisthatsomeone(probablyanapp)issuedan“ALTERTABLE”commandthatfailedtoapplytothecurrentschema.Moreoftenthannot,thisisusererror.

ThenodethatreceivestherequestprocessesitasanyMySQLserverwill,ifitfails,itjustspitsthatfailurebacktotheapp,andtheappneedstodecidewhattodonext.Thatpartisnormal.HOWEVER,inaGaleracluster,allDDLisreplicated,andallreplicationfailuresarelogged.Sointhiscase,thebadALTERTABLEcommandwillberunbybothslavenodes,andifitfails,thoseslavenodeswilllogitasa“replicationfailure”sincetheycan’ttellthedifference.

It’sreallyhardtogetavalidDDLtoworkonsomenodes,yetfailonothers.Usuallythosecasesarelimitedtooutofdiskspaceorworkingmemory.Wehaven’tduplicatedthatyet.

ButIfoundablogarticlethatsuggeststhattheschematacangetoutofsync?

https://www.percona.com/blog/2014/07/21/a-schema-change-inconsistency-with-galera-cluster-for-mysql/

ThekeythingaboutthispostisthathehadtodeliberatelyswitchanodetoRSU,whichMySQLforPivotalCloudFoundry(PCF)neverdoesexceptduringSST.Sothisisademonstrationofwhatispossible,butdoesnotexplainhowacustomermayactuallyexperiencethisinproduction.

MySQLhasblacklisteditsownproxy?Whatdoestheerror, blockedbecauseofmanyconnection

errorsmean?

TherearetimeswhenMySQLwillblacklistitsownproxies:OUT07:44:02.070[paasEnv=MYPASSorgName=MYORGspaceName=MYSPACEappName=dc-routingappId=0123456789][http-nio-8080-exec-5]ERRORo.h.e.jdbc.spi.SqlExceptionHelper-Host'192.0.2.15'isblockedbecauseofmanyconnectionerrors;unblockwith'mysqladminflush-hosts'

YoucansolvethisbyrunningthefollowingonanyoftheMySQLjobVMS:

/var/vcap/jobs/mysql/packages/mariadb/bin/mysqladminflush-hosts

Thisisanartifactofanautomaticpolling-protectionfeature builtintoMySQLandMariaDB.ItisahistoricalfeatureintendedtoblockDenialofServiceattacks.ItisusuallytriggeredbyaLoadBalancerorSystemMonitoringsoftwareperformingempty“portchecks”againsttheMySQLproxies.ThisiswhyitisimportanttoconfigureanyLoadBalancertoperformTCPchecksagainsttheproxyhealth-checkport,default1936.Repeatedportchecksagainst3306willcauseanoutageforallMySQLforPivotalCloudFoundry(PCF)users.

Note:ThisissuehasbeendisabledasofMySQLforPivotalCloudFoundry(PCF)v1.8.0-edge.4.

Cluster Scaling, Node Failure, and QuorumDocumentedherearescenariosinwhichthesizeofaclustermaychange,howtheclusterbehaves,andhowtorestoreservicefunctionwhenimpacted.GaleraCluster isusedtomanagetheMariaDB clusterinourrelease.

HealthyClusterGaleradocumentationreferstonodesinahealthyclusterasbeingpartofaprimarycomponent .Thesenodeswillrespondnormallytoallqueries,reads,writes,anddatabasemodifications.

Ifanindividualnodeisunabletoconnecttotherestofthecluster(ex:networkpartition)itbecomesnon-primary(stopsacceptingwritesanddatabasemodifications).Inthiscase,therestoftheclustershouldcontinuetofunctionnormally.Anon-primarynodemayeventuallyregainconnectivityandrejointheprimarycomponent.

Ifmorethanhalfofthenodesinaclusterarenolongerabletoconnecttoeachother,alloftheremainingnodeslosequorumandbecomenon-primary.Inthiscase,theclustermustbemanuallyrestarted,asdocumentedinthebootstrappingdocs.

GracefulremovalofanodeShuttingdownanodewithmonit(ordecreasingclustersizebyone)willcausethenodetogracefullyleavethecluster.

Clustersizeisreducedbyoneandmaintainshealthystate.Clusterwillcontinuetooperate,evenwithasinglenode,aslongasothernodesleftgracefully.

AddingnewnodesWhennewnodesareaddedtoorremovedfromaMySQLservice,atop-levelpropertyisupdatedwiththenewnodes’IPaddresses.AsBOSHdeploys,itwillupdatetheconfigurationandrestartallofthemysqlnodesandtheproxynodes(toinformthemofthenewIPaddressesaswell).Restartingthenodeswillcauseallconnectionstothatnodetobedroppedwhilethenoderestarts.

Scalingthecluster

Scalingupfrom1toNnodes

WhenanewMariaDbnodecomesonline,itreplicatesdatafromtheexistingnodeinthecluster.Oncereplicationiscomplete,thenodewilljointhecluster.Theproxywillcontinuetorouteallincomingconnectionstotheprimarynodewhileitremainshealthy.

Iftheproxydetectsthatthisnodebecomesunhealthy ,itwillseverexistingconnections,androuteallnewconnectionstoadifferent,healthynode.IftherearenohealthyMariaDbnodes,theproxywillrejectallsubsequentconnections.

Whiletransitioningfromonenodetoacluster,therewillbeanundeterminedperiodofperformancedegradationwhilethenewnodesyncsalldatafromtheoriginalnode.

Note:IfyouareplanningtoscaleupMariaDbnodes,itisrecommendedtodosoindifferentAvailabilityZonestomaximizeclusteravailability.AnAvailabilityZoneisanetwork-distinctsectionofagivenRegion.FurtherdetailsareavailableinAmazon’sdocumentation .

ScalingdownfromNto1node

WhenscalingfrommultiplenodestoasingleMariaDbnode,theproxywilldeterminethatthesoleremainingnodeistheprimarynode(provideditremainshealthy).TheproxyroutesincomingconnectionstotheremainingMariaDbnode.

Rejoiningthecluster(existingnodes)Existingnodesrestartedwithmonitshouldautomaticallyjointhecluster.Ifanexistingnodefailstojointhecluster,itmaybebecauseitstransactionrecord’s( seqno )ishigherthanthatofthenodesintheclusterwithquorum(akatheprimarycomponent).

Ifthenodehasahigher seqno itwillbeapparentintheerrorlog /var/vcap/sys/log/mysql/mysql.err.log .

Ifthehealthynodesofaclusterhavealowertransactionrecordnumberthanthefailingnode,itmightbedesirabletoshutdownthehealthynodesandbootstrapfromthenodewiththemorerecenttransactionrecordnumber.Seethebootstrappingdocsformoredetails.

Manualrecoverymaybepossible,butiserror-proneandinvolvesdumpingtransactionsandapplyingthemtotherunningcluster(outofscopeforthisdoc).

Abandoningthedataisalsoanoption,ifyou’reokwithlosingtheunsynchronizedtransactions.Followthefollowingstepstoabandonthedata(asroot):

Stoptheprocesswith monit stop mariadb_ctrl .Deletethegalerastate( /var/vcap/store/mysql/grastate.dat )andcache( /var/vcap/store/mysql/galera.cache )filesfromthepersistentdisk.Restartingthenodewith monit start mariadb_ctrl .

StateSnapshotTransfer(SST)Whenanewnodeisaddedtotheclusterorrejoinsthecluster,itsynchronizesstatewiththeprimarycomponentviaaprocesscalledSST.Asinglenodefromtheprimarycomponentischosentoactasastatedonor.BydefaultGalerausesrsynctoperformSST,whichblocksforthedurationofthetransfer.However,MySQLforPivotalCloudFoundry(PCF)isconfiguredtouseXtrabackup ,whichallowsthedonornodetocontinuetoacceptreadsandwrites.

QuorumInorderfortheclustertocontinueacceptingrequests,aquorummustbereachedbypeer-to-peercommunication.Morethanhalfofthenodesmustberesponsivetoeachothertomaintainaquorum.

Ifmorethanhalfofthenodesareunresponsiveforaperiodoftimethenodeswillstoprespondingtoqueries,theclusterwillfail,andbootstrappingwillberequiredtore-enablefunctionality.

AvoidanevennumberofnodesItisgenerallyrecommendedtoavoidanevennumberofnodes.Thisisbecauseapartitioncouldcausetheentireclustertolosequorum,asneitherremainingcomponenthasmorethanhalfofthetotalnodes.

A2nodeclustercannottoleratethefailureofsinglenodefailureasthiswouldcauselossofquorum.Assuch,theminimumnumberofnodesrequiredtotoleratesinglenodefailureis3.

Unresponsivenode(s)Anodecanbecomeunresponsiveforanumberofreasons:

networklatencymysqlprocessfailurefirewallrulechangesvmfailure

Unresponsivenodeswillstoprespondingtoqueriesand,aftertimeout,leavethecluster.

Nodeswillbemarkedasunresponsive(inactive)either:

Iftheyfailtorespondtoonenodewithin15secondsORIftheyfailtorespondtoallothernodeswithin5seconds

Unresponsivenodesthatbecomeresponsiveagainwillrejointhecluster,aslongastheyareonthesameIPwhichispre-configuredinthegcommaddressonalltheotherrunningnodes,andaquorumwasheldbytheremainingnodes.

Allnodessuspendwritesoncetheynoticesomethingiswrongwiththecluster(writerequestshang).Afteratimeoutperiodof5seconds,requeststonon-quorumnodeswillfail.Mostclientsreturntheerror: WSREP has not yet prepared this node for application use .Someclientsmayinsteadreturn unknown error .Nodeswhohavereachedquorumwillcontinuefulfillingwriterequests.

Ifdeployedusingaproxy,acontinuallyinactivenodewillcausetheproxytofailover,selectingadifferentmysqlnodetoroutenewqueriesto.

Re-bootstrappingtheclusterafterquorumislostThestartscriptwillcurrentlybootstrapnode0onlyoninitialdeploy.Ifbootstrappingisnecessaryatalaterdate,itmustbedonemanually.Formoreinformationaboutmanuallybootstrappingacluster,seeBootstrappingGalera.

Ifthesinglenodeisbootstrapped,itwillcreateanewone-nodeclusterthatothernodescanjoin.

SimulatingnodefailureTosimulateatemporarysinglenodefailure,use kill -9 onthepidofthemysqlprocess.Thiswillonlytemporarilydisablethenodebecausetheprocessisbeingmonitoredbymonit,whichwillrestarttheprocessifitisnotrunning.

Tomorepermanentlydisabletheprocess,execute monit unmonitor mariadb_ctrl before kill -9 .

Tosimulatemulti-nodefailurewithoutkillinganodeprocess,communicationcanbeseveredbychangingtheiptablesconfigtodisallowcommunication:

iptables-F&&#optional-flushexistingrules\iptables-AINPUT-ptcp--destination-port4567-jDROP&&\iptables-AINPUT-ptcp--destination-port4568-jDROP&&\iptables-AINPUT-ptcp--destination-port4444-jDROP&&\iptables-AINPUT-ptcp--destination-port3306&&\iptables-AOUTPUT-ptcp--destination-port4567-jDROP&&\iptables-AOUTPUT-ptcp--destination-port4568-jDROP&&\iptables-AOUTPUT-ptcp--destination-port4444-jDROP&&\iptables-AOUTPUT-ptcp--destination-port3306

Torecoverfromthis,dropthepartitionbyflushingallrules: iptables-F

Cluster ConfigurationThispagedocumentsthevariousconfigurationdecisionsthathavebeenmadeinrelationtoMariaDBandGaleraincf-mysql-release.

SSTmethodGalerasupportsmultiplemethodsforStateSnapshotTransfer .The rsync methodisusuallyfastest.The xtrabackup methodhastheadvantageofkeepingthedonornodewriteableduringSST.Wehavechosentouse xtrabackup .

InnoDBLogFilesOurclusterdefaultsto1GBforlogfilesizetosupportlargerblob.

MaxUserConnectionsToensureallusersgetfairaccesstosystemresources,wehavecappedeachuser’snumberofconnectionsto40.

SkipExternalLockingSinceeachVirtualMachineonlyhasonemysqldprocessrunning,wedonotneedexternallocking.

MaxAllowedPacketWeallowblobsupto256MB.Thissizeisunlikelytolimitauser’squery,butisalsomanageableforourInnoDBlogfilesize.

InnodbFilePerTableInnodballowsusingeitherasinglefiletorepresentalldata,oraseparatefileforeachtable.Wechosetouseaseparatefileforeachtableasthisprovidesmoreflexibilityandoptimization.Forafulllistofprosandcons,seeMySQL’sdocumentationforInnoDBFile-Per-TableMode .

InnodbFileFormatTotakeadvantageofalltheextrafeaturesavailablewiththe innodb_file_per_table=

ONoption,weusethe Barracuda fileformat.

TemporaryTablesMySQLisconfiguredtoconverttemporaryin-memorytablestotemporaryon-disktableswhenaqueryEITHERgeneratesmorethan16millionrowsofoutputorusesmorethan32MBofdataspace.UserscanseeifaqueryisusingatemporarytablebyusingtheEXPLAINcommandandlookingfor“Usingtemporary,”intheoutput.Iftheserverprocessesverylargequeriesthatuse/tmpspacesimultaneously,itispossibleforqueriestoreceivenospacelefterrors.

Proxy for MySQL for Pivotal Cloud FoundryInMySQLforPivotalCloudFoundry(PCF),Switchboard isusedtoproxyTCPconnectionstohealthyMariaDBnodes.

AproxyisusedtogracefullyhandlefailureofMariaDBnodes.Useofaproxypermitsveryfast,unambiguousfailovertoothernodeswithintheclusterintheeventofanodefailure.

Whenanodebecomesunhealthy,theproxyre-routesallsubsequentconnectionstoahealthynode.Allexistingconnectionstotheunhealthynodeareclosed.

ProxyDashboardTheserviceprovidesadashboardwhereadministratorscanobservehealthandmetricsforeachinstanceintheproxytier.Metricsincludethenumberofclientconnectionsroutedtoeachbackenddatabaseclusternode.

Thedashboardforeachproxyinstancecanbefoundat: http://proxy-<jobindex>-p-mysql.<system-domain> .Thejobindexstartsat0.Forexample,ifyouhavetwoproxyinstancesdeployedandyoursystem-domainis example.com ,dashboardswouldbeaccessibleat:

http://proxy-0-p-mysql.example.com

http://proxy-1-p-mysql.example.com

Basicauthcredentialsarerequiredtoaccessthedashboard.ThesecanbefoundintheCredentialstaboftheMySQLproductinOperationsManager.

ConsistentRoutingAtanygiventime,Switchboardwillonlyroutetooneactivenode.Thatnodewillcontinuetobetheonlyactivenodeuntilitbecomesunhealthy.

IfmultipleSwitchboardproxiesareusedinparallel(ex:behindaload-balancer)thereisnoguaranteethattheproxieswillchoosethesameactivenode.Thiscanresultindeadlocks,whereinattemptstoupdatethesamerowbymultipleclientswillresultonecommitsucceedingandtheotherfails.Thisisaknownissue,withexplorationofmitigationoptionsontheroadmapforthisproduct.Toavoidthisproblem,useasingleproxyinstanceoranexternalfailoversystemtodirecttraffictooneproxyinstanceatatime.

NodeHealth

HealthyTheproxyqueriesanHTTPhealthcheckprocess,co-locatedonthedatabasenode,whendeterminingwheretoroutetraffic.

IfthehealthcheckprocessreturnsHTTPstatuscodeof200,thenodeisaddedtothepoolofhealthynodes.

Aresurrectednodewillnotimmediatelyreceiveconnections.Theproxywillcontinuetorouteallconnections,neworexisting,tothecurrentlyactivenode.Inthecaseoffailover,allhealthynodeswillbeconsideredascandidatesfornewconnections.

UnhealthyIfthehealthcheckreturnsHTTPstatuscode503,thenodeisconsideredunhealthy.

Thishappenswhenanodebecomesnon-primary,asspecifiedbythecluster-behaviordocs.

Theproxywillseverallexistingconnectionstonewlyunhealthynodes.Clientsareexpectedtohandlereconnectingonconnectionfailure.Theproxywillroutenewconnectionstoahealthynode,assumingsuchanodeexists.

UnresponsiveIfnodehealthcannotbedeterminedduetoanunreachableorunresponsivehealthcheckendpoint,theproxywillconsiderthenodeunhealthy.ThismayhappenifthereisanetworkpartitionoriftheVMcontainingthehealthcheckandMariaDBnodedied.

ProxycountIftheoperatorsetsthetotalnumberofproxiesto0hostsinOpsManagerorBOSHdeploymentmanifest,thenapplicationswillconnectdirectlytoonehealthyMariaDBnodemakingthatnodeasinglepointoffailureforthecluster.

Therecommendednumberofproxiesare2;thisprovidesredundancyshouldoneoftheproxiesfail.

RemovingtheproxyasaSPOFTheproxytierisresponsibleforroutingconnectionsfromapplicationstohealthyMariaDBclusternodes,evenintheeventofnodefailure.

BoundapplicationsareprovidedwithahostnameorIPaddresstoreachadatabasemanagedbytheservice.Bydefault,theMySQLservicewillprovideboundapplicationswiththeIPofthefirstinstanceintheproxytier.Evenifadditionalproxyinstancesaredeployed,clientconnectionswillnotberoutedthroughthem.Thismeansthefirstproxyinstanceisasinglepointoffailure.

In order to eliminate the first proxy instance as a single point of failure, operators must configure a load balancer to route clientconnections to all proxy IPs, and configure the MySQL service to give bound applications a hostname or IP address that resolves tothe load balancer.

ConfiguringloadbalancerConfiguretheloadbalancertoroutetrafficforTCPport3306totheIPsofallproxyinstancesonTCPport3306.Next,configuretheloadbalancer’shealthchecktousetheproxyhealthport.ThisisTCPport1936bydefaulttomaintainbackwardscompatibilitywithpreviousreleases.Thisportisnotconfigurable.

ConfiguringMySQLforPCFtogiveapplicationstheaddressoftheloadbalancerToensurethatboundapplicationswillusetheloadbalancertoreachbounddatabases,navigatetotheMySQLforPCFtileinOperationsManager,thentheResourceConfigconfigurationscreenwithinit.On AWS only,enteryourloadbalancer’shostnameinthe“ELBNames”columnfortheProxyrow.

AWSRoute53TosetupaRoundRobinDNSacrossmultipleproxyIPsusingAWSRoute53,followthefollowinginstructions:

1. LogintoAWS.

2. ClickRoute53.

3. ClickHostedZones.

4. Selectthehostedzonethatcontainsthedomainnametoapplyroundrobinroutingto.

5. Click‘GotoRecordSets’.

6. Selecttherecordsetcontainingthedesireddomainname.

7. Inthevalueinput,entertheIPaddressesofeachproxyVM,separatedbyanewline.

Finally,updatethemanifestproperty properties.mysql_node.host forthecf-mysql-brokerjob,asdescribedabove.

APITheproxyhostsaJSONAPIat proxy-<boshjobindex>-p-mysql.<systemdomain>/v0/ .

TheAPIprovidesthefollowingroute:

Request:

Method:GET

Path: /v0/backends

Params:~

Headers:BasicAuth

Response:

[{"name":"mysql-0","ip":"1.2.3.4","healthy":true,"active":true,"currentSessionCount":2},{"name":"mysql-1","ip":"5.6.7.8","healthy":false,"active":false,"currentSessionCount":0},{"name":"mysql-2","ip":"9.9.9.9","healthy":true,"active":false,"currentSessionCount":0}]

FormoreinformationaboutSwitchBoard,readtheproxydocumentation

Creating Application Security Groups for MySQLThistopicdescribeshowtocreateApplicationSecurityGroups (ASGs)forMySQLforPivotalCloudFoundry(PCF).

ToallowsmoketeststorunwhenyouinstalltheMySQLforPCFserviceandallowappstoaccessMySQLforPCFafteritisinstalled,youmustcreateanappropriateASGandbindittotheservice.

Inaddition,applicationcontainersthataccessinstancesofthisservicerequireanoutboundnetworkconnectiontotheloadbalancerconfiguredfortheMySQLforPCFservice.

TocreateASGsfortheMySQLforPCFservice,performthefollowingsteps:

1. CreateaJSONfilewiththefollowingcontentscalled p-mysql-security-group.json :

[{"ports":"3306","protocol":"tcp","destination":"REPLACEWITHTHEP-MYSQLLOADBALANCERIP,RANGEORCIDR"}]

Inthe destination field,addtheIPaddress,range,orCIDRoftheloadbalancerthatyouconfiguredfortheMySQLforPCFservice.

2. LogintoyourPCFdeploymentasanadministrator,andcreateanASGcalled p-mysql-service .

#afterlogginginasanadministrator$cfcreate-security-groupp-mysql-servicep-mysql-security-group.json

3. BindthenewASGtothe default-running ASGsettoallowallapplicationstoaccesstheservice.

$cfbind-running-security-groupp-mysql-service

Iftheserviceshouldonlybemadeavailabletospecificspaces,bindtheASGdirectlytothosespaces.

$cfbind-security-groupp-mysql-serviceORGANIZATION_NAMESPACE_NAME

Note:WithoutanASG,theserviceisnotinstallableorusable.

Monitoring the MySQL ServiceThisdocumentdescribeshowtousetheReplicationCanaryandInterruptortomonitoryourMySQLcluster.

ReplicationCanaryMySQLforPivotalCloudFoundry(PCF)isaclusteredsolutionthatusesreplicationtoprovidebenefitssuchasquickfailoverandrollingupgrades.Thisismorecomplexthanasinglenodesystemwithnoreplication.MySQLforPCFincludesaReplicationCanarytohelpwiththeincreasedcomplexity.TheReplicationCanaryisalong-runningmonitorthatvalidatesthatreplicationisworkingwithintheMySQLcluster.

HowitWorksTheReplicationCanarywritestoaprivatedatasetinthecluster,andattemptstoreadthatdatafromeachnode.Itpausesbetweenwritingandreadingtoensurethatthewritesetshavebeencommittedacrosseachnodeofthecluster.Theprivatedatasetdoesnotuseasignificantamountofdiskcapacity.

Whenreplicationfailstoworkproperly,theCanarydetectsthatitcannotreadthedatafromallnodes,andimmediatelytakestwoactions:

E-mailsapre-configuredaddresswithamessagethatreplicationhasfailed.Seethesamplebelow.

Disablesclientaccesstothecluster.

SampleNotificationE-mailIftheCanarydetectsareplicationfailure,itimmediatelysendsane-mailthroughtheElasticRuntimenotificationservice.Seethefollowingexample:

Subject:CFNotification:p-mysqlReplicationCanary,alert417

Thismessagewassentdirectlytoyouremailaddress.

{alert-code417}Thisisane-mailtonotifyyouthattheMySQLservice'sreplicationcanaryhasdetectedanunsafeclusterconditioninwhichreplicationisnotperformingasexpectedacrossallnodes.

ClusterAccessEachtimetheCanarydetectsclusterreplicationfailure,itinstructsallproxiestodisableconnectionstothedatabasecluster.Ifthereplicationissueresolves,theCanarydetectsthisandautomaticallyrestoresclientaccesstothecluster.

IfyoumustrestoreaccesstotheclusterregardlessoftheReplicationCanary,contactSupport.

DetermineProxyState

YoucandetermineiftheCanarydisabledclusteraccessbyusingtheProxyAPI.Seethefollowingexample:

ubuntu@ip-10-0-0-38:~$curl-kuadmin:PASSWORD_FROM_OPSMGR-XGEThttps://proxy-0-p-mysql.SYSTEM-DOMAIN/v0/cluster;echo{"currentBackendIndex":0,"trafficEnabled":false,"message":"Disablingclustertraffic","lastUpdated":"2016-07-27T05:16:29.197754077Z"}

EnabletheReplicationCanaryToenabletheReplicationCanary,followtheinstructionsbelowtoconfigureboththeElasticRuntimetileandtheMySQLforPCFtile.

Note:Malfunctioningreplicationexposestheclustertothepossibilityofdataloss.Becauseofthis,bothbehaviorsareenabledbydefault.ItiscriticalthatyoucontactPivotalsupportimmediatelyinthecaseofreplicationfailure.Supportwillworkwithyoutodeterminethenatureoftheclusterfailureandprovideguidanceregardingasolution.

ConfiguretheElasticRuntimeTile

1. IntheSMTP Configsection,enteraFrom EmailthattheReplicationCanarycanusetosendnotifications,alongwiththeSMTPserverconfiguration.

2. IntheErrandssection,selecttheNotificationserrand.

ConfiguretheMySQLforPCFTile1. IntheAdvanced Optionssection,selectEnable replication canary.

2. IfyouwanttotheReplicationCanarytosende-mailbutnotdisableaccessattheproxy,selectNotify only.

3. YoucanoverridetheReplication canary time period.TheReplication canary time periodsetshowfrequentlythecanarychecksforreplicationfailure,inseconds.Thisaddsasmallamountofloadtothedatabases,butthecanaryreactsmorequicklytoreplicationfailure.Thedefaultis30seconds.

4. YoucanoverridetheReplication canary read delay.TheReplication canary read delaysetshowlongthecanarywaitstoverifydataisreplicatingacrosseachMySQLnode,inseconds.Clustersunderheavyloadexperiencesomesmallreplicationlagaswritesetsarecommittedacrossthenodes.TheDefaultis20seconds.

5. EnteranE-mail addresstoreceivemonitoringnotifications.Useacloselymonitorede-mailaddressaccount.ThepurposeoftheCanaryistoescalatereplicationfailureasquicklyaspossible.

6. IntheResource Configsection,ensuretheMonitoringjobhasoneinstance.

Note:InatypicalPCFdeployment,thesesettingsarealreadyconfigured.

Note:Pivotalrecommendsleavingthischeckboxunselectedduetothepossibilityofdatalossfromreplicationfailure.

DisabletheReplicationCanaryIfyoudonotneedtheReplicationCanary,forinstanceifyouuseasingleMySQLnode,followthisproceduretodisableboththejobandtheresourceconfiguration.

1. IntheAdvanced OptionssectionoftheMySQLforPCFtile,selectDisable Replication Canary.

2. IntheResource Configpane,settheMonitoringjobtozeroinstances.

InterruptorTherearerarecasesinwhichaMySQLnodesilentlyfallsoutofsyncwiththeothernodesofthecluster.TheReplicationCanarycloselymonitorstheclusterforthiscondition.However,iftheReplicationCanarydoesnotdetectthefailure,theInterruptorprovidesasolutionforpreventingdataloss.

HowitWorksIfthenodereceivingtrafficfromtheproxyfallsoutofsyncwiththecluster,itgeneratesadatasetthattheothernodesdonothave.Ifthesamenodelaterreceivesatransactionthatisnotcompatiblewiththedatasetsoftheothernodes,itdiscardsitslocaldatasetandadoptsthedatasetsoftheothernodes.Thisisgenerallydesiredbehavior,unlessdatareplicationisnotfunctioningacrossthecluster.Thenodecoulddestroyvaliddatabydiscardingitslocaldataset.Whenenabled,theInterruptorpreventsthenodefromdestroyingitslocaldatasetifthereisariskoflosingvaliddata.

Anout-of-syncnodeemploysoneoftwotwomodes tocatchupwiththecluster:

Incremental State Transfer (IST):Ifanodehasbeenoutoftheclusterforarelativelyshortperiodoftime,suchasareboot,thenodeinvokesIST.Thisisnotadangerousoperation,andtheInterruptordoesnotinterfere.

State Snapshot Transfer (SST):Ifanodehasbeenunavailableforanextendedamountoftime,suchasahardwarefailurethatrequiresphysicalrepair,thenodemayinvokeSST.Incasesoffailedreplication,SSTcancausedataloss.Whenenabled,theInterruptorpreventsthismethodofrecovery.

SampleNotificationE-mailTheInterruptorsendsanemailthroughtheElasticRuntimenotificationservicewhenitpreventsanodefromrejoiningacluster.Seethefollowingexample:

Subject:CFNotification:p-mysqlalert100

Thismessagewassentdirectlytoyouremailaddress.

{alert-code100}Hello,justwantedtoletyouknowthattheMySQLnode/clusterhasgonedownandhasbeendisallowedfromre-joiningbytheinterruptor.

Note:IfyoureceiveanotificationthattheInterruptorhasactivated,itiscriticalthatyoucontactPivotalsupportimmediately.Supportwillworkwithyoutodeterminethenatureofthefailure,andprovideguidanceregardingasolution.

InterruptorLogsYoucanconfirmthattheInterruptorhasactivatedbyexamining /var/vcap/sys/log/mysql/mysql.err.log onthefailingnode.Thelogcontainsthefollowingmessage:

WSREP_SST:[ERROR]#####################################################################################(2016061004:33:21.338)WSREP_SST:[ERROR]SSTdisabledduetodangerofdataloss.Verifydataandruntherejoin-unsafeerrand(2016061004:33:21.340)WSREP_SST:[ERROR]#####################################################################################(2016061004:33:21.341)

ForceaNodetoRejointheClusterIngeneral,iftheInterruptorhasactivatedbuttheReplicationCanaryhasnottriggered,itissafeforthenodetorejointhecluster.

1. Followtheseinstructionstochoosethep-mysqlmanifestwiththeBOSHCLI.

2. Run boshrunerrandrejoin-unsafe toforceanodetorejointhecluster:

$boshrunerrandrejoin-unsafe[...][stdout]Startedrejoin-unsafeerrand...Successfullyrepairedclusterrejoin-unsafeerrandcompleted

[stderr]None

Errand`rejoin-unsafe'completedsuccessfully(exitcode0)

DisabletheInterruptorTheInterruptorisenabledbydefault.TodisabletheInterruptor:

IntheAdvanced Optionssection,underEnable optional protections,un-checkPrevent node auto re-join.

Note:ThistopicrequiresyoutoruncommandsfromtheOpsManagerDirector usingtheBOSHCLI.RefertotheAdvancedTroubleshootingwiththeBOSHCLI topicformoreinformation.

Determining Cluster StateConnecttoeachMySQLnodeusingamysqlclientandcheckitsstatus.

Ifallnodesareinthe Primary component,youhaveahealthycluster.Ifsomenodesareina Non-primary component,thosenodesarenotabletojointhecluster.

Seehowmanynodesareinthecluster.

$mysql-hNODE_IP-uroot-pPASSWORD-e'SHOWSTATUSLIKE"wsrep_cluster_size";'+--------------------+-------+|Variable_name|Value|+--------------------+-------+|wsrep_cluster_size|3|+--------------------+-------+

Ifthevalueof wsrep_cluster_size isequaltotheexpectednumberofnodes,thenallnodeshavejoinedthecluster.Otherwise,checknetworkconnectivitybetweennodesanduse monitstatus toidentifyanyissuespreventingnodesfromstarting.

Formoreinformation,seetheofficialGaleradocumentationforCheckingClusterIntegrity .

Bootstrapping a Galera ClusterPage last updated:

ThistopicdescribestheprocedureforrecoveringaterminatedElasticRuntimeclusterusingaprocessknownasbootstrapping.

WhentoBootstrapYoumustbootstrapaclusterthatlosesquorum.Aclusterlosesquorumwhenlessthanhalfofthenodescancommunicatewitheachotherforlongerthantheconfiguredgraceperiod.Ifaclusterdoesnotlosequorum,individualunhealthynodesautomaticallyrejointheclusterafterresolvingtheerror,restartingthenode,orrestoringconnectivity.

Youcandetectlostquorumthroughthefollowingsymptoms:

Allnodesappear“Unhealthy”ontheproxydashboard,viewableat proxy-BOSH-JOB-INDEX.p-mysql.YOUR-SYSTEM-DOMAIN :

Allresponsivenodesreportthevalueof wsrep_cluster_status as non-Primary :

Allresponsivenodesrespondwith ERROR1047 whenqueriedwithmoststatementtypes:

mysql>select*frommysql.user;ERROR1047(08S01)atline1:WSREPhasnotyetpreparednodeforapplicationuse

SeetheClusterScaling,NodeFailure,andQuorumtopicformoredetailsaboutdeterminingclusterstate.

Followthestepsbelowtorecoveraclusterthathaslostquorum.

Step1:ChoosetheCorrectManifest

1. LogintotheBOSHdirectorbyrunning boshtargetDIRECTOR-URL followedby boshloginUSERNAMEPASSWORD .

2. Run boshdeployments .

$boshdeploymentsActingasuser'director'on'p-bosh-30c19bdd43c55c627d70'

+-------------------------+-------------------------------+----------------------------------------------+--------------+|Name|Release(s)|Stemcell(s)|CloudConfig|+-------------------------+-------------------------------+----------------------------------------------+--------------+|cf-e82cbf44613594d8a155|cf-autoscaling/28|bosh-aws-xen-hvm-ubuntu-trusty-go_agent/3140|none|||cf-mysql/23|||||cf/225|||||diego/0.1441.0|||||etcd/18|||||garden-linux/0.327.0|||||notifications-ui/10|||||notifications/19|||||push-apps-manager-release/397|||+-------------------------+-------------------------------+----------------------------------------------+--------------+|p-mysql|p-mysql|||+-----------------------------------------------------------------------------------------------------------------------+

3. Downloadthemanifest.

Note:ThistopicrequiresyoutoruncommandsfromtheOpsManagerDirector usingtheBOSHCLI.RefertotheAdvancedTroubleshootingwiththeBOSHCLI topicformoreinformation.

$boshdownloadmanifestp-mysql/tmp/p-mysql.ymlActingasuser'director'ondeployment[...]Deploymentmanifestsavedto`/tmp/p-mysql.yml'

4. SetBOSHtousethedeploymentmanifestyoudownloaded.

$boshdeployment/tmp/p-mysql.yml

Step2:BootstrapTodeterminewhichsetofinstructionstofollow,youmustdeterminethestateofyourVirtualMachines(VMs).

Iftheoutputof bosh instances showsthestateofthejobsas failing ,thenfollowthestepsinScenario1below.

Iftheoutputof bosh instances showsthestateofthejobsas unknown/unknown ,thenfollowthestepsinScenario2.

Scenario1:VirtualMachinesRunning,ClusterDisruptedInthisscenario,nodesareupandrunning,buttheclusterhasbeendisrupted.

1. SSHtoeachnodeintheclusterand,asroot,shutdownthe mariadb process.ToSSHintoBOSH-deployedVMs,seetheAdvancedTroubleshootingwiththeBOSHCLItopic.

$monitstopmariadb_ctrl

Re-bootstrappingtheclusterwillnotbesuccessfulunlessallothernodeshavebeenshutdown.

2. Chooseanodetobootstrapbylocatingthenodewiththehighesttransactionsequencenumber( seqno ).Youcanobtainthe seqno ofastoppednodeinoneoftwoways:

Ifanodeshutdowngracefully,the seqno isintheGalerastatefileofthenode.

$cat/var/vcap/store/mysql/grastate.dat|grep'seqno:'

Ifthenodecrashedorwaskilled,the seqno intheGalerastatefileofthenodeis -1 .Inthiscase,the seqno mayberecoverablefromthedatabase.

1. Runthefollowingcommandtostartupthedatabase,logtherecoveredsequencenumber,andexit.

$/var/vcap/packages/mariadb/bin/mysqld--wsrep-recover

2. Scantheerrorlogfortherecoveredsequencenumber.Thelastnumberafterthegroupid( uuid )istherecovered seqno :

$grep"Recoveredposition"/var/vcap/sys/log/mysql/mysql.err.log|tail-115022518:09:42mysqld_safeWSREP:Recoveredpositione93955c7-b797-11e4-9faa-9a6f0b73eb46:15

Ifthenodeneverconnectedtotheclusterbeforecrashing,itmaynothaveagroupid( uuid in grastate.dat ).Inthiscase,youcannotrecoverthe seqno .Unlessallnodescrashedthisway,donotchoosethisnodeforbootstrapping.

3. Choosethenodewiththehighest seqno valueasthebootstrapnode.Ifallnodeshavethesame seqno ,youcanchooseanynodeasthebootstrapnode.

4. Onthebootstrapnode,updatethestatefileandrestartthe mariadb process.

$echo-n"NEEDS_BOOTSTRAP">/var/vcap/store/mysql/state.txt$monitstartmariadb_ctrl

Note:Thefollowingstepsarepronetousererrorandcanresultinlostdataiffollowedincorrectly.

Note:Onlyperformthesebootstrapcommandsonthenodewiththehighest seqno .Otherwise,thenodewiththehighest seqno willbeunabletojointhenewclusterunlessitsdataisabandoned.Its mariadb processwillexitwithanerror.SeetheClusterScaling,NodeFailure,andQuorumtopicformoredetailsonintentionallyabandoningdata.

5. Checkthatthe mariadb processhasstartedsuccessfully.

$watchmonitsummary

Itcantakeuptotenminutesfor monit tostartthe mariadb process.

6. Oncethebootstrappednodeisrunning,startthe mariadb processontheremainingnodes:

a. IftheInterruptorisenabled,runthefollowingcommandbeforestarting mariadb :

$touch/var/vcap/sys/run/galera-healthcheck/enable_sst

b. Startthemariadbprocessusing monit :

$monitstartmariadb_ctrl

7. Verifythatthenewnodeshavesuccessfullyjoinedthecluster.Thefollowingcommanddisplaysthetotalnumberofnodesinthecluster:

mysql>SHOWSTATUSLIKE'wsrep_cluster_size';

Scenario2:VirtualMachinesTerminatedorLostInthisscenario,severecircumstancessuchaspowerfailurehaveterminatedallofyourVMs.YouneedtorecreatetheVMsbeforeyoucanrecoverthecluster.

1. IfyouenabledtheVMResurrector inOpsManager,thesystemdetectstheterminatedVMsandautomaticallyattemptstorecreatethem.Runboshtasksrecent--no-filter toseethe scanandfix jobrunbytheVMResurrector.

$boshtasksrecent--no-filter+-----+------------+-------------------------+----------+--------------------------------------------+---------------------------------------------------+|#|State|Timestamp|User|Description|Result|+-----+------------+-------------------------+----------+--------------------------------------------+---------------------------------------------------+|123|queued|2016-01-0800:18:07UTC|director|scanandfix||

IfyouhavenotenabledtheVMResurrector,runtheBOSHcloudcheckcommand boshcck todeleteanyplaceholderVMs.Whenprompted,chooseDeleteVMreference byentering 3 .

$boshcck

Actingasuser'director'ondeployment'cf-e82cbf44613594d8a155'on'p-bosh-30c19bdd43c55c627d70'Performingcloudcheck...

Directortask34Startedscanning22vmsStartedscanning22vms>CheckingVMstates.Done(00:00:10)Startedscanning22vms>19OK,0unresponsive,3missing,0unbound,0outofsync.Done(00:00:00)Donescanning22vms(00:00:10)

Startedscanning10persistentdisksStartedscanning10persistentdisks>Lookingforinactivedisks.Done(00:00:02)Startedscanning10persistentdisks>10OK,0missing,0inactive,0mount-infomismatch.Done(00:00:00)Donescanning10persistentdisks(00:00:02)

Task34done

Started2015-11-2601:42:42UTCFinished2015-11-2601:42:54UTCDuration00:00:12

Scaniscomplete,checkingifanyproblemsfound.

Found3problems

Problem1of3:VMwithcloudID`i-afe2801f'missing.1.Skipfornow2.RecreateVM3.DeleteVMreferencePleasechoosearesolution[1-3]:3

Problem2of3:VMwithcloudID`i-36741a86'missing.1.Skipfornow2.RecreateVM3.DeleteVMreferencePleasechoosearesolution[1-3]:3

Problem3of3:VMwithcloudID`i-ce751b7e'missing.1.Skipfornow2.RecreateVM3.DeleteVMreferencePleasechoosearesolution[1-3]:3

Belowisthelistofresolutionsyou'veprovidedPleasemakesureeverythingisfineandconfirmyourchanges

1.VMwithcloudID`i-afe2801'missing.DeleteVMreference

2.VMwithcloudID`i-36741a86'missing.DeleteVMreference

3.VMwithcloudID`i-ce751b7e'missing.DeleteVMreference

Applyresolutions?(type'yes'tocontinue):yesApplyingresolutions...

Directortask35StartedapplyingproblemresolutionsStartedapplyingproblemresolutions>missing_vm11:DeleteVMreference.Done(00:00:00)Startedapplyingproblemresolutions>missing_vm27:DeleteVMreference.Done(00:00:00)Startedapplyingproblemresolutions>missing_vm26:DeleteVMreference.Done(00:00:00)Doneapplyingproblemresolutions(00:00:00)

Task35done

Started2015-11-2601:47:08UTCFinished2015-11-2601:47:08UTCDuration00:00:00Cloudcheckisfinished

2. Run boshinstances andexaminetheoutput.TheVMstransitionfrom unresponsiveagent to starting .Ultimately,twoappearas failing .DonotproceedtothenextstepuntilallthreeVMsareinthe starting or failing state.

$boshinstances[...]+--------------------------------------------------+----------+------------------------------------------------+------------+|mysql-partition-e97dae91e44681e0b543/0|starting|mysql-partition-e97dae91e44681e0b543|192.0.2.60||mysql-partition-e97dae91e44681e0b543/1|failing|mysql-partition-e97dae91e44681e0b543|192.0.2.61||mysql-partition-e97dae91e44681e0b543/2|failing|mysql-partition-e97dae91e44681e0b543|192.0.2.62|+--------------------------------------------------+----------+------------------------------------------------+------------+

3. CompletethestepsforScenario1.

4. Run boshinstances andexaminetheoutputtoconfirmthatthebootstrapwassuccessful.Someinstancesmaystillappearas failing .

Backing Up MySQL for Pivotal Cloud FoundryThistopicdescribeshowtoenable,configure,andusebackupsinMySQLforPivotalCloudFoundry(PCF).

OverviewAutomatedbackupshavethefollowingfeatures:

Periodicallycreateanduploadbackupartifactssuitableforrestoringthecompletesetofdatabaseinstancesallocatedintheservice

Nolocks,nodowntime

TheonlyeffectontheservingsystemsistheamountofI/OrequiredtocopythedatabaseandlogfilesoffoftheVM

Includesametadatafilethatcontainsthecriticaldetailsofthebackupartifact,includingtheeffectivecalendartimeofthebackup

BackupartifactsareencryptedwithintheMySQLforPCFclusterofVMs;unencrypteddataisnevertransportedoutsideoftheMySQLforPCFdeployment

EnableAutomatedBackupsYoucanconfigureMySQLforPCFtoautomaticallybackupitsdatabasestoexternalstorage.

How and Where:Therearetwooptionsforhowautomatedbackupstransferbackupdataandwherethedatasavesoutto:

MySQLforPCFrunsan scp commandthatsecure-copiesbackupfilestoaVMorphysicalmachineoperatingoutsideofPCF.TheoperatorprovisionsthebackupmachineseparatelyfromtheirPCFinstallation.Thisisthemostefficientoption.MySQLforPCFrunsanS3 clientthatsavesbackupstoanAmazonS3bucket,Ceph storagecluster,orotherS3-compatibleendpointcertifiedbyPivotal.

When:Backupsfollowaschedulethatyouspecifywithacron expression.

What:Youcanbackupjusttheprimarynode,orallnodesinthecluster.

Toenableautomatedbackupsandconfigurethemforoptionsabove,performthefollowingsteps:

1. NavigatetotheMySQLforPivotalCloudFoundrytileontheOpsManagerInstallationDashboard.

2. ClickBackups.

3. UnderBackups,clickEnable Backups.

4. ForCron Schedule,enteracronscheduleforthebackups.Thesyntaxissimilartotraditionalcron,withadditionalfeaturessuchas @every 1d ,whichspecifiesdailybackups.SeethecronGolibrarydocumentation formoreinformation.

5. Ifyouwanttobackupallnodes,selecttheBack up all nodescheckbox.

6. ToenablebackupsusingCeph orAWS,continuetotheCephorAWSsection.ToenablebackupsusingSCP,continuetotheSCPsection.

CephorAWSTobackupyourdatabaseonCephorAmazonWebServices(AWS)S3,performthefollowingsteps:

1. SelectCeph or Amazon S3.

2. EnteryourS3 Endpoint URL.Forinstance, https://s3.amazonaws.com .

3. EnteryourS3 Bucket Name.Donotincludean s3:// prefix ,atrailing / ,orunderscores.Ifthebucketdoesnotalreadyexist,itwillbecreatedautomatically.

4. ForBucket Path,specifyafolderwithinthebuckettoholdyourMySQLbackups.Donotincludeatrailing / .Ifthefolderdoesnotalreadyexist,itwillbecreatedautomatically.

5. ForAWS Access Key IDandAWS Secret Access Key,enteryourCephorAWScredentials.ForAWS,PivotalrecommendscreatinganIAM

credentialthatonlyhasaccesstothisbucket.

6. ClickSave.

SCPTobackupyourdatabaseusingSCP,performthefollowingsteps:

Note:Youmustusethisfolderexclusivelyforthiscluster’sbackupartifacts.Mixingthebackupartifactsfromdifferentclusterswithinasinglefoldercancauseconfusionandpossibleinadvertentlossofbackupartifacts.

1. SelectSCP to a Remote Host.

2. EntertheUsername,Hostname,andDestination Directoryforthebackups.

3. ForPrivate Key,pasteintheprivatekeythatwillbeusedtoencrypttheSCPtransfer.

4. EntertheSCP Port.SCPrunsonport22bydefault.

5. ClickSave.

DisableAutomatedBackupsTodisableautomatedbackups,performthefollowingsteps:

Note:PivotalrecommendsusingaVMnotwithinthePCFdeploymentforthedestinationofSCPbackups.SCPenablestheoperatortouseanydesiredstoragesolutiononthedestinationVM.

2. ClickBackups.

3. UnderBackups,clickDisable Backups.

4. UnderBackup Destination,clickNo Backups.

5. ClickSave.

6. Intheleftnavigation,clickResource Config.

7. ChangethenumberofinstancesforBackup Prepare Nodefrom 1 to 0 .

8. ClickSave.

9. ReturntotheOpsManagerInstallationDashboardandclickApply Changes.

ToconfigureautomatedbackupsforMySQLforPCF,performthefollowingsteps:

2. ClickBackups.

UnderstandBackupsThesectionsbelowdescribetheprocessthatMySQLforPCFcomponentjobsfollowwhenperformingautomatedbackups,andtheformatforthemetadatafilethatrecordsinformationabouteachbackup.

BackupProcessOperatorsuseOpsManagertoconfigurethescheduleforautomatedbackupsandthelocationandcredentialsneededtostorebackupartifacts.

ThediagrambelowshowstheprocessthroughwhichMySQLforPCFjobsinitiateandrunautomatedbackups.

sequenceDiagramparticipantBlobstoreparticipantServiceBackupjobNoteoverServiceBackupjob:Triggeredbytimer,followingscheduleconfiguredinOpsManagerServiceBackupjob->>StreamingBackupclient:RequestbackupStreamingBackupclient->>StreamingBackuptool:RequestbackupStreamingBackuptool->>MySQLserver:RequestbackupNoteoverMySQLserver:FlushtableswithreadlockMySQLserver->>StreamingBackuptool:DataStreamingBackuptool->>StreamingBackupclient:DataStreamingBackupclient->>ServiceBackupjob:DataNoteoverServiceBackupjob:CompressandencryptServiceBackupjob->>Blobstore:BackupartifactNoteoverBlobstore:Storebackupartifact,usingcredsconfiguredinOpsManagerBlobstore-->>ServiceBackupjob:ConfirmartifactstoredNoteoverServiceBackupjob:Cleanuplocalstorage

TwoMySQLforPCFcomponentVMshostthejobslistedaboveasfollows:

Job Job name in the code Host VM

ServiceBackup service-backup BackupPrepareVMStreamingBackupclient streaming-backup-client

StreamingBackuptool streaming-backup-tool

MySQLVMMySQLserver mysql

BackupMetadataAlongwitheachbackupartifact,MySQLforPCFuploadsa mysql-backup-XXXXXXXXXX.txt metadatafile.

Thecontentsofthemetadatafileresemblethefollowing:

compact=Nencrypted=Ntool_version=2.4.5server_version=10.1.20-MariaDBend_time=2017-05-0523:26:19binlog_pos=filename'mysql-bin.000016',position'7000000',GTIDofthelastchange'0-1-30000'incremental=Nformat=tarcompressed=Nuuid=30000000-3000-1000-9000-40000000000fname=lock_time=0innodb_from_lsn=0innodb_to_lsn=6286393partial=Ntool_command=--user=admin--password=...--stream=tartmp/ibbackup_version=2.4.5tool_name=innobackupexstart_time=2017-05-0523:26:17

Withinthisfile,themostimportantitemsarethe start_time andthe server_version entries.Transactionsthathavenotbeencompletedatthestartofthebackupeffortarenotpresentintherestoredartifact.

RestoreaBackupArtifactMySQLforPCFkeepsatleasttwocompletecopiesofthedata.Inmostcases,ifaclusterisstillabletoconnecttopersistentstorage,youcanrestoreaclustertohealthusingthebootstrapprocess.Beforeresortingtoadatabaserestore,contactPivotalSupport toensureyourexistingclusterisbeyondhelp.

ThedisasterrecoverybackupsfeatureofMySQLforPCFisprimarilyintendedasawaytorecoverdatatothesamePCFdeploymentfromwhichthedatawasbackedup.Thisprocessreplaces100%ofthedataandstateofarunningMySQLforPCFcluster.Thisisespeciallyrelevantwithregardtoserviceinstancesandbindings.

Intheeventofatotalclusterloss,theprocesstorestoreabackupartifacttoaMySQLforPCFclusterisentirelymanual.Performthefollowingstepstousetheoffsitebackupstorestoreyourclustertoitspreviousstate:

1. DiscovertheencryptionkeysintheCredentialstaboftheMySQLforPCFtile.

2. Ifnecessary,installthesameversionoftheMySQL for PCFproductintheOpsManagerInstallationDashboard.

3. PerformthefollowingstepstoreducethesizeoftheMySQLforPCFclustertoasinglenode:

a. FromtheOpsManagerInstallationDashboard,clicktheMySQL for PCFtile.b. ClickResource Config.c. SetthenumberofinstancesforMySQL Serverto1.d. ClickSave.e. ReturntotheOpsManagerInstallationDashboardandclickApply Changes.

Note:Both compressed and encrypted showas N inthisfile,yettheartifactuploadedbyMySQLforPCFisbothcompressedandencrypted.Thisisaknownbug.

Note:Becauseofhowservicesinstancesaredefined,youcannotrestoreaMySQLforPCFdatabasetoadifferentPCFdeployment.

4. Afterthedeploymentfinishes,performthefollowingstepstopreparethefirstnodeforrestoration:

a. SSHintotheOpsManagerDirector.Formoreinformation,seetheSSHintoOpsManager sectioninthetopic.

b. RetrievetheIPaddressfortheMySQLserverbynavigatingtotheMySQL for PCFtileandclickingtheStatustab.c. RetrievetheVMcredentialsfortheMySQLserverbynavigatingtotheMySQL for PCFtileandclickingtheCredentialstab.d. FromtheOpsManagerDirectorVM,usetheBOSHCLItoSSHintothefirstMySQLjob.Formoreinformation,seetheBOSHSSH sectioninthe

topic.e. OntheMySQLserverVM,becomesuperuser:

$sudosu

f. Pausethelocaldatabaseserver:

$monitstopall

g. Confirmthatalljobsarelistedas not monitored :

$watchmonitsummary

h. DeletetheexistingMySQLdatathatisstoredondisk:

$rm-rf/var/vcap/store/mysql/*

5. Performthefollowingstepstorestorethebackup:

a. Movethecompressedbackupfiletothenodeusing scp .b. Decryptandexpandthefileusing gpg ,sendingtheoutputtotar:

$gpg--decryptmysql-backup.tar.gpg|tar-C/var/vcap/store/mysql-xvf-

c. Changetheownerofthedatadirectory,becauseMySQLexpectsthedatadirectorytobeownedbyaparticularuser:

$chown-Rvcap:vcap/var/vcap/store/mysql

d. Startallserviceswith monit :

$monitstartall

e. Watchthesummaryuntilalljobsarelistedas running :

$watchmonitsummary

f. ExitoutoftheMySQLnode.

6. Performthefollowingstepstoincreasethesizeoftheclusterbacktothree:

a. FromtheOpsManagerInstallationDashboard,clicktheMySQL for PCFtile.b. ClickResource Config.c. SetthenumberofinstancesforMySQL Serverto 3 .d. ClickSave.e. ReturntotheOpsManagerInstallationDashboardandclickApply Changes.

PerformManualBackupIfyoudonotwanttousetheautomatedbackupsincludedinMySQLforPCF,youcanperformbackupsmanually.

RetrieveIPAddressandCredentialsPerformthefollowingstepstoretrievetheIPaddressandcredentialsrequiredforamanualbackup:

1. FromtheOpsManagerInstallationDashboard,clicktheMySQL for PCFtile.

2. ClicktheStatustab.

3. LocatetheIPaddressfortheMySQLnodeunderMySQL Server.

4. IntheCredentialstab,fromtheMySQL ServerjobandMysql Admin Passwordname,obtaintheadminpassword.

ManualBackupBackupyourdatamanuallywithmysqldump .Thisbackupacquiresaglobalreadlockonalltables,butdoesnotholditfortheentiredurationofthedump.

TobackupalldatabasesintheMySQLdeployment:

$mysqldump-uadmin-p-h$MYSQL_NODE_IP--all-databases--single-transaction>user_databases.sql

Tobackupasingledatabase,specifythedatabasename:

$mysqldump-uadmin-p-h$MYSQL_NODE_IP$DB_NAME--single-transaction>user_databases.sql

ManualRestoreTheprocedureforrestoringfromabackupisthesamewhetheroneormultipledatabaseswerebackedup.ExecutingtheSQLdumpwilldrop,recreate,andrefillthespecifieddatabasesandtables.

Preparetorestore:

IfrunninginHAconfiguration,reducethesizeoftheMySQLforPCFclustertoasinglenode,followingtherestoreinstructionsabove.

LocatetheMySQLAdmincredentialsintheCredentialstab,asabove.

UsetheMySQLpasswordandIPaddresstoenablethecreationoftablesusinganystorageengine.

$mysql-uadmin-p-h$MYSQL_NODE_IP-e"SETGLOBALenforce_storage_engine=NULL"

UsetheMySQLpasswordandIPaddresstorestoretheMySQLdatabasesbyrunningthefollowingcommand.

$mysql-uadmin-p-h$MYSQL_NODE_IP<user_databases.sql

UsetheMySQLpasswordandIPaddresstorestoreoriginalstorageenginerestriction.

WARNING:Restoringadatabasedeletesalldatathatexistedinthedatabasebeforetherestore.Restoringadatabaseusingafullbackupartifact,producedby mysqldump--all-

databasesforexample,replacesalldataanduserpermissions.

$mysql-uadmin-p-h$MYSQL_NODE_IP-e"SETGLOBALenforce_storage_engine='InnoDB'"

TorestoreHAmode,re-configureMySQLforPCFtorunusingthreenodesinthesamewayastherestoringinstructionsabove.

IfnotrunningHAmode,it’simportanttorestartthedatabaseserver.ThisstepisnotnecessaryifscalingbacktothreeMySQLnodes.

$monitstopmariadb_ctrl$monitstartmariadb_ctrl

Formoreexamplesofmanualbackupandrestoreprocedures,seetheMariaDBdocumentation .

Scaling Down MySQLThistopicdescribeshowtosafelyscaledownyourMySQLforPivotalCloudFoundry(PCF)clustertoasinglenode.

BydefaultMySQLforPCFisasinglenode.TotakeadvantageofthehighavailabilityfeaturesofMySQLforPCF,youmayhavescaledtheconfigurationuptothreenodes.

ChecktheHealthofYourClusterBeforescalingdownyourMySQLcluster,performthefollowingactionstoensuretheclusterishealthy.

1. ObtaintheIPaddressesofyourMySQLserverbyperformingthefollowingsteps:

a. FromthePivotalCloudFoundry(PCF)Installation Dashboard,clicktheMySQL for Pivotal Cloud Foundrytile.b. ClicktheStatustab.c. RecordtheIPaddressesforallinstancesoftheMySQL Serverjob.

2. ObtaintheadmincredentialsforyourMySQLserverbyperformingthefollowingsteps:

a. FromtheMySQLtile,clicktheCredentialstab.b. LocatetheMysql Admin PasswordentryintheMySQL ServersectionandclickLink to Credential.c. Recordthevaluesfor identity and password .

3. SSHintotheOpsManagerVM.BecausetheproceduresvarybyIaaS,reviewtheSSHintoOpsManager sectionoftheAdvancedTroubleshootingwiththeBOSHCLItopicforspecificinstructions.

4. FromtheOpsManagerVM,placesomedatainthefirstnodebyperformingthefollowingsteps,replacing FIRST-NODE-IP-ADDRESS withtheIPaddressofthefirstnoderetrievedaboveand YOUR-IDENTITY withthe identity valueobtainedabove.Whenpromptedforapassword,providethepassword valueobtainedabove.

a. Createadummydatabaseinthefirstnode:

$mysql-hFIRST-NODE-IP-ADDRESS-uYOUR-IDENTITY-p-e"createdatabaseverify_healthy;"

b. Createadummytableinthedummydatabase:

$mysql-hFIRST-NODE-IP-ADDRESS-uYOUR-IDENTITY-p-Dverify_healthy-e"createtabledummy_table(idintnotnullprimarykeyauto_increment,infotext)engine='InnoDB';"

c. Insertsomedataintothedummytable:

$mysql-hFIRST-NODE-IP-ADDRESS-uYOUR-IDENTITY-p-Dverify_healthy-e"insertintodummy_table(info)values('dummydata'),('moredummydata'),('evenmoredummydata');"

d. Querythetableandverifythatthethreerowsofdummydataexistonthefirstnode:

mysql-hFIRST-NODE-IP-ADDRESS-uYOUR-IDENTITY-p-Dverify_healthy-e"select*fromdummy_table;"Enterpassword:+----+----------------------+|id|info|+----+----------------------+|4|dummydata||7|moredummydata||10|evenmoredummydata|+----+----------------------+

5. VerifythattheothernodescontainthesamedummydatabyperformingthefollowingstepsforeachoftheremainingMySQLserverIPaddressesobtainedabove:

a. Querythedummytable,replacing NEXT-NODE-IP-ADDRESS withtheIPaddressoftheMySQLserverinstanceand YOUR-IDENTITY withtheidentity valueobtainedabove.Whenpromptedforapassword,providethe password valueobtainedabove.

$mysql-hNEXT-NODE-IP-ADDRESS-uYOUR-IDENTITY-p-Dverify_healthy-e"select*fromdummy_table;"

b. Examinetheoutputofthe mysql commandandverifythatthenodecontainsthesamethreerowsofdummydataastheothernodes.

Note:IfyouareonlyrunningtheMySQLclusterwithasinglenode,youdonotneedtoperformthesesteps.

+----+----------------------+|id|info|+----+----------------------+|4|dummydata||7|moredummydata||10|evenmoredummydata|+----+----------------------+

6. IfeachMySQLserverinstancedoesnotreturnthesameresult,contactPivotalSupport beforeproceedingfurtherormakinganychangestoyourdeployment.IfeachMySQLserverinstancedoesreturnthesameresult,thenyoucansafelyproceedtoscalingdownyourclustertoasinglenodebyperformingthestepsinthefollowingsection.

ScaleDownYourCluster1. Deletethedummydatabase,replacing FIRST-NODE-IP-ADDRESS withtheIPaddressofthefirstMySQLservernodeand YOUR-IDENTITY withthe

identity valueobtainedabove.Whenpromptedforapassword,providethe password valueobtainedabove.

$mysql-hFIRST-NODE-IP-ADDRESS-uYOUR-IDENTITY-p-e"dropdatabaseverify_healthy;"

2. FromthePCFInstallation Dashboard,clicktheMySQL for Pivotal Cloud Foundrytile.

3. ClicktheSettingstab.

4. ClickResource Configandusethedrop-downmenutochangetheInstancescountforMySQL Serverto 1 .

5. ClickSavetoapplythechanges.

Rotating MySQL for PCF CredentialsPage last updated:

ThistopicdescribeshowtorotatecredentialsforMySQLforPivotalCloudFoundry(MySQLforPCF).IfyouarealsousingElasticRuntimeMySQL,reviewthenotesinthisprocedureinordertorotatecredentialsforbothproducts.

PrerequisitesToperformthestepsbelow,youneedtoobtainthefollowing:

1. YourrootCAcertificateina .crt file.ToretrievetherootCAcertificateofyourdeployment,runthefollowingcommand:

$curl"https://YOUR-OPSMAN-IP-ADDRESS/api/v0/security/root_ca_certificate"

2. YourMySQLforPCFrootpassword.ToretrieveyourMySQLforPCFrootpassword,navigatetotheOpsManagerInstallationDashboardandselectMySQL for Pivotal Cloud Foundry > Credentials.YourMySQLforPCFrootpasswordiscalled MysqlAdminPassword .

RotateYourMySQLforPCFCredentials1. InstalltheUserAccountandAuthentication(UAA)CommandLineInterface(UAAC).

$geminstallcf-uaac

2. Makesure uaac gemisinstalled.

$whichuaac/Users/pivotal/.gem/ruby/2.3.0/bin/uaac

3. TargetyourOpsManagerUAAandprovidethepathtoyourrootCAcertificate.

$uaactargethttps://YOUR-OPSMAN-FQDN/uaa/--ca-certYOUR-ROOT-CA.crtTarget:https://YOUR-OPSMAN-FQDN/uaa/

4. Getyourtokenwith uaactokenownerget .Enter opsman for Client ID .Pressenterfor Client secret toleaveitblank.UsetheusernameandpasswordyouusedabovetologintotheOpsManagerwebinterfacefor User name and Password .

$uaactokenownergetClientID:opsmanClientsecret:Username:adminPassword:*********Successfullyfetchedtokenviaownerpasswordgrant.Target:https://YOUR-OPSMAN-FQDN/uaaContext:admin,fromclientopsman

5. RunthefollowingcommandtodisplaytheusersandapplicationsauthorizedbytheUAAserver,andthepermissionsgrantedtoeachuserand

Note:TheOpsManagerAPIreturnsthecertificateinJSONformatwith \n foreverynewline.Removealloccurrencesof \n whenyoucopythecertificateintoa .crt file.

Note:IfyouuseElasticRuntimeMySQL,youalsoneedyourElasticRuntimeMySQLrootpassword.ToretrieveyourElasticRuntimeMySQLrootpassword,navigatetotheOpsManagerInstallation DashboardandselectMySQL > Credentials.YourElasticRuntimeMySQLrootpasswordiscalled MysqlAdminCredentials .

application.

$uaaccontext[1][https://YOUR-OPSMAN-FQDN/uaa]skip_ssl_validation:trueca_cert:/Users/pivotal/.ssh/YOUR-ROOT-CA.crt[0]*[admin]user_id:75acfdfa-9449-4497-a093-ce40ded250acclient_id:opsmanaccess_token:LONG_ACCESS_TOKEN_STRINGtoken_type:bearerrefresh_token:LONG_REFRESH_TOKEN_STRINGexpires_in:43199scope:clients.readopsman.useruaa.adminscim.readopsman.adminclients.writescim.writejti:8419c793d377429aa40eea07fb6e7686

6. Createafilecalled uaac-token thatcontainsonlythe LONG_ACCESS_TOKEN_STRING fromtheoutputabove.

7. Use curl tomakearequesttotheOpsManagerAPI.Authenticatewiththecontentsofthe uaac-token fileandpipetheresponseintoinstallation_settings_current.json .

$curl-skH"Authorization:Bearer$(catuaac-token)"https://YOUR-OPSMAN-FQDN/api/installation_settings>installation_settings_current.json

8. ChecktoseethattheMySQLforPCFrootpasswordisinthecurrentinstallationsettingsfile:

$grep-cYOUR-MYSQL-FOR-PCF-ROOT-PASSWORDinstallation_settings_current.json

9. Removetherootpasswordfromtheinstallationsettingsfile.

$sed-e's/"value":{"identity":"root","password":"[^"]*"},$"identifier":"mysql_admin$/\1/g'installation_settings_current.json>installation_settings_updated.json

10. Validatethattherootpasswordhasbeenremovedfromthe installation_settings_updated.json file.

$grep-cYOUR-MYSQL-FOR-PCF-ROOT-PASSWORDinstallation_settings_updated.json0

11. Uploadtheupdatedinstallationsettings.

$curl-skXPOST-H"Authorization:Bearer$(catuaac-token)""https://YOUR-OPSMAN-FQDN/uaa/api/installation_settings"-F'installation[file]=@installation_settings_updated.json'{}

12. NavigatetotheOpsManagerInstallation DashboardandclickApply Changes.

13. Oncetheinstallationhascompleted,validatethattheMySQLforPCFrootpasswordhasbeenchanged.RetrievethenewpasswordfromMySQL >Credentials.UsetheIPaddressfortheMySQL ProxylocatedintheStatustab.

$mysql-uroot-p-h198.51.100.1Enterpassword:WelcometotheMariaDBmonitor.Commandsendwith;or\g.[...]

Note:IfyouuseElasticRuntimeMySQL,youshouldalsorunthefollowingcommand: $grep-cYOUR-ERT-MYSQL-ROOT-PASSWORD

installation_settings_current.json

Note:IfyouuseElasticRuntimeMySQL,youshouldalsorunthefollowingcommand: $grep-cYOUR-ERT-MYSQL-ROOT-PASSWORD

installation_settings_updated.json

Note:IfyouuseElasticRuntimeMySQL,youshouldalsovalidatethattheElasticRuntimeMySQLrootpasswordhasbeenchanged.RetrievethenewpasswordfromElastic Runtime > Credentials.UsetheIPaddressfortheMySQL Proxy,locatedintheStatustab.

Running mysql-diagThistopicdiscusseshowtousethe mysql-diag toolinMySQLforPivotalCloudFoundry(PCF). mysql-diag relaysthestateofyourMySQLserviceandsuggestsstepstotakeintheeventofanodefailure.InconjunctionwithPivotalSupport,thistoolhelpsexpeditethediagnosisandresolutionofproblemswithMySQLforPCF.

InMySQLforPCF1.9.0andlater, mysql-diag isautomaticallyinstalledandconfigured.IfyouarerunningMySQLforPCF1.8.xandearlierthenyouwillneedtocreateaconfigurationfileinordertouse mysql-diag .

PrepareYourEnvironmentMySQLforPCF1.9.0andlatershipswiththe mysql-diag toolandcomeswithanautomaticallygeneratedconfigurationfile.Inversions1.9.0.andlater,youcanfind mysql-diag onthe mysql-monitor node.

IfyouarerunningMySQLforPCF1.8.xorearlierthenyoumustdownload mysql-diag andcreateaconfigurationfile.Ifyoudonothaveamonitornode,asisthecasewithsomeolderversionsofthesoftware,Pivotalrecommendsthatyouuseoneofthemysqlclusternodesinstead.

OnlycompletethedownloadandconfigurationinstructionsbelowifyouareonMySQLforPCF1.8.xorearlier.

DownloadandRunmysql-diagTodownload mysql-diag :

1. Downloadthefilelabeledmysql-diag.confattachedtotheDiagnosingproblemswithElasticRuntimeMySQLorthePivotalMySQLTile

KnowledgeBasearticle.

2. Copythatbinarytothe mysql-monitor VMwiththefollowingcommand: boshscpJOB-NAMEJOB-INSTANCE-NUMBER--uploadLOCAL-FILE-PATHREMOTE-

FILE-PATH

Runningthe boshinstances commandwilldisplaytheinformationneededtoinserttheJOB-NAMEandJOB-INSTANCE-NUMBERoptions.Formoreinformationonthe boshistances command,seetheboshdocumentation onsystemadministrationtasks.TheLOCAL-FILE-PATHoptionisthepathtowhereyouwanttolocatethemysql-diag.conffile.TheREMOTE-FILE-PATHoptionistheinitiallocationofthemysql-diag.conffile.

1. Executethemysql-diag.conffilewiththefollowingcommand:

mysql-diag-c./mysql-diag.conf

Configuremysql-diagToconfigure mysql-diag :

1. PastetheConfigurationFileTemplatebelowintoatexteditor

{"mysql":{"username":"repcanary","password":"password","port":3306,"nodes":[{"host":"10.244.7.4",},{"host":"10.244.8.4",},{"host":"10.244.9.4",}]}}

2. ReplacethepasswordswiththevaluesthatyoufindinOpsManwithintheCredentialstab.

3. CopythecompletedtemplateintothesameVMthatyoudownloadedthe mysql-diag tool,usingthe boshscp command.

4. Movetheconfigurationfiletothesamedirectoryasthe mysql-diag tool.

5. Runthefollowingcommandinordertostartthetool:

$mysql-diag-c./diag-config.json

mysql-diag-agentMySQLforPCF1.9.0andlaterwillhavethe mysql-diag-agent present.Versions1.8.xandearlierofMySQLforPCFdonothavethe mysql-diag-agent .Ifthemysql-diag-agent isnotavailable,youroutputfromthe mysql-diag toolwillnotincludethepercentageofPersistentandEphemeralDiskspaceusedbya

ExampleHealthyOutputThereplicationcanaryinversions1.7.10andearlierofMySQLforPCFdoesnotprovideareplicationAPI.ForthoseversionsofPivotalMySQL, mysql-diag

willnotbeabletodetermineifyourcanarystatusis“healthy”or“unhealthy.”

Checkingcanarystatus...Gethttp://127.0.0.1:8111/api/v1/status:dialtcp127.0.0.1:8111:getsockopt:connectionrefused

Hereisasample mysql-diag outputafterthetoolhasidentifiedahealthyclusterinaMySQLforPCFversionthatdoesnotcontainthemysql-diag-agent :

Checkingclusterstatusofmysql/a1at10.0.16.44...Checkingclusterstatusofmysql/c3at10.0.32.10...Checkingclusterstatusofmysql/b2at10.0.16.45...Checkingclusterstatusofmysql/a1at10.0.16.44...doneCheckingclusterstatusofmysql/c3at10.0.32.10...doneCheckingclusterstatusofmysql/b2at10.0.16.45...done+------------+-----------+-------------------+----------------------+--------------------+|HOST|NAME/UUID|WSREPLOCALSTATE|WSREPCLUSTERSTATUS|WSREPCLUSTERSIZE|+------------+-----------+-------------------+----------------------+--------------------+|10.0.16.44|mysql/a1|Synced|Primary|3||10.0.32.10|mysql/c3|Synced|Primary|3||10.0.16.45|mysql/b2|Synced|Primary|3|+------------+-----------+-------------------+----------------------+--------------------+Idon'tthinkbootstrapisnecessaryCheckingdiskstatusofmysql/a1at10.0.16.44...Checkingdiskstatusofmysql/c3at10.0.32.10...Checkingdiskstatusofmysql/b2at10.0.16.45...Checkingdiskstatusofmysql/a1at10.0.16.44...dialtcp10.0.16.44:getsockopt:connectionrefuseCheckingdiskstatusofmysql/c3at10.0.32.10...dialtcp10.0.16.44:getsockopt:connectionrefuseCheckingdiskstatusofmysql/b2at10.0.16.45...dialtcp10.0.16.44:getsockopt:connectionrefuse

ExampleUnhealthyOutputIntheeventofabrokencluster,running mysql-diag outputsactionablestepsmeanttoexpeditetherecoveryofthecluster.Belowisasample mysql-diag

outputafterthetoolidentifiedanunhealthyclusterinaMySQLforPCFversionthatdoesnotcontainthe mysql-diag-agent :

Checkingclusterstatusofmysql/a1at10.0.16.44...Checkingclusterstatusofmysql/c3at10.0.32.10...Checkingclusterstatusofmysql/b2at10.0.16.45...Checkingclusterstatusofmysql/a1at10.0.16.44...dialtcp10.0.16.44:getsockopt:connectionrefusedCheckingclusterstatusofmysql/c3at10.0.32.10...dialtcp10.0.32.10:getsockopt:connectionrefusedCheckingclusterstatusofmysql/b2at10.0.16.45...dialtcp10.0.16.45:getsockopt:connectionrefused

+------------+-----------+-------------------+----------------------+--------------------+|HOST|NAME/UUID|WSREPLOCALSTATE|WSREPCLUSTERSTATUS|WSREPCLUSTERSIZE|+------------+-----------+-------------------+----------------------+--------------------+|10.0.16.44|mysql/a1|N/A-ERROR|N/A-ERROR|N/A-ERROR||10.0.16.45|mysql/b2|N/A-ERROR|N/A-ERROR|N/A-ERROR||10.0.32.10|mysql/c3|N/A-ERROR|N/A-ERROR|N/A-ERROR|+------------+-----------+-------------------+----------------------+--------------------+

Checkingdiskstatusofmysql/a1at10.0.16.44...Checkingdiskstatusofmysql/c3at10.0.32.10...Checkingdiskstatusofmysql/b2at10.0.16.45...Checkingdiskstatusofmysql/a1at10.0.16.44...dialtcp10.0.16.44:getsockopt:connectionrefusedCheckingdiskstatusofmysql/c3at10.0.32.10...dialtcp10.0.32.10:getsockopt:connectionrefusedCheckingdiskstatusofmysql/b2at10.0.16.45...dialtcp10.0.16.45:getsockopt:connectionrefused

[CRITICAL]Thereplicationprocessisunhealthy.Writesaredisabled.

[CRITICAL]Runthedownload-logscommand:$download-logs-d/tmp/output-n10.0.16.44-n10.16.45-n10.0.32.10Forfullinformationabouthowtodownloadandusethedownload-logscommandseehttps://discuss.pivotal.io/hc/en-us/articles/221504408

[WARNING]DonotperformthefollowingunlessinstructedbyPivotalSupport:-Donotscaledowntheclustertoonenodethenscaleback.Thisputsuserdataatrisk.-Avoid“boshrecreate”and“boshcck”.TheseoptionsremovelogsontheVMsmakingithardertodiagnoseclusterissues.

Table of Contents - docs.pivotal.iodocs.pivotal.io/archives/mysql-docs-1.7.pdf... , and Service...

Documents

Foundry Products (Ideal Foundry

ncrib.net · boof africa ins. brokers ltd. bridgate assocate brokers ltd. brokers ltd. brightways brokers 0. bsi ins. brokers ltd. busag brokers ito (risk solution) brokers ltd. capacity

CLOUD Lc08 Introduction to CloudFoundry - Cloud …mse-cloud.s3-website-eu-west-1.amazonaws.com/docs/CLOUD Lc08...Cloud Foundry Components Service Broker ... Makes create/delete/bind/unbind

(Stock-Brokers and Sub-Brokers) Regulations, 1992

Cloud Foundry Roadmap (Cloud Foundry Summit 2014)

Foundry THE FOUNDRY - WordPress.com · THE FOUNDRY: A CROSSROADS FOR FORGING NEW STEAM OPPORTUNITIES 4 The Foundry. ... The Foundry Project will be accomplished under the administration

Ties to Unbind: Political Ties and Firm Sell-Offs During … · 2017. 9. 1. · Ties to Unbind: Political Ties and Firm Sell-Offs During Institutional Transition Weiting Zheng Hong

RIGHTS AND OBLIGATIONS OF STOCK BROKERS, SUB-BROKERS …

DATA SHEET - Icecatobjects.icecat.biz/objects/mmo_76848051_1575029424_3985_30657… · Foundry Katana • • Foundry Mari • Foundry Modo • Foundry Nuke • VR Authoring. Adobe

Director of Technology Elkem Foundry DivisionElkem Foundry Division Kristiansand, Norway€¦ · · 2012-11-20Elkem Foundry DivisionElkem Foundry Division Kristiansand, Norway Elkem

FOUNDRY PRODUCTS CATALOGUE - Molten Metalmoltenmetalproducts.com/products/foundry/foundry-products.pdf · FOUNDRY PRODUCTS CATALOGUE Innovating tomorrow’s solutions . . . today

Water brokers and exchanges - your fair trading obligations › system › files › Water brokers... · Water brokers and exchanges— your fair trading obligations Water brokers

Jesus said, Unbind him, set him free, and let him go. John 11:44b · 2019. 5. 13. · turned to those standing nearby and said,"Unbind him; set him free; let him go." (John 11:44b)

Gießerei-Industrie Foundry Industry€¦ · foundry industry, we offer machines and systems for the ‘green sand foundry’ and ‘no-bake sand foundry’ processes like coolers,

Who are we? Polish Foundry Group (Polska Grupa Odlewnicza) is an association of two foundries: Pioma-Odlewnia Foundry and Iron Foundry Śrem.Polish Foundry

WFO WORLD FOUNDRY SUMMIT World Foundry Summit...CEO’s Reshaping the Foundry Industry The World Foundry Organization is delighted to be hosting a World Foundry Summit in Rome, Italy,

Ties that Unbind? Ethnic Identity, Social Rules and ... · Ties that Unbind? Ethnic Identity, Social Rules and Electoral Politics in Solomon Islands Terence Wood A thesis submitted

Investment Foundry. TYPES OF FOUNDRY TYPES OF FOUNDRY 1. Sand Foundry 1. Sand Foundry 2. Investment Foundry 2. Investment Foundry Investment Foundry Process

Rheumatism Foundry Workersoem.bmj.com/content/oemed/23/1/42.full.pdf · Rheumatism in Foundry Workers ... Foundry Expected Foundry Expected Foundry ... significantly less frequent

Business Brokers NYC -LISITEN ASSOCIATES BUSINESS BROKERS