Upload
others
View
38
Download
0
Embed Size (px)
Citation preview
RabbitMQ for PCF®
Version 1.10
User's Guide
© Copyright Pivotal Software Inc, 2013-present
236
14182130313339404951697476799296
100103111
Table of ContentsTable of ContentsRabbitMQ for PCFRabbitMQ® for PCF Release NotesUnlocking the Power of On-Demand RabbitMQ for PCFOn-Demand Service ArchitectureInstalling and Configuring the On-Demand ServiceSmoke TestsUpgrading RabbitMQ for PCFSetting Limits for On-Demand Service InstancesControlling Access to Service Plans by OrgMonitoring and KPIs for On-Demand RabbitMQ for PCFClustering and Network PartitionsTroubleshooting On-Demand RabbitMQ for PCFUsing On-Demand RabbitMQ for PCFRabbitMQ® Environment VariablesTroubleshooting InstancesInstalling and Configuring the Pre-Provisioned ServiceCreating Isolation with the RabbitMQ for PCF ReplicatorDeploying the RabbitMQ Pre-Provisioned ServiceSetting Default Policies for the RabbitMQ ServiceMonitoring and KPIs for Pre‑Provisioned RabbitMQ for PCFManaging the RabbitMQ® Service
© Copyright Pivotal Software Inc, 2013-2018 2 1.10
RabbitMQforPCF
AboutRabbitMQforPCFRabbitMQforPivotalCloudFoundry(PCF)enablesPCFappdeveloperstoprovisionandusetheRabbitMQmessagebrokerwithasinglecommand.
Asofversion1.8,RabbitMQforPCFsupportstwotypesofservice,anon-demandserviceandapre-provisionedservice.Thistablesummarizesthemaindifferencesbetweenthetwo:
AvailableSince VMsitRunsOn HowVMsareCreated MetricsName
Prefix
On-DemandService v1.8
DedicatedVMthatservesasingleserviceinstance.Seethistopicfordetails.
PCFcreateseachVMon-demandwhenappdevelopercreatesserviceinstance
p.rabbitmq
(withadot)
Pre-ProvisionedService
v1.2Multi-tenantVMssharedbyappsacrossPCFdeployment
PCFcreatesallVMswhenoperatordeploysorupdatesservice
p-rabbitmq
(withadash)
ThisRabbitMQforPCFdocumentationdescribesbothservicetypes.DocumentationforRabbitMQforPCFv1.7andearlieronlydescribesapre-provisionedservice.
WhatareOn-DemandInstancesInRabbitMQforPCFversionsbeforev1.8.0,theRabbitMQserviceinstancescorrespondtoauniqueRabbitMQVhostonthemulti-tenantRabbitMQcluster.RabbitMQforPCFv1.8.0introducedOn-DemandBroker(ODB) support.Thatmeansthatanew,single-tenant,clustercanbecreatedanddedicatedtoasingleapp.
Formoreinformation,seeUnlockingthePowerofOn-DemandRabbitMQforPCFandOn-DemandServiceArchitecture.
AboutRabbitMQRabbitMQisafastanddependableopen-sourcemessageserver,whichsupportsawiderangeofusecasesincludingreliableintegration,content-basedroutingandglobaldatadelivery,andhigh-volumemonitoringanddataingestion.
Emergingasthedefactostandardforcloudmessaging,RabbitMQisusedforefficientcommunicationbetweenservers,appsanddevices,andcreateslastingvaluebyenablingrapiddevelopmentofmoderndecentralizedappanddataarchitecturesthatcanscalewithyourbusinessneeds.
ProductSnapshotThefollowingtableprovidesversionandversion-supportinformationaboutRabbitMQforPCF.
Element Details
Version v1.10.20
Releasedate July13,2018
Softwarecomponentversion RabbitMQOSSv3.6.16
CompatibleOpsManagerversion(s) v1.11.xandv1.12.x
CompatibleElasticRuntimeversion(s) v1.11.xandlater
IaaSsupport AWS,Azure,GCP,OpenStack,andvSphere
IPsecsupport No
Note:RabbitMQforPCFv1.10isnolongersupported.Thesupportperiodforv1.10hasexpired.Tostayup-to-datewiththelatestsoftwareandsecurityupdates,upgradetoRabbitMQforPCFv1.12orlater.
© Copyright Pivotal Software Inc, 2013-2018 3 1.10
Features
On-DemandProvisionon-demandsinglenodededicatedinstancesofRabbitMQwithauniqueRabbitMQVhost(virtualhost)
Provisionon-demandclustereddedicatedinstancesofRabbitMQ
Bindappstoaninstanceoftheplan,providinguniquecredentialsforeachbinding
Managementdashboardaccesstoappdevelopers
Deploymentintoanavailabilityzonespecifiedbytheplan
AutomatedupgradesofRabbitMQformajor,minor,andpatchreleases(seereleasenotesfordowntimerequirements)
RabbitMQSyslogforwardingconfigurationinheritedfromthepre-provisionedconfiguration
RabbitMQmetricsareexposedonthefirehose
Runsmoketestsforon-demandplans
Formoreinformation,seeUnlockingthePowerofOn-DemandRabbitMQforPCF.
Pre-ProvisionedProvisionaninstanceoftheRabbitMQservice,whichcorrespondstoauniqueRabbitMQVhost(virtualhost)
Bindappstoaninstanceoftheplan,providinguniquecredentialsforeachbinding
ManagementdashboardaccesstoPCFOperatorsandappdevelopers
Deploymentacrossmultipleavailabilityzones,withnodesstripedacrosstheAZsautomatically
EnableSSL(SecureSocketsLayer)fortheAMQP,MQTT,STOMPprotocols
HAProxyloadbalanceracrossallnodestobalanceconnections
Pluginconfigurationcanbeeasilychangedatanytimeandtheclusterredeployedandupdated
Theclustertopologycanbechangedandeasilyscaledout
AutomatedupgradesofRabbitMQformajor,minor,andpatchreleases(seeDowntimeWhenUpgradingfordowntimerequirements)
ConfiguretheendpointfortheRabbitMQSyslog
RabbitMQandHAProxymetricsareexposedonthefirehose
Syslogforwardingonbydefault
ReleaseNotesandKnownIssuesCheckthereleasenotesforyourreleaseversionforimportantinformationandknownissues.Toseereleasenotesforanotherversion,selecttheversionfromthedropdownlistatthetopofthepage.
RabbitMQforPCFandOtherPCFServicesSomePCFservicesofferon-demandserviceplans.Theseplansletdevelopersprovisionserviceinstanceswhentheywant.
Thesecontrastwiththemorecommonpre-provisionedserviceplans,whichrequireoperatorstoprovisiontheserviceinstancesduringinstallationandconfigurationthroughtheservicetileUI.
ThefollowingPCFservicesofferon-demandserviceplans:
MySQLforPCFv2.0andlater
RabbitMQforPCF
RedisforPCF
PivotalCloudCache(PCC)
Theseservicespackageanddelivertheiron-demandserviceofferingsdifferently.Forexample,someservices,likeRedisforPCF,haveonetile,andyouconfigurethetiledifferentlydependingonwhetheryouwanton-demandserviceplansorpre-provisionedserviceplans.
© Copyright Pivotal Software Inc, 2013-2018 4 1.10
Forotherservices,likePCC,youinstallonetileforon-demandserviceplansandadifferenttileforpre-provisionedserviceplans.
ThefollowingtablelistsandcontraststhedifferentwaysthatPCFservicespackageon-demandandpre-provisionedserviceofferings.
PCFservicetile Standaloneproductrelatedtotheservice Versionssupportingondemand Versionssupportingpre-provisioned
RabbitMQforPCF PivotalRabbitMQ v1.8andlater Allversions
RedisforPCF Redis v1.8andlater Allversions
MySQLforPCF MySQLv2.x(basedonPerconaServer)
v1.x(basedonMariaDBandGalera)
PCC PivotalGemFire Allversions NA
GemFireforPCF PivotalGemFire NA Allversions
Pleaseprovideanybugs,featurerequests,orquestionstothePCFFeedbacklist.
© Copyright Pivotal Software Inc, 2013-2018 5 1.10
RabbitMQ®forPCFReleaseNotes
UpgradetotheLatestVersionPivotalrecommendsthatyouupgradetothelatestversionofyourcurrentminorline,thenupgradetothelatestavailableversionofthenewminorline.Forexample,ifyouuseanolderv1.7.xversion,upgradetothelatestv1.7.xversionbeforeupgradingtothelatestv1.8.xversion.
SeetheProductCompatibilityMatrix forproductversionsandupgradepaths.
v1.10.20ReleaseDate:July13,2018
FeaturesRequiresstemcell3445.51
KnownIssuesSmoketestsmightfailbecausetime-outvaluesinthesmoketestsaretoolow.Formoreinformationabouttroubleshootingsmoketests,seeSmokeTests.
ClusterscalingorchangingtheErlangCookievaluerequiresclusterdowntime,andmightresultinfaileddeployments.Formoreinformation,seeClusterScalingKnownIssueandChangingtheErlangCookieValueKnownIssue.
Changingnetworksand/orIPaddressesforthe RabbitMQ Server jobresultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .
WhenerrandrunrulesaresettoWhenChanged,OpsManagermaynotruntheerrandswhenthetilehasrelevantchanges.Formoreinformation,seeManagingErrandsinOpsManager .PivotalrecommendsleavingthedefaultrunrulesettoOn.
PackagesOSSRabbitMQv3.6.16
Erlangv20.3.8.1
HAProxyv1.6.13
v1.10.19ReleaseDate:June5,2018
FeaturesRequiresstemcell3445.48
cf-cligolangversionreducedtov1.9.5fromv1.10becauseofanissueparsingx509certificates.Formoreinformation,seetheGolangissuecrypto/x509:CANotAuthorizedForExtKeyUsageispremature .
KnownIssuesSmoketestsmightfailbecausetime-outvaluesinthesmoketestsaretoolow.Formoreinformationabouttroubleshootingsmoketests,seeSmokeTests.
ClusterscalingorchangingtheErlangCookievaluerequiresclusterdowntime,andmightresultinfaileddeployments.Formoreinformation,seeClusterScalingKnownIssueandChangingtheErlangCookieValueKnownIssue.
Changingnetworksand/orIPaddressesforthe RabbitMQ Server jobresultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIP
© Copyright Pivotal Software Inc, 2013-2018 6 1.10
AddressesResultsinaFailedDeployment .
WhenerrandrunrulesaresettoWhenChanged,OpsManagermaynotruntheerrandswhenthetilehasrelevantchanges.Formoreinformation,seeManagingErrandsinOpsManager .PivotalrecommendsleavingthedefaultrunrulesettoOn.
PackagesOSSRabbitMQv3.6.15
Erlangv19.3.6.4
HAProxyv1.6.13
v1.10.18ReleaseDate:May16,2018
FeaturesRequiresstemcell3445.45
Timestampsaddedtologentriesthatdidnothavethem.
Golangversionreducedtov1.9.5fromv1.10becauseofanissueparsingx509certificates.Formoreinformation,seetheGolangissuecrypto/x509:CANotAuthorizedForExtKeyUsageispremature .
KnownIssuesSmoketestsmightfailbecausetime-outvaluesinthesmoketestsaretoolow.Formoreinformationabouttroubleshootingsmoketests,seeSmokeTests.
ClusterscalingorchangingtheErlangCookievaluerequiresclusterdowntime,andmightresultinfaileddeployments.Formoreinformation,seeClusterScalingKnownIssueandChangingtheErlangCookieValueKnownIssue.
Changingnetworksand/orIPaddressesforthe RabbitMQ Server jobresultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .
WhenerrandrunrulesaresettoWhenChanged,OpsManagermaynotruntheerrandswhenthetilehasrelevantchanges.Formoreinformation,seeManagingErrandsinOpsManager .PivotalrecommendsleavingthedefaultrunrulesettoOn.
PackagesOSSRabbitMQv3.6.15
Erlangv19.3.6.4
HAProxyv1.6.13
v1.10.17ReleaseDate:March28,2018
FeaturesRequiresstemcell3445.30
Updatesthefollowing:
Javapackagetoversion1.9On-DemandSevicesSDKtov0.20.0
Loggregatorreleasetov102.1
Routingreleasetov0.174.0
© Copyright Pivotal Software Inc, 2013-2018 7 1.10
KnownIssuesClusterscalingorchangingtheErlangCookievaluerequiresclusterdowntime,andmightresultinfaileddeployments.Formoreinformation,seeClusterScalingKnownIssueandChangingtheErlangCookieValueKnownIssue.
Changingnetworksand/orIPaddressesforthe RabbitMQ Server jobresultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .
WhenerrandrunrulesaresettoWhenChanged,OpsManagermaynotruntheerrandswhenthetilehasrelevantchanges.Formoreinformation,seeManagingErrandsinOpsManager .PivotalrecommendsleavingthedefaultrunrulesettoOn.
PackagesOSSRabbitMQv3.6.15
Erlangv19.3.6.4
HAProxyv1.6.13
v1.10.14ReleaseDate:February26,2018
Features[SecurityFix]Requiresstemcell3445.28 .
FixedIssuesUpgradesnolongerfailwhenSYSLOGisnotconfigured.
Specialcharacterssuchas []^_!"#$%&()*+,-./\:;<=>`' nolongerleadtoissuesloggingintotheManagementUI.Youcannowusespecialcharacters,withtheexceptionof ` and ' ,inRabbitMQusernamesandpasswords.
Metricsarenowemittedwhenusingspecialcharactersinpasswords.
IfyouchangetheRabbitMQadminusername,theusernolongerlosesaccesstopreexistingvhosts.
KnownissuesClusterscalingorchangingtheErlangCookievaluerequiresclusterdowntimeandcanresultinfaileddeployments.Formoreinformation,seeClusterScalingKnownIssueandChangingtheErlangCookieValueKnownIssue.
ChangingnetworksorIPaddressesforthe RabbitMQ Server jobresultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .
WhenerrandrunrulesaresettoWhenChanged,OpsManagermightnotruntheerrandswhenthetilehasrelevantchanges.Formoreinformation,seeManagingErrandsinOpsManager .PivotalrecommendsleavingthedefaultrunrulesettoOn.
PackagesOSSRabbitMQv3.6.15
Erlangv19.3.6.4
HAProxyv1.6.13
v1.10.13ReleaseDate:January30,2018
FeaturesUpdateRabbitMQtov3.6.15.Formoreinformation,seeRabbitMQ3.6.15ReleaseNotes .
© Copyright Pivotal Software Inc, 2013-2018 8 1.10
KnownissuesClusterscalingorchangingtheErlangCookievaluerequiresclusterdowntimeandcanresultinfaileddeployments.Formoreinformation,seeClusterScalingKnownIssueandChangingtheErlangCookieValueKnownIssue.
ChangingnetworksorIPaddressesforthe RabbitMQ Server jobresultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .
WhenerrandrunrulesaresettoWhenChanged,OpsManagermightnotruntheerrandswhenthetilehasrelevantchanges.Formoreinformation,seeManagingErrandsinOpsManager .PivotalrecommendsleavingthedefaultrunrulesettoOn.
TheRabbitMQadministratorpasswordmustbeacombinationofuppercaseandlowercasealphanumerics.Specialcharacterssuchas []^_!"#$%&()*+,-
./\:;<=>`' canleadtoissuesloggingintotheManagementUIasthatuser.
PackagesOSSRabbitMQv3.6.15
Erlangv19.3.6.4
HAProxyv1.6.13
v1.10.12-WithdrawnReleaseDate:January24,2018WithdrawalDate:February5,2018
ThisreleasewaswithdawnduetoanissueupgradingwhentherearespecialcharactersintheRabbitMQadministratorcredentials.Skipthisversionandupgradetov1.10.13orlater.
Features[SecurityFix]Requiresstemcell3445.24 toaddressSpectrevulnerabilities.
KnownissuesClusterscalingorchangingtheErlangCookievaluerequiresclusterdowntimeandcanresultinfaileddeployments.Formoreinformation,seeClusterScalingKnownIssueandChangingtheErlangCookieValueKnownIssue.
ChangingnetworksorIPaddressesforthe RabbitMQ Server jobresultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .
WhenerrandrunrulesaresettoWhenChanged,OpsManagermightnotruntheerrandswhenthetilehasrelevantchanges.Formoreinformation,seeManagingErrandsinOpsManager .PivotalrecommendsleavingthedefaultrunrulesettoOn.
PackagesOSSRabbitMQv3.6.14
Erlangv19.3.6.4
HAProxyv1.6.13
v1.10.11ReleaseDate:January22,2018
Features[SecurityFix]Requiresstemcell3445.23 toaddressGNUCLibraryvulnerabilities.
Knownissues
© Copyright Pivotal Software Inc, 2013-2018 9 1.10
ClusterscalingorchangingtheErlangCookievaluerequiresclusterdowntimeandcanresultinfaileddeployments.Formoreinformation,seeClusterScalingKnownIssueandChangingtheErlangCookieValueKnownIssue.
ChangingnetworksorIPaddressesforthe RabbitMQ Server jobresultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .
WhenerrandrunrulesaresettoWhenChanged,OpsManagermightnotruntheerrandswhenthetilehasrelevantchanges.Formoreinformation,seeManagingErrandsinOpsManager .PivotalrecommendsleavingthedefaultrunrulesettoOn.
PackagesOSSRabbitMQv3.6.14
Erlangv19.3.6.4
HAProxyv1.6.13
v1.10.9ReleaseDate:January12,2018
Features[SecurityFix]Requiresstemcell3445.22 toaddresstheMeltdownsecurityissue.FormoreinformationaboutMeltdown,seePivotalVulnerabilityReport:MeltdownandSpectreAttacks
Servicemetricsnowusesadrainscripttopreventmonittimeoutissues
On-Demand
smoke-tests errandisrenamedto on-demand-broker-smoke-tests
SomeCFtimeoutsforsmoketestshaveincreased
KnownissuesClusterscalingorchangingtheErlangCookievaluerequiresclusterdowntimeandcanresultinfaileddeployments.Formoreinformation,seeClusterScalingKnownIssueandChangingtheErlangCookieValueKnownIssue.
ChangingnetworksorIPaddressesforthe RabbitMQ Server jobresultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .
WhenerrandrunrulesaresettoWhenChanged,OpsManagermightnotruntheerrandswhenthetilehasrelevantchanges.Formoreinformation,seeManagingErrandsinOpsManager .PivotalrecommendsleavingthedefaultrunrulesettoOn.
PackagesOSSRabbitMQv3.6.14
Erlangv19.3.6.4
HAProxyv1.6.13
v1.10.8ReleaseDate:December12,2017
FeaturesRequiresstemcell3445.19
KnownIssues
© Copyright Pivotal Software Inc, 2013-2018 10 1.10
ClusterscalingorchangingtheErlangCookievaluerequireclusterdowntime,andmightresultinfaileddeployments.Formoreinformation,seeClusterScalingKnownIssueandChangingtheErlangCookieValueKnownIssue.
Changingnetworksand/orIPaddressesforthe RabbitMQ Server jobresultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .
WhenerrandrunrulesaresettoWhenChanged,OpsManagermaynotruntheerrandswhenthetilehasrelevantchanges.Formoreinformation,seeManagingErrandsinOpsManager .PivotalrecommendsleavingthedefaultrunrulesettoOn.
PackagesOSSRabbitMQv3.6.14
Erlangv19.3.6.4
HAProxyv1.6.13
v1.10.7ReleaseDate:December8,2017
FeaturesRequiresstemcell3445.17
UpdatetoRabbitMQv3.6.14(releasenotesavailablehere )
UpdatetoErlang19.3.6.4
Goupdate(securityfixes)
MinorupdatesinPre-Provisionedservice(multi-tenantbroker):
FixmisspellinginlogmessageImproveserviceofferingdescription
KnownIssuesClusterscalingorchangingtheErlangCookievaluerequireclusterdowntime,andmightresultinfaileddeployments.Formoreinformation,seeClusterScalingKnownIssueandChangingtheErlangCookieValueKnownIssue.
Changingnetworksand/orIPaddressesforthe RabbitMQ Server jobresultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .
WhenerrandrunrulesaresettoWhenChanged,OpsManagermaynotruntheerrandswhenthetilehasrelevantchanges.Formoreinformation,seeManagingErrandsinOpsManager .PivotalrecommendsleavingthedefaultrunrulesettoOn.
PackagesOSSRabbitMQv3.6.14
Erlangv19.3.6.4
HAProxyv1.6.13
v1.10.5ReleaseDate:November3,2017
Features
On-Demand
Requiresstemcell3445.16
Addsqueuedepth,memoryalarm,anddiskalarmmetrics
© Copyright Pivotal Software Inc, 2013-2018 11 1.10
Pre-Provisioned
Requiresstemcell3445.16
IncreasesmaxHAProxyconnectionsto64000
RemovesthestatsUIonHAProxyinstances
AllowsformoregranularTLSprotocolversionselection
Addsqueuedepth,memoryalarm,anddiskalarmmetrics
Fixesknownissuewheresomespecialcharactersintheadministratorpasswordcausedlogonproblems
KnownIssuesClusterscalingorchangingtheErlangCookievaluerequireclusterdowntime,andmightresultinfaileddeployments.Formoreinformation,seeClusterScalingKnownIssueandChangingtheErlangCookieValueKnownIssue.
Changingnetworksand/orIPaddressesforthe RabbitMQ Server jobresultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .
WhenerrandrunrulesaresettoWhenChanged,OpsManagermaynotruntheerrandswhenthetilehasrelevantchanges.Formoreinformation,seeManagingErrandsinOpsManager .PivotalrecommendsleavingthedefaultrunrulesettoOn.
PackagesOSSRabbitMQv3.6.12
Erlangv19.3.6.2
HAProxyv1.6.13
v1.10.3ReleaseDate:October16,2017
Features
On-Demand
On-demandRabbitMQclusters
MetricsforallinstancesautomaticallysenttotheFirehose
Singleconfigurationforsyslogforwardingforallinstances
Administratorpermissionsavailablethroughaservice-key
UpdatetoRabbitMQ3.6.12
Requiresstemcell3445.11
UpdatetoRabbitMQ3.6.12forallinstancesinvolvesmajorchanges tothememorycalculations
On-demandmetricsprefixedwith p.rabbitmq tomatchtheservicename
Smoketestsrunforon-demandplans
Pre-Provisioned
UpdatetoRabbitMQ3.6.12
Requiresstemcell3445.11
ThisversionofthetilecanonlybeinstalledinOpsManagerv1.11andv1.12environments
UpdatetoRabbitMQ3.6.12—thisinvolvesmajorchanges tothememorycalculations
HAProxyinstancecountcanbesettozeroifanexternalloadbalancerisused
KnownIssuesClusterscalingorchangingtheErlangCookievaluerequireclusterdowntime,andmightresultinfaileddeployments.Formoreinformation,see
© Copyright Pivotal Software Inc, 2013-2018 12 1.10
ClusterScalingKnownIssueandChangingtheErlangCookieValueKnownIssue.
Changingnetworksand/orIPaddressesforthe RabbitMQ Server jobresultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .
WhenerrandrunrulesaresettoWhenChanged,OpsManagermaynotruntheerrandswhenthetilehasrelevantchanges.Formoreinformation,seeManagingErrandsinOpsManager .PivotalrecommendsleavingthedefaultrunrulesettoOn.
CertainspecialcharacterscannotbeusedwiththeRabbitMQadministratorpassword,andusingthemcanleadtoissuesloggingintotheManagementUIasthatuser.
PackagesOSSRabbitMQv3.6.12
Erlangv19.3.6.2
HAProxyv1.6.13
ViewReleaseNotesforAnotherVersionToviewthereleasenotesforanotherproductversion,selecttheversionfromthedrop-downlistatthetopofthispage.
© Copyright Pivotal Software Inc, 2013-2018 13 1.10
UnlockingthePowerofOn-DemandRabbitMQforPCF
IntroductionRabbitMQforPivotalCloudFoundry(PCF)v1.10respondstothedemandsofPCFoperatorstoofferaRabbitMQon-demandclusterfortheirapplicationteams,inadditiontotheexistingsingle-nodeon-demandplan.Theon-demandclusterplanisaimedatworkloadsthatrequirethesameresiliencerequirementsasthePre-Provisionedoffering,butalsorequiretheirworkloadsbeisolated.
TheplatformoperationsteamcannowconfigureaRabbitMQforPCFclustertomeettheirbusinessrequirementsandempowerappdevelopmentteamstoself-servetheirownRabbitMQcluster.
RabbitMQforPCFv1.10alsoprovidessmoketestsfortheon-demandplanssothatoperationsteamscanvalidatetheappdeveloperworkflowforon-demandservices.SeeDedicatedInstanceSmokeTestProcess .
PlatformoperatorscannowoffertheirappdevelopersthreetypesofRabbitMQforPCFserviceplans:
Pre-provisioned—Forlighttomoderatemessagingneeds,thisserviceisfullyoperatedandmanagedbyplatformoperatorsasaservice.
On-demandsinglenode—Forapplicationteamsrequiringgreaterisolationthanprovidedbythepre-provisionedapproach.Appdevelopmentteamscanhavefullaccesstotheirownmessagebrokertoadapttheruntimeparameterstotheirrequirements.Formoreinformationontheseparameters,seeParametersandPolicies intheRabbitMQdocumentation.
On-demandcluster—Foranincreasedlevelofmessageresilienceandclusteravailability,aswellasthebenefitsofworkloadisolationmentionedabove.
Thistopicexplainshowtobenefitfromthetwoon-demandplansabove.
Forinformationaboutthepre-provisionedplan,seeDeployingtheRabbitMQPre-ProvisionedService.Forinformationonusingpre-provisionedplanstoisolateworkloads,seeCreatingIsolationwiththeTileReplicator.
On-DemandSingleNodePlanThisplanisdesignedtobesimpletoconfigure,deploy,anduse.ItgivesapplicationteamsfastaccesstothepoweroftheleadingopensourcemessagebrokerbackedbyBOSHtomeetallbutthemostdemandinghighavailabilityappmessagingrequirements.
Thisplancansuithigh-performanceworkloadsrequiringmessagingresilienceandasynchronousmessagingreplication.RabbitMQcopiesmessagestodiskforresilienceandallowsasynchronousmessagingreplicationthroughtheRabbitMQFederationplug-in.
Thisplanoffers:
FastaccesstoanisolatedinstanceofRabbitMQscopedfortheapplicationteams
OrgandSpaceAdministratoraccesstotheRabbitMQManagementUIsoapplicationteamscanhavefullcontroloverthenode
Updatesandupgradesinitiatedandcontrolledbytheoperatortokeeptheinstanceup-to-datewiththelatestsecuritypatchesandbugfixes
MessageresilienceprovidedthroughRabbitMQexchange,queueFederation,andShovelplugins.
On-DemandClusterPlanLikethesinglenodeplan,thisplanisdesignedtobesimpletoconfigure,deployanduse.ItgivesapplicationteamsfastaccesstothepoweroftheleadingOpenSourcemessagebrokerbackedbyBOSHtomeetallbutthemostdemandinghighavailabilityappmessagingrequirements.
Thisplancansuithighperformanceworkloadsrequiringmessagingresilience(copiedtodisk)andasynchronousmessagingreplicationthroughtheRabbitMQFederationplugin.Withthisplan,however,youalsoscaleoutRabbitMQforPCFtomultiplenodes.
Thisplanoffers:
Fastaccesstoanisolated,clusteredinstanceofRabbitMQscopedtotheapplicationteamOrgsandSpaces
AdministratoraccesstotheRabbitMQManagementUItogiveapplicationteamsfullcontroloverthecluster
Updatesandupgradesinitiatedandcontrolledbytheoperatortokeeptheinstanceup-to-datewiththelatestsecuritypatchesandbugfixes.
MessageresilienceprovidedbymirroringqueuesacrossRabbitMQnodes,andtheoptiontousetheFederationandShovelplugins.
© Copyright Pivotal Software Inc, 2013-2018 14 1.10
GeneralPrinciplesoftheClusterPlanThefollowingaresomegeneralprinciplestobeawareofwhenconfiguringtheclusterplan:
DesignedforConsistency
RabbitMQclusteringisnotprimarilyasolutionforincreasedavailability.Instead,itisdesignedforconsistencyandpartitiontolerance,asdescribedintheCAPtheorem .RabbitMQclusteringprovidesincreasedmessageconsistencythroughqueuemirroring.Thismeansthatmessagesaccessedinonequeueareexactlythesameasinanotherqueue.Formoreinformation,seeConsistencyorAvailabilityTradeoff.
Otheroptionscanbeusedforavailabilityrequirements,suchastheuseoffederationbetweenexchangesorqueues.
ForadetaileddescriptionofdistributedRabbitMQbrokers,seetheRabbitMQdocumentation .
NumberofNodes
Everynodeintheon-demandclustermaintainsacompletedatabaseofallmetadata,andallchangestothemetadataareconfirmedbyeverynodeinthecluster.Therefore,goingbeyondsevennodescanhaveasignificantnegativeimpactonperformance.Foroptimumresilienceandperformance,Pivotalrecommendsthreenodesformostworkloads.
NetworkLatency
RabbitMQclustersareonlyrecommendedfordeploymentinlowlatencynetworks,whichnormallymeansthatitisnotadvisabletodeploytheseclustersacrossavailabilityzones(AZs).ThestabilityandperformanceoftheRabbitMQclusterisheavilyinfluencedbytheworkloadonthenodes,replicationchoices,andnetworklatency.
Forthisreason,PivotalrecommendsthatyoudeployRabbitMQclustersintoasingleOpsManagerAZ.However,wheredifferentAZsareinthesamedatacenter,withreliablelowlatencylinks,spanningAZscanbeused.
ForcloudIaaSdeployments,Pivotaldoesnotrecommendthatdeploymentsspanregions.Forexample,inAmazonWebServices(AWS)terms,deployingaRabbitMQclusteracrossAZswithinaregionshouldprovidehighenoughnetworkperformancetopreventimpactingclusterstability.However,deployingacrossAWSregionsislikelytoleadtoclusterinstability.Formoreinformation,seetheAWSdocumentation .
ConsistencyorAvailabilityTradeoffInadistributedmessagingsystem,atradeoffmustbemadebetweenavailabilityorconsistencywhenanetworkpartitioneventoccursandoneormorenodesarenotabletocommunicatewitheachother.TheclusterplanletsoperatorsdecidehowtheywanttheRabbitMQclustertoreactintheeventofanetworkpartition.
Pivotalrecommendskeepingthedefaultclusterpartitionoptionof pause_minority becausethissatisfiesmostusecases.Choosingthe pause_minority
partition-handlingstrategyfavorsmessageconsistencyoveravailability.Formoreinformationabouttheoptionsforhandlingpartitions,seetheRabbitMQdocumentation .ForadetaileddescriptionoftheoptionsavailableinRabbitMQforPCF,seeClusteringandNetworkPartitions.
Hereisanexampleofhow pause_minority works.IfyoucreateaRabbitMQclusterwiththreenodesandonenodebecomesunabletocommunicatewiththeothertwo,thisnodeisintheminority.Thenodethatisintheminorityispaused,andtheothertwonodescontinueservingtraffic.Ifeachofthenodeslosesconnectivitywiththeothertwo,thentheentireclusterispausedtopreservedatasincenomajoritycanbeestablished.Theclusterhealswhentwoormorenodesareabletocommunicatewitheachother.
RabbitMQQueueAvailabilityItisimportanttobeawarethatmessagequeueavailabilityisdifferentfromclusteravailability.So,havingclusteravailabilitydoesnotmeanthatallofthemessageswithinthequeuesarealsoavailable.
Bydefault,queueswithinaRabbitMQclusterarelocatedonasinglenode—thenodeonwhichtheywerefirstdeclared.However,queuescanbeconfiguredtomirroracrossmultiplenodes,sothatanymessagepublishedtothequeueisreplicatedtoallmirrors.Enablingmirroringcanhaveanegativeimpactonqueueperformancebecausemessagesmustbecopiedtoallmirrorsbeforebeingacknowledged.
Eachmirroredqueueconsistsofonemasterandoneormoremirrors,withtheoldestmirrorbeingpromotedtothenewmasteriftheoldmasterdisappearsforanyreason.Consumersareconnectedtothemasterregardlessofwhichnodetheyconnectto,andmirrorsdropmessagesthathavebeen
© Copyright Pivotal Software Inc, 2013-2018 15 1.10
acknowledgedatthemaster.Queuemirroringenhancesqueueavailability,butdoesnotdistributeloadacrossnodesbecauseeachoftheparticipatingnodesmuststilldoallthework.
Appdevelopersmustdecideiftheywanttousequeuemirroringanddeterminethepolicytheywanttoapplytotheirqueues.Thesechoiceshavesignificantimpactontheavailabilityoftheirqueues.Formoreinformation,seetheRabbitMQdocumentation .
Unlikethepre-provisionedplan,theclusterplandoesnotshipwithadefaultloadbalancer.Therefore,developersmustconfiguretheirapptousethearrayofhostsprovidedin VCAP_SERVICES .Ifdevelopersenablequeuemirroring,theymustalsoensuretheirappshavere-trylogicandreconnectionlogicthatiteratesovertherangeofhostsprovided.MostcommonRabbitMQclientshavethislogicbuiltintothem.Formoreinformation,seetheSpringAMQPdocumentation .
Becausetheclusterplanisdesignedtoenableapplicationteamstoself-serve,nothavingaloadbalancerinfrontoftheRabbitMQclusterhasthesebenefits:
Manageresourcesbetter,asfewerVMsareneeded.
Helpwithtroubleshooting.ClientIPisnowtheIPofthesourcecontainerandnottheHAProxy.
Reducethenumberofhopsbetweenappsandbroker.Thishelpswithlatency.
Determinequeueplacement.Thismakessenseforlargerscaledeployments.
Empowerapplicationteamstomanagetheirclusterinthebestwayfortheirapp.
Requirere-trylogicinanappifitneedsHAaccesstoaqueue.Thus,allnodescanroutetoaqueueifitisavailable.
ManagingOn-DemandResourcesThroughPlansInconfiguringeachplan,thereareanumberofoperationalcontrolsthatplatformoperationsteamscanusetomanagetheresourcesconsumedbyon-demandRabbitMQ:
ControlAccess—Operatorscanchoosetheappdevelopmentorgsandspacesforwhichtheplansareavailableandvisible.Eachplancanbeenabledordisabled,andserviceaccessandvisibilitycaneitherbeglobal,orenabledperorgandspacethroughthecommandline.
Forexample,youmaydecidetoenablethesinglenodeon-demandplanacrossallapplicationteamstomeettheirdemandtoisolatetheirworkload.Youmaythenchoosetooffertheon-demandclusterplanonlytoasubsetofapplicationteamswhorequiretheextraresources.
SetQuotas—Youcansetaglobalquotaforallon-demandinstancesthattakesprecedenceovereachplanquota.Thisletsyouguardagainsttheriskofover-committingresources,butallowstheflexibilityofover-committingeachplan,soyoucanmeetthefluctuatingdemandsofyourappdevelopers.
ControlResourceConsumption—Eachplanoffersmorefine-grainedcontroloverindividualplanresourceconsumption.Atthehighestlevel,youcanusetheplanquotatocontrolthenumberofinstancesthatcanbedeployedwithinafoundation.Foreachplan,youcanalsoconfigurethenumberofnodesthatconstituteacluster(3,5,or7),theinstancetype,andpersistentdiskstoragesizetobestsuityourrequirements.
Monitor—Youcanmonitorthenumberofinstancesthathavebeendeployedagainstthequotayouhavesetsothatyoucanplanfutureresourcerequirements.
CustomizingPlanOptionsTheRabbitMQforPCFon-demandplansexposeanumberofconfigurationoptions.Inmostcases,thedefaultconfigurationsmeetmostappdemands.However,itisimportantforanoperationsteamtoconsidertheoptionstoensurethattheyprovidethebestservicetotheirappdevelopers.Thissectionexplainstheseoptions.
ConfigurationOptions
SingleNodeandClusterPlansEnable/Disableplan
Determinewhichorgsandspacescanseeandaccesstheplan
SetServiceInstanceQuota
SelectAZplacement(whereapplicable)
SetRabbitMQinstancesize(CPUandMemory)
Setpersistentdisksize(PersistedMessageStore)fortheRabbitMQinstance.Ensurethesizeofthepersistentdiskisatleasttwiceaslargeasthe
© Copyright Pivotal Software Inc, 2013-2018 16 1.10
instancememory.
ClusterPlanOnlySetnumberofnodesto3,5,or7
Determinenetworkpartitionbehavior.SeeConsistencyorAvailabilityTradeoffabove.
ThingsThatArePreconfiguredThefollowingarepre-configuredforboththesinglenodeandtheclusterplans:
Metrics—EmittedtotheLoggregatorFirehoseforallon-demandinstances.ThepollingintervalissetintheOpsManager,intheMetricspollingintervalfield,intheRabbitMQtaboftheRabbitMQforPCFtile.Duetotheimpactofsomeoftheclustersettingsdetailedbelow,PivotalstronglyrecommendsthatyoumonitortheexposedmetricsandconfigurealarmsasrecommendedinMonitoringandKPIsforOn-DemandRabbitMQforPCF.SeealsoMonitoringOn-DemandRabbitMQClustersbelow.
Logs—RabbitMQon-demandinstancelogsareforwardedusingthesameconfigurationascontainedintheSyslogtaboftheRabbitMQforPCFtile.
Diskfreespacelimit—Thediskfreespacelimitissetto150%ofRAMoftheinstancetypeyouselect.Forexample,ifyouselectaninstancetypewith10GBofRAM,thediskfreespacelimitissetto15GB.Acluster-widealarmistriggerediftheamountoffreediskspacedropsbelowthis,andallpublishersareblocked.InstancesmustbeconfiguredtohavepersistentdisksthatareatleasttwicethesizeofinstanceRAM.Formoreinformation,seetheRabbitMQdocumentation .
Memorythresholdfortriggeringflowcontrol—Thresholdatwhichflowcontrolistriggeredissetto40%oftheinstanceRAM.Thismeansthatwhenthealarmistriggered,allconnectionspublishingmessagesareblockedcluster-wideuntilthealarmiscleared.
Forexample,ifyouselectaninstancetypewith10GBofRAM,whenmorethan4GBofmemoryisused,allpublishingconnectionsareblocked.Formoreinformation,seeMemoryAlarms intheRabbitMQdocumentation.
Memorypagingthreshold—ThisisthelevelatwhichRabbitMQtriestofreeupmemorybyinstructingqueuestopagetheircontentsouttodisk.Thisisdonetotrytoavoidreachingthehighwatermarkandblockingpublishers.Thisthresholdissetto50%oftheconfiguredhighwatermark,whichis20%ofconfiguredmemory.Formoreinformationonmemorycalculation,seeChangestoMemoryAllocationwhenUpgrading .
Forexample,ifyouselectaninstancetypewith10GBofRAM,whenmorethan2GBofmemoryisused,allqueuesstartwritingallqueuecontentstodisk.Formoreinformation,seetheRabbitMQdocumentation .
MonitoringOn-DemandRabbitMQClustersItisimportanttomonitorandcomparethenumberofinstancesthathavebeendeployedagainstthequotayousetviathemetricexposedontheFirehose.
Eachinstanceispre-configuredtoemitmetricstotheFirehoseandcanbeidentifiedbythe deployment tag,whichhastheserviceinstanceID.ItisimportanttomonitorthesemetricsasrecommendedinMonitoringandKPIsforOn-DemandRabbitMQforPCF.
Note:Aloadbalancer,suchasHAProxy,isnotdeployedwithon-demandclusterplans.
© Copyright Pivotal Software Inc, 2013-2018 17 1.10
On-DemandServiceArchitectureThistopicdescribesthearchitectureforon-demandRabbitMQ®forPivotalCloudFoundry(PCF).
Forinformationaboutarchitectureoftheolder,pre-provisionedservice,seeDeployingtheRabbitMQ®Service.
BOSH2.0andServiceNetworksWhenyoudeployPCF,youmustcreateastaticallydefinednetworktohostthecomponentvirtualmachinesthatconstitutethePCFinfrastructure.
PCFcomponents,liketheCloudControllerandUAA,runonthisinfrastructurenetwork.InPCFv2.0andearlier,on-demandPCFservicesrequirethatyouhostthemonanetworkthatrunsseparatelyfromthisnetwork.
Cloudoperatorspre-provisionserviceinstancesfromOpsManager.Then,foreachservice,OpsManagerallocatesandrecoversstaticIPaddressesfromapre-definedblockofaddresses.
Toenableon-demandservicesinPCFv2.0andearlier,operatorsmustcreateaservicenetworksinOpsManagerDirectorandselecttheServiceNetworkcheckbox.Operatorsthencanselecttheservicenetworktohoston-demandserviceinstanceswhentheyconfigurethetileforthatservice.
DefaultNetworkandServiceNetworkOn-demandPCFservicesrelyontheBOSH2.0abilitytodynamicallydeployVMsinadedicatednetwork.Theon-demandservicebrokerusesthiscapabilitytocreatesingle-tenantserviceinstancesinadedicatedservicenetwork.
On-demandservicesusethedynamically-provisionedservicenetworktohostthesingle-tenantworkerVMsthatrunasserviceinstanceswithindevelopmentspaces.ThisarchitectureletsdevelopersprovisionIaaSresourcesfortheirserviceinstancesatcreationtime,ratherthantheoperatorpre-provisioningafixedquantityofIaaSresourceswhentheydeploytheservicebroker.
Bymakingservicessingle-tenant,whereeachinstancerunsonadedicatedVMratherthansharingVMswithunrelatedprocesses,on-demandserviceseliminatethe“noisyneighbor”problemwhenoneapplicationhogsresourcesonasharedcluster.Single-tenantservicescanalsosupportregulatorycompliancewheresensitivedatamustbecompartmentalizedacrossseparatemachines.
Anon-demandservicesplitsitsoperationsbetweenthedefaultnetworkandtheservicenetwork.Sharedcomponentsoftheservice,suchasexecutivecontrollersanddatabases,runcentrallyonthedefaultnetworkalongwiththeCloudController,UAA,andotherPCFcomponents.Theworkerpooldeployedtospecificspacesrunsontheservicenetwork.
ThediagrambelowshowsworkerVMsinanon-demandserviceinstancerunningonaseparateservicesnetwork,whileothercomponentsrunonthedefaultnetwork.
© Copyright Pivotal Software Inc, 2013-2018 18 1.10
RequiredNetworkingRulesforOn-DemandServicesPriortodeployinganyservicetilethatusestheon-demandbroker(ODB),theoperatormustrequestthenetworkconnectionsneededtoallowvariouscomponentsofPivotalCloudFoundry(PCF)tocommunicatewithODB.ThespecificsofhowtoopenthoseconnectionsvariesforeachIaaS.
Thefollowingtableshowstheresponsibilitiesofthekeycomponentsinanon-demandarchitecture.
KeyComponents TheirResponsibility
BOSHDirector
CreatesandupdatesserviceinstancesasinstructedbyODB
BOSHAgentBOSHincludesanAgentoneveryVMthatitdeploys.TheAgentlistensforinstructionsfromtheDirectorandcarriesoutthoseinstructions.TheAgentreceivesjobspecificationsfromtheDirectorandusesthemtoassignarole,orJob,totheVM.
BOSHUAA AsanOAuth2provider,BOSHUAAissuestokensforclientstousewhentheyactonbehalfofBOSHusers.
ERT Containstheappsthatareconsumingservices
ODB InstructsBOSHtocreateandupdateservices,andconnectstoservicestocreatebindings
Deployedserviceinstance
Runsthegivendataservice(forexample,thedeployedRedisforPCFserviceinstancerunstheRedisforPCFdataservice)
Regardlessofthespecificnetworklayout,theoperatormustensurenetworkrulesaresetupsothatconnectionsareopenasdescribedinthetablebelow.
Thiscomponent…
Mustcommunicatewith…
DefaultTCPPort Communicationdirection(s) Notes
ODB
BOSHDirector
BOSHUAA
25555
8443 One-way Thedefaultportsarenotconfigurable.
ODBDeployedserviceinstances
15672(RabbitMQManagementUI)
One-wayThisconnectionisforadministrativetasks.Avoidopeninggeneraluse,app-specificportsforthisconnection.
ODB ERT 8443 One-way Thedefaultportisnotconfigurable.
ErrandVMs
ERT
ODB
DeployedServiceInstances
8443
8080
15672(RabbitMQManagementUI)
5671-2(AMQP/AMQPS)
One-way Thedefaultportisnotconfigurable.
BOSHAgent BOSHDirector 4222 Two-way
TheBOSHAgentrunsoneveryVMinthesystem,includingtheBOSHDirectorVM.TheBOSHAgentinitiatestheconnectionwiththeBOSHDirector.Thedefaultportisnotconfigurable.
DeployedappsonERT
Deployedserviceinstances
15672(RabbitMQManagementUI)
5671-2(AMQP/AMQPS)
61613-4(STOMP/STOMPS)
1883,8883(MQTT/MQTTS)
One-way Thisconnectionisforgeneraluse,app-specifictasks.Avoidopeningadministrativeportsforthisconnection.
ERT ODB 8080 One-wayThisportmaybedifferentforindividualservices.Thisportmayalsobeconfigurablebytheoperatorifallowedbythetiledeveloper.
© Copyright Pivotal Software Inc, 2013-2018 19 1.10
© Copyright Pivotal Software Inc, 2013-2018 20 1.10
InstallingandConfiguringtheOn-DemandServiceThistopicprovidesinstructionstoPivotalCloudFoundry(PCF)operatorsabouthowtoinstall,configure,anddeploytheRabbitMQforPCFtiletoprovideon-demandservice.
TheRabbitMQopensourceproductprovidesadditionaldocumentation.FormoreinformationaboutgettingstartedwithRabbitMQandensuringproductionreadiness,seetheProductionChecklistintheRabbitMQDocumentation .
PrerequisitesforDeployingtheOn-DemandServiceBeforedeployingRabbitMQforPCFasanon-demandservice,youmustensurethattherequirednetworkrulesareinplacetoallowvariouscomponentstocommunicate.
SeeRequiredNetworkingRulesforOn-DemandServicesfordetailsonthenetworkconnectionsthatmustbeopentoenabletheon-demandservice.
Forinformationabouttheon-demandservicearchitecture,seeOn-DemandServiceArchitecture.
DownloadandInstallRabbitMQforPCF1. DownloadtheproductfilefromPivotalNetwork .
2. NavigatetotheOpsManagerInstallationDashboardandclickImportaProducttouploadtheproductfile.
3. UndertheImportaProductbutton,click+nexttotheversionnumberofRabbitMQforPCF.Thisaddsthetiletoyourstagingarea.
4. ClickthenewlyaddedRabbitMQforPCFtile.Thisletsyoubeginconfiguringthetile.Theinstallationiscompletewhenyouapplythechangesfromtheconfiguration.
ConfigureOn-DemandRabbitMQforPCFTheconfigurationscreenbelowappearswhenyouclicktheRabbitMQforPCFtileinOpsManager.Anorangecirclebesideatabindicatesthatyoumustcompleteaconfigurationinthetab.Agreencheckmarkindicatesthatthetabispreconfiguredandyoumayoptionallychangeitssettings.
Note:Forinstructionsonhowtoinstall,configure,anddeploytheRabbitMQforPCFtileasapre-provisionedservice,seeInstallingandConfiguringthePre-ProvisionedService.
© Copyright Pivotal Software Inc, 2013-2018 21 1.10
WhichSettingsTabstoConfigurefortheOn-DemandServiceConfigurethefollowingtabsfortheon-demandservice:
RabbitMQSettingsTab Instructions
AssignAZsandNetworks ConfigureAZsandNetworks
RabbitMQ SettheMetricspollingintervalonly.SeeWhatAreMetrics.
Syslog SetupSyslogForwarding
DedicatedInstance:GlobalSettings ConfigureGlobalSettings
DedicatedInstance:SingleNodePlans ConfiguretheServicePlan
DedicatedInstance:ClusterPlan ConfiguretheServicePlan
Errands Errands
Stemcell VerifytheStemcell
ConfigureAZsandNetworksFollowthestepsbelowtoconfiguretheAZsandnetworks.
1. ClickAssignAZsandNetworks.
2. ConfigurethefieldsontheAssignAZsandNetworksasfollows:
Field Instructions
Important:YoucannotchangetheregionsornetworksafteryouhaveclickedApplyChangesintheApplyChangesfromYourConfigurationbelow.
© Copyright Pivotal Software Inc, 2013-2018 22 1.10
Placesingletonjobsin
SelecttheregionthatyouwantforsingletonVMs.PCFcreatestheRabbitMQbrokerinthisAZ.
Balanceotherjobsin
Selectadditionalregion.Thisselectiondoesnotaffecttheon-demandRabbitMQforPCFservice.
Network
SelectanetworkfortheRabbitMQOn-DemandBroker.
ThisshouldbeaseparatenetworkfromtheoneyouselectforServiceNetwork.FormoreinformationabouttheDefaultNetwork,seeDefaultNetworkandServiceNetwork.
Typically,youselectthenetworkusedforthePivotalElasticRuntimecomponents.
ServiceNetwork
Selectaseparatenetworkthattheon-demandserviceinstancesrunon.
Atypicalpracticeistoputallon-demandservicesonasinglenetwork,separatefromthenetworkthatERTandtheOn-DemandBrokerrunon.ForinformationabouttheServiceNetwork,seeDefaultNetworkandServiceNetwork.
Thisfieldisalsorequiredforthepre-provisionedservice,thoughinthatcase,itdoesn’tmatterwhichnetworkyouselect.
3. ClickSave.
SettheMetricsPollingIntervalOntheRabbitMQtab,setametricspollinginterval.Formoreinformation,seeWhatareMetrics.
ConfigureLoggingtoMonitorRabbitMQforPCFPivotalrecommendsthatyouconfigureloggingtomonitorthehealthofRabbitMQforPCF.FollowSetUpSyslogForwardingtoconfigurelogging.
ConfigureGlobalSettingsFollowthestepsbelowtoconfigureglobalsettings.
1. ClickDedicatedInstance:GlobalSettingsandconfigurethefollowing:
Serviceinstancequotamin:0,max:50setthetotalnumberofdedicatedserviceinstanceswhichcanbedeployed.Formoreinformation,seeSettingLimitsforOn-DemandServiceInstances.
VMoptions:
Allowoutboundinternetaccess(IaaS-dependent).Thischeckboxmustbetickedtoallowexternallogforwarding,sendingbackupartifactstoexternaldestinations,andcommunicatingwithanexternalBOSHblobstore.
2. ClickSave.
ConfiguretheServicePlanInordertoenabletheon-demandserviceplan(s),youmustenableoneorbothoftheseplans:
DedicatedInstance:SingleNodePlan—ConfiguresasinglenodeofRabbitMQServer.Youmustcompletetherequiredfieldsonthistabevenifyoudisableaccesstothisplan(seeCFServiceAccessinthetablebelow).
DedicatedInstance:ClusterPlan—ConfiguresaclusterwiththreetosevenRabbitMQServernodes.Thisplanisdisabledbydefault.
WARNING:ChangingtheNetworkorServiceNetworkafteryouhaveconfiguredthem,orchangingtheirIPconfigurations,resultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .
Note:Iftheon-demandserviceisnotenabled,theon-demandbrokerisdeployedalongsidetheRabbitMQinstallation,butitisnotavailablein
© Copyright Pivotal Software Inc, 2013-2018 23 1.10
1. Choosethededicatedserviceinstanceyouwanttoconfigure:
SingleNodePlan(completerequiredfieldsevenifyoudisablethisplan):
ClusterPlan:
theMarketplace.
© Copyright Pivotal Software Inc, 2013-2018 24 1.10
2. Configurethefieldsasfollows:
Field Instructions
Enablethisplan (ClusterPlanOnly)Toenable,selectPlanEnable.
CFServiceAccess
Enableordisableaccesstothisplan,orleaveaccessunchanged.
Ifyouenabletheclusterplan,thedefaultsettingisEnableServiceAccess.Ifyouchangethisdefaultsetting,thesmoketestsfail.Therefore,settheDedicatedInstanceSmokeTestserrandtoOffifyouenabletheclusterplanandchangethisdefault.Formoreinformation,seeErrands.
EnableServiceAccess—Givesaccesstoallorgs,anddisplaystheserviceplantoalldevelopersintheMarketplace.DisableServiceAccess—Disablesaccesstoallorgs,andhidestheserviceplantoalldevelopersintheMarketplace.ThissettingcannotbeselectedatalatertimeintheUI.LeaveServiceAccessUnchanged—Keepsanyexistingaccesssettings,andonlydisplaystheserviceplanintheMarketplacetomembersoforgsthathaveaccesstotheplan.YoucanchangetheaccesssettingslaterusingthecfCLI.Forinstructions,seeControllingAccesstoServicePlansbyOrg.
Planname Acceptthedefaultorenteraname.ThisisthenamethatappearsintheMarketplace.
Plandescription
Acceptthedefaultorenteradescription.
Serviceinstancequota
Enterthemaximumnumberofon-demandserviceinstancesthatcanbeavailableatonetime.Formoreinformation,seeSettingLimitsforOn-DemandServiceInstances.
PlanAcceptthedefaultorenterinformationabouttheplantohelpyourappdevelopers.
© Copyright Pivotal Software Inc, 2013-2018 25 1.10
featuresNumberofnodes
(ClusterPlanOnly)Enter3,5or7.Thissettingonlyaffectsnewserviceinstances.Previouslydeployedserviceinstancesarenotupdated.
Networkpartitionbehaviour
(ClusterPlanOnly)Selectpauseminorityorautoheal.Pivotalrecommendsusingpauseminority.Formoreinformation,seeConsistencyorAvailabilityTradeoff.
RabbitMQVMType
SelectalargeVMtype.Theplancreatesaserviceinstanceofthissize.
Persistentdisktype
ThisiswhereRabbitMQpagesmessagestodisk.ServiceinstancedeploymentsfailifthisvalueislessthantwicethevolumeofRAMoftheselectedRabbitMQVMType.Formoreinformation,seeResourceTypesandDiskSizeConcernsbelow.
AZplacement
ThisfieldisavailableafteryoucompletetheAssignAZsandNetworkspage.
Forasingle-nodeplan,selectoneormoreAZs.Foraclusterplan,selectonlyoneAZ.PivotalrecommendsdeployingaclusterplantoonlyoneAZtominimizerisksduetonetworklatencyandpartitions.SeeNetworkLatencyandConsistencyorAvailabilityTradeofffordetails.
Ifyouchangethisselectionafterdeployment,existinginstancesarenotaffectedbythechange.SeeDeterminewhichAZsaServiceInstanceUses.
3. ClickSave.
DeterminewhichAZsaServiceInstanceUses
TodeterminewhichAZsaserviceinstanceisplacedin,dooneofthefollowing:
RetrievetheserviceGUIDusingthe cf service SERVICE_INSTANCE --guid commandandthenruntheBOSH instances commandfortheservice-instance_GUID deployment.
Withsyslogforwardingenabled,inspecttheservicebrokerlogswhenrunningtheUpgradeAllServiceInstanceserrand.Foreachexistingserviceinstance,thelogmessageincludestheserviceinstanceGUIDandtheAZstheserviceinstanceisrunningin.
ResourceTypesandDiskSizeConcerns
ItispossibletoconfiguretheVMtypeforRabbitMQandthesizeofthepersistentdiskthatisgoingtobeattachedtotheRabbitMQinstances.SuggestedvalueistwicetheamountofRAMoftheselectedVMtype.Formoreinformation,seethePivotalRabbitMQ documentation.
RabbitMQraisesalarmswhendiskspacedropsbelowtheconfiguredlimit.Incorrectdisksizesmightcausethedeployedinstancenottostart.RabbitMQdeclinestostartifthereisnotenoughspaceavailableaccordingtothethreshold.
Important:IfyouchangethisconfigurationafteryouhaveselectedAZsanddeployedserviceinstances,existinginstancesarenotplacedinthenewlyconfiguredAZswhentheUpgradeAllServiceInstanceserrandisrun.Thispreventsre-creationoftheVMsindifferentAZs,whichcanleadtodataloss.Allnewserviceinstances,however,willbecreatedinthenewlyconfiguredAZs.
© Copyright Pivotal Software Inc, 2013-2018 26 1.10
On-demandinstancesareconfiguredwithathresholdsettothe150%ofthememory(RAM)oftheVM.Usethefollowingtableasaguidewhenselectingthesizeofthepersistentdisk.
ThefollowingtableshowsanexampleofpossibleRAMvalues,absoluteminimalvaluebelowwhichRabbitMQdeclinestostart,andthedisksizesuggestedforanaverageusecase.
RAM Freediskalarmthreshold(1.5xRAM) Suggesteddisksize(2xRAM)
10GB 15GB 20GB
16GB 24GB 32GB
32GB 48GB 64GB
Formoreinformation,seethefollowing:
MemoryandDiskAlarms
DiskAlarms
VerifytheStemcell1. ClickStemcell.
2. Verifyand,ifnecessary,importanewstemcellversion.Formoreinformation,seetheinformationaboutimportingthestemcellforyourIaaS:AWS,Azure ,GCP ,orvSphere .
ApplyChangesfromYourConfigurationYourinstallationisnotcompleteuntilyouapplyyourconfigurationchanges.Followthestepsbelow:
1. ReturntotheOpsManagerInstallationDashboard.
2. ClickApplyChanges.
ErrandsWhendeployingorupdatingRabbitMQforPCF,OpsManagercanoptionallyrunaseriesofPost-DeployErrands.Anexampleisthe Smoke
Testserrand,
whichchecksthehealthoftheRabbitMQclusterafteradeployorupgrade.
Youcandecidewhethertorunerrandsbytogglingthemonoroffbeforeanupdate.Thisisaone-timesettingontheinstallationdashboard:
However,ifnecessary,youcanchangethesedefaultsbyclickingErrandsintheRabbitMQforPCFSettingstab.
Important:AsofRabbitMQforPCFv1.9.0,allpost-deployerrandsareonbydefault.Pivotalrecommendskeepingthesedefaults,becausethesmoketestscanencounterunexpectedissues,andon-demandinstancesofRabbitMQforPCFmayfallbehindiftheUpgradeAllServiceInstanceserrandisnotonbydefault.
© Copyright Pivotal Software Inc, 2013-2018 27 1.10
Formoreinformationonerrandrunrules,seeErrandRunRules .
Post-DeployErrands
Errand Description
BrokerRegistrar
Makesthepre-provisionedRabbitMQserviceplansavailableintheMarketplace
SmokeTests
Checksthatapre-provisionedRabbitMQserviceinstancecanbeboundtoaCloudFoundryapp,andthattheappcanpublishandsubscribetoaRabbitMQcluster.SeePre-ProvisionedInstanceSmokeTests.
RegisterOn-DemandServiceBroker
Makestheon-demandRabbitMQserviceplansavailableintheMarketplace.IfyouchangetheServicePlanConfiguration,youmustrunthiserrandinorderforthechangestobereflectedintheMarketplace.
DedicatedInstanceSmokeTests
Checksthaton-demandRabbitMQserviceinstancescanbeboundtoaCloudFoundryapp,andthattheappcanpublishandsubscribetoaRabbitMQcluster.SeeDedicatedInstanceSmokeTestsbelow.
UpgradeAllServiceInstances
On-demandinstancesareupdatedandredeployediftherearechangestotheDedicatedInstancesettingsorthetileisupgraded.Ifthiserrandissetto“Off”or“WhenChanged”,updatestoDedicatedInstancesettingswillnotbeappliedtoexistingserviceinstances.Pivotalstronglyrecommendsthatthiserrandisconfiguredtoalwaysrun.
Pre-DeleteErrands
Errand Description
BrokerDeregistrarRemovesthepre-provisionedRabbitMQservicefromtheMarketplaceanddeletesallassociatedserviceinstancesandbindings
DeleteAllServiceInstancesUnbindsanddeletesexistingon-demandserviceinstances.Thedurationofthiserranddependsonthenumberofdeployedon-demandinstances.
DeregisterOn-DemandServiceBroker
Removestheon-demandRabbitMQservicefromtheMarketplace
DedicatedInstanceSmokeTestsSmoketestsonlyrunagainsttheclusterplan.Formoreinformationaboutthesmoketestsprocess,seeSmokeTests.
CreateanAdminUserforaServiceInstanceIfyouwanttogiveappdevelopersadminprivilegestotheRabbitMQManagementUI,youcancreateanadminuserforaserviceinstanceandsharetheusercredentialswithappdevelopers.
Bothoperatorsandappdeveloperscanusethisprocedure.
TocreateanadminuseronaRabbitMQinstancedothefollowing:
1. Runthiscommandtocreateaservicekey:
cf create-service-key SERVICE_INSTANCE SERVICE_KEY -c '{"tags": "administrator"}'
where:
SERVICE_INSTANCE isthenameyousuppliedwhenyouran cf create-service .SERVICE_KEY isanameyouchoosetoidentifytheservicekey.
Forexample:
© Copyright Pivotal Software Inc, 2013-2018 28 1.10
$ cf create-service-key my-instance my-admin-key -c '{"tags":"administrator"}'
Creating service key my-admin-key for service instance my-instance as [email protected]
2. Runthiscommandtogettheadminusercredentials:
cf service-key SERVICE_INSTANCE SERVICE_KEY wherethevariablesarethesameasabove.
ThisreturnsaDashboardURLcontainingtheadmincredentials,whichcanbeusedtoaccessthemanagementUI.Forexample:
$ cf service-key my-instance my-admin-key
Getting key my-admin-key for service instance my-instance as [email protected]...{ "dashboard_url": "https://my-instance.bosh-lite.com/#/login/admin-username/admin-password", "username": "admin-username", "password": "admin-password", ...}
RabbitMQServerSettingsThatCannotBeDisabledThefollowingpluginsareenabledbydefaultandcannotbedisabled:
rabbitmq_management
rabbitmq_mqtt
rabbitmq_stomp
rabbitmq_federation
rabbitmq_federation_management
rabbitmq_shovel
rabbitmq_shovel_management
© Copyright Pivotal Software Inc, 2013-2018 29 1.10
SmokeTestsRabbitMQforPCFrunsasetofsmoketestsduringinstallationtoconfirmsystemhealth.
SmokeTestStepsThesmoketestsperformthefollowingforeachavailableserviceplan:
1. Targetstheorg system andcreatesaspacetorunthetests.
2. DeploysaninstanceoftheCFRabbitMQExampleApp tothisspace
3. CreatesaRabbitMQserviceinstanceandbindsittotheCFRabbitMQExampleApp
4. ChecksthattheCFRabbitMQExampleAppcanwritetoandreadfromtheRabbitMQserviceinstance
5. Cleansupalldeployedapplicationandallitsservicebindings.Finally,thecfspaceisdeleted.
TroubleshootingIferrorsoccurwhilethesmoketestsrun,theyaresummarizedattheendoftheerrandlogoutput.Detailedlogscanbefoundwherethefailureoccurs.
Whenencounteringanerrorwhenrunningsmoketests,itcanbehelpfultosearchthelogforotherinstancesoftheerrorsummaryprintedattheendofthetests,forexample, Failed to target Cloud Foundry .Lookoutfor TIP: ... inthelogsnexttoanyerroroutputforfurthertroubleshootinghints.
Note:Smoketestsfailunlessyouenableglobaldefaultapplicationsecuritygroups(ASGs).YoucanenableglobaldefaultASGsbybindingtheASGtothe system orgwithoutspecifyingaspace.ToenableglobaldefaultASGs,use cf bind-running-security-
groupp>
© Copyright Pivotal Software Inc, 2013-2018 30 1.10
UpgradingRabbitMQforPCFThisproductenablesautomatedupgradesbetweenversionsoftheproductandisdeployedthroughOpsManager.Insomecases,youmightberequiredtotaketheclusteroffline.Whenthisisnecessary,itisclearlynotedinthereleasenotesforthatversion.
TheupgradepathsforeachversionaredetailedatPivotalNetwork-RabbitMQforPCFpage .
DowntimeWhenUpgradingAguidefordowntimeduringupgradedeploymentsisshowninthetablebelow.Insomecases,theclusterremainsavailableduringatileupgrade,butindividualqueuesonclusternodesmaybetakenoffline.
Thisisonlyaguide,sobeforeupgrading,checkthereleasenotesfortheversionyouareupgradingto.
UpgradeType WillDowntimeBeRequiredForThisUpgrade/Update
MajorTileVersion TheRabbitMQclusteristakenofflineforthedurationoftheupgrade.
MinorTileVersion TheRabbitMQclusteristakenofflineforthedurationoftheupgrade.
PatchTileVersion
Normallythesearerollingdeploymentswitheachnodebeingupdatedinturn.Inthesecasestheclusterremainsavailable,butindividualqueuesmaybetakenofflineaseachnodeisrestarted.Therearespecificmigrationpathsthatrequiredowntime,whichareidentifiedinthereleasenotesforthatversion.
Stemcell-OnlyPatchTileVersion
Wherethepatchupdateisonlyanewstemcellversionthesearerollingdeploymentswitheachnodebeingupdatedinturn.Inthesecasestheclusterremainsavailable,butindividualqueuesmaybetakenofflineaseachnodeisrestarted.
NotesontheUpgradeProcessReviewthefollowingbeforestartinganupgradeofRabbitMQforPCF:
Upgradingtoanewerversionoftheproductdoesnotcauseanylossofdataorconfiguration.
ItmaytakebusyRabbitMQnodesalongtimetoshutdownduringtheupgradeandyoumustnotinterruptthisprocess.
ThebenefityougetfromstemcellrollingupgradesdependsonhowyouhaveconfigurednetworkpartitionhandlingandtheResourceConfigtab.AnHAProxyinstancecountof2andaRabbitMQnodecountof3arerequiredforrollingstemcellupgrades.Asofv1.7.7,thesecountsarethedefault.Formoreinformation,seeClusteringandNetworkPartitions.
ThelengthofthedowntimedependsonwhetherthereisastemcellupdatetoreplacetheoperatingsystemimageoriftheexistingVMcanjusthavetheRabbitMQsoftwareupdated.StemcellupdatesincuradditionaldowntimewhiletheIaaScreatesthenewVM.
TheclusterbecomesunavailableonlywhenupgradingbetweenspecificversionsofErlangorRabbitMQ.Thisisclearlystatedinthereleasenotesforthoseversions.
OpsManagerensurestheinstancesareupdatedwiththenewpackagesandanyconfigurationchangesareappliedautomatically.
ForissueswithupgradingRabbitMQforPCF,seeTroubleshootingOn-DemandRabbitMQforPCF.
BeforeUpgradingtoaNewerVersionofRabbitMQorErlangReviewthefollowingbeforestartinganupgradethatincludesanewversionofRabbitMQorErlang:
EnsuretheclusterishealthyviatheRabbitMQManagementUI.YoucannotrelyontheBOSH instances output,thatreflectsthestateofErlangVM,notRabbitMQ.
StopRabbitMQwhenupgradingtheRabbitMQorErlangVMversion.
UpgradeRabbitMQforPCFToupgradetheproduct,followthesesteps:
1. DownloadthelatestversionoftheproductfromPivotalNetwork .
© Copyright Pivotal Software Inc, 2013-2018 31 1.10
2. Uploadthenew.pivotalfiletoOpsManager.
3. Uploadthestemcellassociatedwiththeupdate(ifrequired).
4. Updateanynewmandatoryconfigurationparameters(ifrequired).
5. ClickApplychangesintheOpsManagerInstallationDashboard.Therestoftheprocessisautomated.
ReleasePolicyWhenanewversionofRabbitMQisreleased,anewversionofRabbitMQforPCFisreleasedsoonafter.
WherethereisanewversionofRabbitMQoranotherdependentsoftwarecomponent,suchasthestemcellreleasedduetoacriticalCVE,Pivotal’sgoalistoreleaseanewversionoftheproductwithin48hours.
© Copyright Pivotal Software Inc, 2013-2018 32 1.10
SettingLimitsforOn-DemandServiceInstancesOn-demandprovisioningisintendedtoaccelerateappdevelopmentbyeliminatingtheneedfordevelopmentteamstorequestandwaitforoperatorstocreateaserviceinstance.However,tocontrolcosts,operationsteamsandadministratorsmustensureresponsibleuseofresources.
Thereareseveralwaystocontroltheprovisioningofon-demandserviceinstancesbysettingvariousquotasattheselevels:
Global
Plan
Org
Space
Afteryousetquotas,youcan:
ViewCurrentOrgandSpace-levelQuotas
MonitorQuotaUseandServiceInstanceCount
CalculateResourceCostsforOn-DemandPlans
CreateGlobal-levelQuotasEachPivotalCloudFoundry(PCF)servicehasaseparateservicebroker.Aglobalquotaattheservicelevelsetsthemaximumnumberofserviceinstancesthatcanbecreatedbyagivenservicebroker.Ifaservicehasmorethanoneplan,thenthenumberofserviceinstancesforallplanscombinedcannotexceedtheglobalquotafortheservice.
TheoperatorsetsaglobalquotaforeachPCFserviceindependently.Forexample,ifyouhaveRedisforPCFandRabbitMQforPCF,youmustsetaseparateglobalservicequotaforeachofthem.
Whentheglobalquotaisreachedforaservice,nomoreinstancesofthatservicecanbecreatedunlessthequotaisincreased,orsomeinstancesofthatservicearedeleted.
TheglobalquotaissetintheservicetileinOpsManager,shownforanexampleservicebelow.
Note:Thisisanexampleimageonly.Thefollowingscreenmaylookslightlydifferentforyourserviceorreleaseversion.
© Copyright Pivotal Software Inc, 2013-2018 33 1.10
CreatePlan-levelQuotasAservicemayofferoneormoreplans.Youcansetaseparatequotaperplansothatinstancesofthatplancannotexceedtheplanquota.Foraservicewithmultipleplans,thetotalnumberofinstancescreatedforallplanscombinedcannotexceedtheglobalquotafortheservice.
Whentheplanquotaisreached,nomoreinstancesofthatplancanbecreatedunlesstheplanquotaisincreasedorsomeinstancesofthatplanaredeleted.
TheplanquotaissetintheservicetileinOpsManager,shownforanexampleserviceplanbelow.
Note:Thisisanexampleimageonly.Thefollowingscreenmaylookslightlydifferentforyourserviceorreleaseversion.
© Copyright Pivotal Software Inc, 2013-2018 34 1.10
CreateandSetOrg-levelQuotasAnorg-levelquotaappliestoallPCFservicesandsetsthemaximumnumberofserviceinstancesanorganizationcancreatewithinPCF.Forexample,ifyousetyourorg-levelquotato100,developerscancreateupto100serviceinstancesinthatorgusinganycombinationofPCFservices.
Whenthisquotaismet,nomoreserviceinstancesofanykindcanbecreatedintheorgunlessthequotaisincreasedorsomeserviceinstancesaredeleted.
Tocreateandsetanorg-levelquota,dothefollowing:
1. Runthiscommandtocreateaquotaforserviceinstancesattheorglevel:
cf create-quota QUOTA-NAME -m TOTAL-MEMORY -i INSTANCE-MEMORY -r ROUTES -s SERVICE-INSTANCES --allow-paid-service-plans
wherethesevariablesare:
QUOTA-NAME —AnameforthisquotaTOTAL-MEMORY —MaximummemoryusedbyallserviceinstancescombinedINSTANCE-MEMORY —MaximummemoryusedbyanysingleserviceinstanceROUTES —MaximumnumberofroutesallowedforallserviceinstancescombinedSERVICE-INSTANCES —Maximumnumberofserviceinstancesallowedfortheorg
Forexample:cf create-quota myquota -m 1024mb -i 16gb -r 30 -s 50 --allow-paid-service-plans
© Copyright Pivotal Software Inc, 2013-2018 35 1.10
2. Associatethequotayoucreatedabovewithaspecificorgbyrunningthefollowingcommand:
cf set-quota ORG-NAME QUOTA-NAME
Forexample: cf set-quota dev_org myquota
Formoreinformationonmanagingorg-levelquotas,seeCreatingandModifyingQuotaPlans .
CreateandSetSpace-levelQuotasAspace-levelservicequotaappliestoallPCFservicesandsetsthemaximumnumberofserviceinstancesthatcanbecreatedwithinagivenspaceinPCF.Forexample,ifyousetyourspace-levelquotato100,developerscancreateupto100serviceinstancesinthatspaceusinganycombinationofPCFservices.
Whenthisquotaismet,nomoreserviceinstancesofanykindcanbecreatedinthespaceunlessthequotaisupdatedorsomeserviceinstancesaredeleted.
Tocreateandsetaspace-levelquota,dothefollowing:
1. Runthefollowingcommandtocreatethequota:
cf create-space-quota QUOTA -m TOTAL-MEMORY -i INSTANCE-MEMORY -r ROUTES -s SERVICE-INSTANCES --allow-paid-service-plans
wherethesevariablesare:
QUOTA-NAME —AnameforthisquotaTOTAL-MEMORY —MaximummemoryusedbyallserviceinstancescombinedINSTANCE-MEMORY —MaximummemoryusedbyanysingleserviceinstanceROUTES —MaximumnumberofroutesallowedforallserviceinstancescombinedSERVICE-INSTANCES —Maximumnumberofserviceinstancesallowedfortheorg
Forexample: cf create-space-quota myspacequota -m 1024mb -i 16gb -r 30 -s 50 --allow-paid-service-plans
2. Associatethequotayoucreatedabovewithaspecificspacebyrunningthefollowingcommand:
cf set-space-quota SPACE-NAME QUOTA-NAME
Forexample:cf set-space-quota myspace myspacequota
Formoreinformationonmanagingspace-levelquotas,seeCreatingandModifyingQuotaPlans .
ViewCurrentOrgandSpace-levelQuotasTovieworgquotas,runthefollowingcommand.
cf org ORG-NAME
Toviewspacequotas,runthefollowingcommand:
cf space SPACE-NAME
Formoreinformationonmanagingorgandspace-levelquotas,seetheCreatingandModifyingQuotaPlans .
MonitorQuotaUseandServiceInstanceCountService-levelandplan-levelquotause,andtotalnumberofserviceinstances,areavailablethroughtheon-demandbrokermetricsemittedtoLoggregator.Thesemetricsarelistedbelow:
MetricName Description
© Copyright Pivotal Software Inc, 2013-2018 36 1.10
on-demand-broker/SERVICE-NAME/quota_remaining Quotaremainingforallinstancesacrossallplans
on-demand-broker/SERVICE-NAME/PLAN-NAME/quota_remaining Quotaremainingforaspecificplan
on-demand-broker/SERVICE-NAME/total_instances Totalinstancescreatedacrossallplans
on-demand-broker/SERVICE-NAME/PLAN-NAME/total_instances Totalinstancescreatedforaspecificplan
CalculateResourceCostsforOn-DemandPlansOn-demandplansusededicatedVMs,disks,andvariousotherresourcesfromanIaaS,suchasAWS.Tocalculatemaximumresourcecostforplansindividuallyorcombined,youmultiplythequotabythecostofVMandPersistentDisktypesselectedintheplanconfiguration(s).ThespecificcostsdependonyourIaaS.
TheimagebelowshowsanexampleoftheVMtypeandpersistentdiskselected,aswellasthequotaforthisplan.
CalculateMaximumResourceCostPerOn-DemandPlanTocalculatethemaximumcostofVMsandpersistentdiskforeachplan,dothefollowingcalculation:
planquotaxcostofselectedresources
Forexample,ifyouselectedtheoptionsintheaboveimage,youhaveselectedaVMtypemicro.cpuandapersistentdisktype20GB,andtheplanquotais15.TheVMandpersistentdisktypeshaveanassociatedcostfortheIaaSyouareusing.Therefore,tocalculatethemaximumcostofresourcesforthisplan,multiplythecostoftheresourcesselectedbytheplanquota:
(15xcostofmicro.cpuVMtype)+(15xcostof20GBpersistentdisk)
CalculateMaximumResourceCostforAllOn-DemandPlansTocalculatethemaximumcostforallplanscombined,addtogetherthemaximumcostsforeachplan.Thisassumesthatthesumofyourindividualplanquotasislessthantheglobalquota.
Hereisanexample:
(plan1quotaxplan1resourcecost)+(plan2quotaxplan2resourcecost)=maxcostforallplans
Note:Quotametricsarenotemittedifnoquotahasbeenset.
Important:Althoughoperatorscanlimiton-demandinstanceswithplanquotasandaglobalquota,asdescribedintheabovetopics,IaaSresourceusagestillvariesbasedonthenumberofon-demandinstancesprovisioned.
© Copyright Pivotal Software Inc, 2013-2018 37 1.10
CalculateActualResourceCostofallOn-DemandPlansTocalculatethecurrentactualresourcecostacrossallyouron-demandplans:
1. Findthenumberofinstancescurrentlyprovisionedforeachactiveplanbylookingatthe total_instance metricforthatplan.
2. Multiplythe total_instance countforeachplanbythatplan’sresourcecosts.Recordthecostsforeachplan.
3. AddupthecostsnotedinStep2togetyourtotalcurrentresourcecosts.
Forexample:
(plan1totalinstancesxplan1resourcecost)+(plan2totalinstancesxplan2resourcecost)=currentcostforallplans
© Copyright Pivotal Software Inc, 2013-2018 38 1.10
ControllingAccesstoServicePlansbyOrgIfyouwanttorestrictaccesstoaserviceplantoaspecificorg,followtheinstructionsbelow.
Youcanalsolimitthenumberofserviceinstancesbysettingquotas—forinstructions,seeSettingLimitsforOn-DemandInstances.
ChangeAccesstoServicePlans
Torestrictaccesstoaplanforaspecificorg,runthiscommand:
cf enable-service-access p.rabbitmq -p PLAN_NAME -oORG_NAME
Forexample:
$ cf enable-service-access p.rabbitmq -p my-cluster-plan -o my-dev-org
Formoreinformationontheabovecommand,seeAccessControl .
Note:Iftheplanyouarerestrictingiscurrentlyenabledforallorgs,youmustfirstDisableServiceAccessfortheplan,thengrantaccesstotheplantospecificorgs.UsetheCFServiceAccessfieldtodisableaccessintheserviceplanconfiguration.
© Copyright Pivotal Software Inc, 2013-2018 39 1.10
MonitoringandKPIsforOn-DemandRabbitMQforPCFThistopicexplainshowtomonitorthehealthoftheon-demandversionoftheRabbitMQforPivotalCloudFoundry(PCF)serviceusingthelogs,metrics,andKeyPerformanceIndicators(KPIs)generatedbyRabbitMQforPCFcomponentVMs.
On-demandRabbitMQforPCFcomponentsgeneratemanyofthesamemetricsasthepre-provisionedRabbitMQservicecomponents.Forinformationaboutmetricsforthepre-provisionedservice,seeMonitoringandKPIsforPre-ProvisionedRabbitMQforPCF.
SeeLoggingandMetrics forgeneralinformationaboutloggingandmetricsinPCF.
SetupSyslogForwardingAsofRabbitMQforPCFv1.9.0,syslogforwardingispreconfiguredandsettoonbydefault.Pivotalrecommendsthatyoukeepthedefaultsettingbecauseitisgoodoperationalpractice.However,youcanoptoutbyselectingNoforDoyouwanttoconfiguresyslog?intheOpsManagerSettingstab.
ToenablemonitoringforRabbitMQforPCF,operatorsdesignateanexternalsyslogendpointforRabbitMQcomponentlogmessages.TheendpointservesastheinputtoamonitoringplatformsuchasDatadog,Papertrail,orSumoLogic.
TospecifythedestinationforRabbitMQforPCFlogmessages,dothefollowing:
1. FromtheOpsManagerInstallationDashboard,clicktheRabbitMQtile.
2. IntheRabbitMQtile,clicktheSettingstab.
Note:On-demandservicemetricsareprefixedwith p.rabbitmq (withadot),todistinguishthemfromthepre-provisionedservicemetrics.
© Copyright Pivotal Software Inc, 2013-2018 40 1.10
3. ClickSyslog.
4. ConfigurethefieldsontheSyslogpaneasfollows:
Option Description
Syslogaddress IPorDNSaddressofthesyslogserver
Syslogport Portofthesyslogserver
Transportprotocol
Transportprotocolofthesyslogserver.Oneof udp , tcp , relp .
Formatforlogs Formatforlogs.Pivotalrecommends RFC 5424 ,but Legacy Format canbeusedforcompatibilityreasons.
EnableTLS EnableTLStothesyslogserver.
Ifthereareseveralpeerserversthatcanrespondtoremotesyslogconnections,thenyoumayprovideawildcardinthe
© Copyright Pivotal Software Inc, 2013-2018 41 1.10
PermittedPeer domain,suchas *.example.com .
CustomCACertificate
Iftheservercertificateisnotsignedbyaknownauthority,forexample,aninternalsyslogserver,providetheCAcertificateofthelogmanagementserviceendpoint.
5. ClickSave.
6. ReturntotheOpsManagerInstallationDashboardandclickApplyChangestoredeploywiththechanges.
LoggingFormatWithon-demandRabbitMQforPCFloggingconfigured,twotypesofcomponentsgeneratelogs:theservernodesandtheservicebroker:
ThelogsforRabbitMQservernodesfollowtheformat [job=rabbitmq-server-partition-GUID index=0]
ThelogsfortheRabbitMQservicebrokerfollowtheformat [job=rabbitmq-broker-partition-GUID index=0]
TheRabbitMQVMslogatthe info levelandcaptureerrors,warnings,andinformationalmessages.
Forusersfamiliarwithdocumentationforpreviousversionsofthetile,thetagweusedtocallthe app_name isnowcalledthe program_name .
Thegenericlogformatisasfollows:
<PRI>TIMESTAMP IP_ADDRESS PROGRAM_NAME [job=NAME index=JOB_INDEX id=JOB_ID] MESSAGE
Therawlogslooksimilartothefollowing:
<7>2017-06-28T16:06:10.733560+00:00 10.244.16.133 vcap.agent [job=rmq index=0 id=e37ecdca-5b10-4141-abd8-e1d777dfd8b5] 2017/06/28 16:06:10 CEF:0|CloudFoundry|BOSH|1|agent_api|ssh|1|duser=director.be5a66bb-a9b4-459f-a0d3-1fc5c9c3ed79.be148cc6-91ef-4eed-a788-237b0b8c63b7 src=10.254.50.4 spt=4222 shost=5ae233e0-ecc5-4868-9ae0-f9767571251b<86>2017-06-28T16:06:16.704572+00:00 10.244.16.133 useradd [job=rmq index=0 id=e37ecdca-5b10-4141-abd8-e1d777dfd8b5] new group: name=bosh_ly0d2rbjr, GID=1003<86>2017-06-28T16:06:16.704663+00:00 10.244.16.133 useradd [job=rmq index=0 id=e37ecdca-5b10-4141-abd8-e1d777dfd8b5] new user: name=bosh_ly0d2rbjr, UID=1001, GID=1003, home=/var/vcap/bosh_ssh/bosh_ly0d2rbjr, shell=/bin/bash<86>2017-06-28T16:06:16.736932+00:00 10.244.16.133 usermod [job=rmq index=0 id=e37ecdca-5b10-4141-abd8-e1d777dfd8b5] add 'bosh_ly0d2rbjr' to group 'admin'<86>2017-06-28T16:06:16.736964+00:00 10.244.16.133 usermod [job=rmq index=0 id=e37ecdca-5b10-4141-abd8-e1d777dfd8b5] add 'bosh_ly0d2rbjr' to group 'vcap'
LogssenttoexternalloggingtoolssuchasPapertrailmaybepresentedinadifferentformat.
Thefollowingtabledescribestheloggingtagsusedinthistemplate:
Tag Description
PRI Thisisavaluewhichinfuturewillbeusedtodescribetheseverityofthelogmessageandwhichfacilityitcamefrom.
TIMESTAMPThisisthetimestampofwhenthelogisforwarded,forexample, 2016-08-24T05:14:15.000003Z .Thetimestampvalueistypicallyslightlyafterwhenthelogmessagewasgenerated.
IP_ADDRESS TheinternalIPaddressofserveronwhichthelogmessageoriginated
PROGRAM_NAMEProcessnameoftheprogramthegeneratedthemessage.Sameas app_name beforev1.9.0.Formoreinformationaboutprogramname,seeRabbitMQProgramNamesbelow.
NAME TheBOSHinstancegroupname(forexample, rabbitmq_server )
JOB_INDEX BOSHjobindex.Usedtodistinguishbetweenmultipleinstancesofthesamejob.
JOB_IDBOSHVMGUID.ThisisdistinctfromtheCIDdisplayedintheOpsManagerStatustab,whichcorrespondstotheVMIDassignedbytheinfrastructureprovider.
MESSAGE Thelogmessagethatappears
RabbitMQProgramNames
ProgramName Description
rabbitmq_server_cluster_check ChecksthattheRabbitMQclusterishealthy.Runsaftereverydeploy.
rabbitmq_server_node_check ChecksthattheRabbitMQnodeishealthy.Runsaftereverydeploy.
rabbitmq_route_registrar_stderr RegisterstherouteforthemanagementAPIwiththeGorouterinyourElasticRuntimedeployment.
rabbitmq_route_registrar_stdout RegisterstherouteforthemanagementAPIwiththeGorouterinyourElasticRuntimedeployment.
rabbitmq_server TheErlangVMandRabbitMQapps.Logsmayspanmultiplelines.
rabbitmq_server_drain ShutsdowntheErlangVMandRabbitMQapps.RunsaspartoftheBOSHlifecycle.
rabbitmq_server_http_api_access AccesstotheRabbitMQmanagementUI.
© Copyright Pivotal Software Inc, 2013-2018 42 1.10
rabbitmq_server_init StartstheErlangVMandRabbitMQ.
rabbitmq_server_post_deploy_stderr Runsthenodecheckandclustercheck.Runsaftereverydeploy.
rabbitmq_server_post_deploy_stdout Runsthenodecheckandclustercheck.Runsaftereverydeploy.
rabbitmq_server_pre_start Runsbeforetherabbitmq-serverjobisstarted.
rabbitmq_server_sasl Supervisor,progress,andcrashreportingfortheErlangVMandRabbitMQapps.
rabbitmq_server_shutdown_stderr StopstheRabbitMQappandErlangVM.
rabbitmq_server_shutdown_stdout StopstheRabbitMQappandErlangVM.
rabbitmq_server_startup_stderr StartstheRabbitMQappandErlangVM,thenconfiguresusersandpermissions.
rabbitmq_server_startup_stdout StartstheRabbitMQappandErlangVM,thenconfiguresusersandpermissions.
rabbitmq_server_upgrade ShutsdownErlangVMandRabbitMQappifrequiredduringanupgrade.
ProgramName Description
WhatAreMetricsMetricsareregularly-generatedlogmessagesthatreportmeasuredcomponentstates.Themetricspollingintervaldefaultsto30seconds.ThemetricspollingintervalisaconfigurationoptionontheRabbitMQtile(Settings>RabbitMQ).Theintervalsettingappliestoallcomponentsdeployedbythetile.
Metricsarelong,singlelinesoftextthatfollowtheformat:
origin:"p.rabbitmq" eventType:ValueMetric timestamp:1441188462382091652 deployment:"cf-rabbitmq" job:"cf-rabbitmq-node" index:"0" ip:"10.244.3.46" valueMetric: < name:"/p.rabbitmq/rabbitmq/system/memory" value:1024 unit:"MB">
ThefollowingsectionsdescribethemetricsusedasKeyPerformanceIndicatorsandotherusefulmetricsformonitoringtheRabbitMQforPCFon-demandservice.
KeyPerformanceIndicatorsKeyPerformanceIndicators(KPIs)forRabbitMQforPCFaremetricsthatoperatorsfindmostusefulformonitoringtheirRabbitMQservicetoensuresmoothoperation.KPIsarehigh-signal-valuemetricsthatcanindicateemergingissues.KPIscanberawcomponentmetricsorderivedmetricsgeneratedbyapplyingformulastorawmetrics.
PivotalprovidesthefollowingKPIsasgeneralalertingandresponseguidancefortypicalRabbitMQforPCFinstallations.Pivotalrecommendsthatoperatorscontinuetofine-tunethealertmeasurestotheirinstallationbyobservinghistoricaltrends.Pivotalalsorecommendsthatoperatorsexpandbeyondthisguidanceandcreatenew,installation-specificmonitoringmetrics,thresholds,andalertsbasedonlearningfromtheirowninstallations.
ForalistofallRabbitMQforPCFrawcomponentmetrics,seeComponentMetricsReference.
ComponentHeartbeatsKeyRabbitMQforPCFcomponentsperiodicallyemitheartbeatmetrics:theRabbitMQservernodes,HAProxynodes,andtheServiceBroker.TheheartbeatsareBooleanmetrics,where 1 meansthesystemisavailable,and 0 ortheabsenceofaheartbeatmetricmeanstheserviceisnotrespondingandshouldbeinvestigated.
ServiceBrokerHeartbeat
p.rabbitmq/service_broker/heartbeat
Description
RabbitMQServiceBroker is alive poll,whichindicatesifthecomponentisavailableandabletorespondtorequests.
Use:IftheServiceBrokerdoesnotemitheartbeats,thisindicatesthatitisoffline.TheServiceBrokerisrequiredtocreate,update,anddeleteserviceinstances,whicharecriticalfordependenttilessuchasSpringCloudServicesandSpringCloudDataFlow.
Origin:Doppler/FirehoseType:boolean
© Copyright Pivotal Software Inc, 2013-2018 43 1.10
Frequency:30s(default),10s(configurableminimum)
Recommendedmeasurement Averageoverlast5minutes
RecommendedalertthresholdsYellowwarning:N/ARedcritical:<1
Recommendedresponse
ChecktheRabbitMQServiceBrokerlogsforerrors.YoucanfindthisVMbytargetingyourRabbitMQdeploymentwithBOSHandrunningoneofthefollowingcommandsdependingonyourOpsManagerVersion:
ForOpsManagerv1.10orearlier:bosh vms service-instance_GUID
ForOpsManagerv1.11orlater:bosh2 -d service-instance_GUID vms
ServerHeartbeat
p.rabbitmq/rabbitmq/heartbeat
Description
RabbitMQServer is alive poll,whichindicatesifthecomponentisavailableandabletorespondtorequests.
Use:Iftheserverdoesnotemitheartbeats,thisindicatesthatitisoffline.Tobefunctional,serviceinstancesrequireRabbitMQServer.
Origin:Doppler/FirehoseType:booleanFrequency:30s(default),10s(configurableminimum)
Recommendedmeasurement Averageoverlast5minutes
RecommendedalertthresholdsYellowwarning:N/ARedcritical:<1
Recommendedresponse
ChecktheRabbitMQServerlogsforerrors.YoucanfindtheVMbytargetingyourRabbitMQdeploymentwithBOSHandrunningoneofthefollowingcommands,whichlists rabbitmq :
ForOpsManagerv1.10orearlier:bosh vms service-instance_GUID
ForOpsManagerv1.11orlater:bosh2 -d service-instance_GUID vms
RabbitMQServerKPIsThefollowingKPIsfromtheRabbitMQservercomponent:
FileDescriptors
p.rabbitmq/rabbitmq/system/file_descriptors
Description
Filedescriptorsconsumed.
Use:Ifthenumberoffiledescriptorsconsumedbecomestoolarge,theVMmightlosetheabilitytoperformdiskIO,whichcancausedataloss.
Origin:Doppler/FirehoseType:countFrequency:30s(default),10s(configurableminimum)
Recommendedmeasurement Averageoverlast10minutes
Note:Thisassumesnon-persistentmessagesarehandledbyretriesorsomeotherlogicbytheproducers.
© Copyright Pivotal Software Inc, 2013-2018 44 1.10
RecommendedalertthresholdsYellowwarning:>50000Redcritical:>55000
Recommendedresponse
Thedefault ulimit forRabbitMQforPCFv1.6andlateris60000.Ifthismetricismetorexceededforanextendedperiodoftime,consideroneofthefollowingactions:
ScalingtherabbitnodesinthetileResourceConfigpane.
Increasingthe ulimit
ErlangProcesses
p.rabbitmq/rabbitmq/system/erlang_processes
Description
Erlang processesconsumedbyRabbitMQ,whichrunsonanErlangVM.
Use:Thisisthekeyindicatoroftheprocessingcapabilityofanode.
Origin:Doppler/FirehoseType:countFrequency:30s(default),10s(configurableminimum)
Recommendedmeasurement Averageoverlast10minutes
RecommendedalertthresholdsYellowwarning:>900000Redcritical:>950000
RecommendedresponseThedefaultErlangprocesslimitinRabbitMQforPCFv1.6andlateris1,048,816.Ifthismetricmeetsorexceedstherecommendedthresholdsforextendedperiodsoftime,considerscalingtheRabbitMQnodesinthetileResourceConfigpane.
BOSHSystemHealthMetricsTheBOSHlayerthatunderliesPCFgenerates healthmonitor metricsforallVMsinthedeployment.However,thesemetricsarenotincludedintheLoggregatorFirehosebydefault.Togetthesemetrics,doeitherofthefollowing:
TosendBOSHHMmetricsthroughtheFirehose,installtheopen-sourceHMForwarder .
ToretrieveBOSHhealthmetricsoutsideoftheFirehose,installtheJMXBridge forPCFtile.
AllBOSH-deployedcomponentsgeneratethefollowingsystemhealthmetrics.ComingfromRabbitMQforPCFcomponents,thesesystemmetricsserveasKPIsfortheRabbitMQforPCFservice.
RAM
system.mem.percent
Description
RAMbeingconsumedbythe p.rabbitmq VM.
Use:RabbitMQisconsideredtobeinagoodstatewhenithasfewornomessages.Inotherwords,“anemptyrabbitisahappyrabbit.”Alertingonthismetriccanindicatethattherearetoofewconsumersorappsthatreadmessagesfromthequeue.
HealthmonitorreportswhenRabbitMQusesmorethan40%ofitsRAMforthepasttenminutes.
Origin:JMXBridgeorBOSHHMType:percentFrequency:30s(default),10s(configurableminimum)
Recommendedmeasurement Averageoverlast10minutes
RecommendedalertthresholdsYellowwarning:>40Redcritical:>50
Recommendedresponse Addmoreconsumerstodrainthequeueasfastaspossible.
© Copyright Pivotal Software Inc, 2013-2018 45 1.10
CPU
system.cpu.percent
Description
CPUbeingconsumedbythe p.rabbitmq VM.
Use:AnodethatexperiencescontextswitchingorhighCPUusagebecomesunresponsive.Thisalsoaffectstheabilityofthenodetoreportmetrics.
HealthmonitorreportswhenRabbitMQusesmorethan40%ofitsCPUforthepasttenminutes.
Origin:JMXBridgeorBOSHHMType:percentFrequency:30s(default),10s(configurableminimum)
Recommendedmeasurement Averageoverlast10minutes
RecommendedalertthresholdsYellowwarning:>60Redcritical:>75
Recommendedresponse Rememberthat“anemptyrabbitisahappyrabbit”.Addmoreconsumerstodrainthequeueasfastaspossible.
EphemeralDisk
system.disk.percent
Description
EphemeralDiskbeingconsumedbythe p.rabbitmq VM.
Use:Ifsystemdiskfillsup,therearetoofewconsumers.
HealthmonitorreportswhenRabbitMQusesmorethan40%ofitsCPUforthepasttenminutes.
Origin:JMXBridgeorBOSHHMType:percentFrequency:30s(default),10s(configurableminimum)
Recommendedmeasurement Averageoverlast10minutes
RecommendedalertthresholdsYellowwarning:>60Redcritical:>75
Recommendedresponse Rememberthat“anemptyrabbitisahappyrabbit”.Addmoreconsumerstodrainthequeueasfastaspossible.
PersistentDisk
persistent.disk.percent
Description
PersistentDiskbeingconsumedbythe p.rabbitmq VM.
Use:Ifsystemdiskfillsup,therearetoofewconsumers.
HealthmonitorreportswhenRabbitMQusesmorethan40%ofitsCPUforthepasttenminutes.
Origin:JMXBridgeorBOSHHMType:percentFrequency:30s(default),10s(configurableminimum)
Recommendedmeasurement Averageoverlast10minutes
RecommendedalertthresholdsYellowwarning:>60Redcritical:>75
Recommendedresponse Rememberthat“anemptyrabbitisahappyrabbit”.Addmoreconsumerstodrainthequeueasfastaspossible.
© Copyright Pivotal Software Inc, 2013-2018 46 1.10
DetermineIfThereIsaNetworkPartitionYoucanusethe reachable_nodes metrictohelptoidentifynetworkpartitions.Thismetricshowshowmanynodesintheclustereachindividualnodeisawareof.Agoodindicationthatanodemightbeinapartitioniswhenitisawareofonlyitself.
Hereisanexampleofthismetrics:
origin:"p.rabbitmq" eventType:ValueMetric timestamp:1441188462382091652 deployment:"cf-rabbitmq" job:"cf-rabbitmq-node" index:"0" ip:"10.244.3.46" valueMetric: < name:"/p.rabbitmq/rabbitmq/erlang/reachable_nodes" value:3 unit:"count">
Youcancreatemonitorstoemitalertsincaseaclusterseemstobeinapartition.Inahealthyclusterthatisnotundergoingupgrades,eachnode’sreachable_nodes countisequaltothenumberofnodesinthecluster.
Tomonitorfornetworkpartition,Pivotalrecommendsalertingwhenoneofthenodesstartsreportinga reachable_nodes countthatislessthanthesizeofthecluster.
Duringrollingupgrades,nodeslosecontactwithothernodes.Therefore,onlyalertifalowered reachable_nodes countpersistslongerthantheexpectedupgradetime.
RecoverfromaNetworkPartitionForinformationabouthowtorecoverfromanetworkpartition,seetheRabbitMQdocumentation .
ComponentMetricReferenceRabbitMQforPCFcomponentVMsemitthefollowingrawmetrics.Thefullnameofthemetricfollowstheformat: /p.rabbitmq/COMPONENT/METRIC-NAME
RabbitMQServerMetricsRabbitMQforPCFmessageservercomponentsemitthefollowingmetrics.
FullName Unit Description
/p.rabbitmq/rabbitmq/heartbeat boolean IndicateswhethertheRabbitMQserverisavailableandabletorespondtorequests
/p.rabbitmq/rabbitmq/erlang/erlang_processes
count ThenumberofErlangprocesses
/p.rabbitmq/rabbitmq/erlang/reachable_nodes count Thenumberofnodesthecurrentnodecanreach
/p.rabbitmq/rabbitmq/system/memory MB ThememoryinMBusedbythenode
/p.rabbitmq/rabbitmq/system/mem_alarm boolean Indicatesifthememoryalarmhasgoneoff
/p.rabbitmq/rabbitmq/system/disk_free_alarm boolean Indicatesifthediskfreealarmhasgoneoff
/p.rabbitmq/rabbitmq/connections/count count Thetotalnumberofconnectionstothenode
/p.rabbitmq/rabbitmq/consumers/count count Thetotalnumberofconsumersregisteredinthenode
/p.rabbitmq/rabbitmq/messages/delivered count Thetotalnumberofmessageswiththestatus deliver_get onthenode
/p.rabbitmq/rabbitmq/messages/delivered_no_ack
count Thenumberofmessageswiththestatus deliver_no_ack onthenode
/p.rabbitmq/rabbitmq/messages/delivered_rate
rate Therateatwhichmessagesarebeingdeliveredtoconsumersorclientsonthenode
/p.rabbitmq/rabbitmq/messages/published count Thetotalnumberofmessageswiththestatus publish onthenode
/p.rabbitmq/rabbitmq/messages/published_rate
rate Therateatwhichmessagesarebeingpublishedbythenode
/p.rabbitmq/rabbitmq/messages/redelivered count Thetotalnumberofmessageswiththestatus redeliver onthenode
/p.rabbitmq/rabbitmq/messages/redelivered_rate
rate Therateatwhichmessagesaregettingthestatus redeliver onthenode
/p.rabbitmq/rabbitmq/messages/get_no_ack count Thenumberofmessageswiththestatus get_no_ack onthenode
/p.rabbitmq/rabbitmq/messages/get_no_ack_rate
rate Therateatwhichmessagesgetthestatus get_no_ack onthenode
© Copyright Pivotal Software Inc, 2013-2018 47 1.10
/p.rabbitmq/rabbitmq/messages/pending count Thenumberofmessageswiththestatus messages_unacknowledged onthenode
/p.rabbitmq/rabbitmq/messages/pending count Thenumberofmessageswiththestatus messages_unacknowledged onthenode
/p.rabbitmq/rabbitmq/messages/depth countThenumberofmessageswiththestatus messages_unacknowledged ormessages_ready onthenode
/p.rabbitmq/rabbitmq/system/filedescriptors
count Thenumberofopenfiledescriptorsonthenode
/p.rabbitmq/rabbitmq/exchanges/count count Thetotalnumberofexchangesonthenode
/p.rabbitmq/rabbitmq/messages/available count Thetotalnumberofmessageswiththestatus messages_ready onthenode
/p.rabbitmq/rabbitmq/queues/count count Thenumberofqueuesonthenode
/p.rabbitmq/rabbitmq/channels/count count Thenumberofchannelsonthenode
/p.rabbitmq/rabbitmq/queues/VHOST-NAME/QUEUE-NAME/consumers
count Thenumberofconsumerspervirtualhostperqueue
/p.rabbitmq/rabbitmq/queues/VHOST-NAME/QUEUE-NAME/depth
countThenumberofmessageswiththestatus messages_unacknowledged ormessages_ready pervirtualhostperqueue
© Copyright Pivotal Software Inc, 2013-2018 48 1.10
ClusteringandNetworkPartitions
ClusteringinRabbitMQforPCFInRabbitMQforPCF,theRabbitMQ®brokerisalwaysdeployedasaclusterofoneormorevirtualmachines(nodes).ARabbitMQbrokerisalogicalgroupingofoneorseveralErlangnodes,eachrunningtheRabbitMQapplicationandsharingusers,virtualhosts,queues,exchanges,bindings,andruntimeparameters.
WhatisReplicatedbetweennodesinaRabbitMQcluster?
Alldata/staterequiredfortheoperationofaRabbitMQbrokerisreplicatedacrossallnodes.Anexceptiontothisaremessagequeues,whichbydefaultresideononenode,thoughtheyarevisibleandreachablefromallnodes.ThismeansthattheRabbitMQclustermaybeavailableandservingrequests,whileanindividualqueueresidingonasinglenodeisoffline.
Replicatingmessagequeuesacrossnodesisanexpensiveoperationandshouldonlybedonetotheextentneededbytheapplication.Tounderstandmoreaboutreplicatingqueuesacrossnodesinacluster,seethedocumentation onhighavailability.
AutomaticNetworkPartitionBehaviorsinRabbitMQClustersTheRabbitMQ®tileusesthe pause_minority optionforhandlingclusterpartitionsbydefault.Thisensuresdataintegritybypausingthepartitionoftheclusterintheminority,andresumesitwiththedatafromthemajoritypartition.Youmustmaintainmorethantwonodes.Ifthereisapartitionwhenyouonlyhavetwonodes,bothnodesimmediatelypause.
Youcanalsochoosethe autoheal optionintheRabbitMQPolicytab.Inthismode,ifapartitionoccurs,RabbitMQautomaticallydecidesonawinningpartition,andrestartsallnodesthatarenotinthewinningpartition.Thisoptionallowsyoutocontinuetoreceiveconnectionstobothpartsofpartitions.
DetectingaNetworkPartitionWhenanetworkpartitionoccurs,alogmessageiswrittentotheRabbitMQnodelog:
=ERROR REPORT==== 15-Oct-2012::18:02:30 ===Mnesia(rabbit@da3be74c053640fe92c6a39e2d7a5e46): ** ERROR ** mnesia_event got {inconsistent_database, running_partitioned_network, rabbit@21b6557b73f343201277dbf290ae8b79}
Youcanalsorunthe rabbitmqctl cluster_status commandonanyoftheRabbitMQnodestoseethenetworkpartition.Torun rabbitmqctl cluster_status ,dothefollowing:
1. $ sudo su -
2. $ cd /var/vcap/packages
3. $ export ERL_DIR=$PWD/erlang/bin/
4. $ cd rabbitmq-server/bin/
5. $ ./rabbitmqctl cluster_status
[... {partitions, [{rabbit@da3be74c053640fe92c6a39e2d7a5e46, [rabbit@21b6557b73f343201277dbf290ae8b79]}]}]
RecoveringBecausetheRabbitMQtileusesthe pause_minority option,minoritynodesrecoverautomaticallyafterthepartitionisresolved.Afteranoderecovers,itresumesaccessingthequeuealongwithdatafromthequeuesontheothernodes.However,ifyourqueuesuse ha-mode: all ,theyonlysynchronizefullyafterconsumingallthemessagescreatedwhilethenodewasdown.Thisissimilartohowmessagessynchronizewhenyoucreateanewqueue.
© Copyright Pivotal Software Inc, 2013-2018 49 1.10
ManuallySynchronizingafteraPartitionAfteranetworkpartition,aqueueonaminoritynodesynchronizesafterconsumingallthemessagescreatedwhileitwasdown.Youcanalsorunthe
sync_queue commandtosynchronizeaqueuemanually.Torun sync_queue ,dothefollowingoneachnode:
1. $ sudo su -
2. $ cd /var/vcap/packages
3. $ export ERL_DIR=$PWD/erlang/bin/
4. $ cd rabbitmq-server/bin/
5. $ ./rabbitmqctl list_queues
6. $ ./rabbitmqctl sync_queue name
© Copyright Pivotal Software Inc, 2013-2018 50 1.10
TroubleshootingOn-DemandRabbitMQforPCFInthistopic:
AbouttheBOSHCLI
TroubleshootingErrors
FailedInstallCannotCreateorDeleteServiceInstancesBrokerRequestTimeoutsCannotBindtoorUnbindfromServiceInstancesCannotConnecttoaServiceInstanceUpgradeAllInstancesFailsMissingLogsandMetricsFailedDeploymentonUpgradeorafterApplyChanges
TroubleshootingComponents
BOSHproblemsConfigurationAuthenticationNetworkingQuotas
TechniquesforTroubleshooting
ParseaCloudFoundry(CF)ErrorMessageAccessBrokerandInstanceLogsandVMsRunServiceBrokerErrandstoManageBrokersandInstancesSelecttheBOSHDeploymentforaServiceInstanceGetAdminCredentialsforaServiceInstanceReinstallaTileViewResourceSaturationandScalingIdentifyServiceInstanceOwnerMonitorQuotaSaturationandServiceInstanceCount
FrequentlyAskedQuestions
WhatshouldIcheckbeforedeployinganewversionofthetile?WhatisthecorrectwaytostopandstartRabbitMQinPCF?WhathappenswhenIstoptheRabbitMQserver?WhathappenswhentheRabbitMQserverfailstostop?WhatdoIdowhentheRabbitMQserverfailstostop?HowcanImanuallybackupthestateoftheRabbitMQcluster?Whatpre-upgradechecksshouldIdo?
KnowledgeBase(Community)
FileaSupportTicket
Thistopicprovidesoperatorswithbasicinstructionsfortroubleshootingon-demandRabbitMQ®forPivotalCloudFoundry(PCF).
AbouttheBOSHCLITheBOSHCLIisavailableintwomajorversions,v1andv2.PivotalrecommendsthatyouusetheBOSHCLIv2whenpossible.
ThistopicprovidesexamplesofusingeachversionoftheBOSHCLI.ConsultthetablebelowtodeterminewhichversionoftheCLIissupportedforyourinstallation.
OpsManagerVersion BOSHCLIVersion
1.10 CLIv1
1.11 CLIv1orCLIv2(PivotalrecommendsCLIv2)
1.12andlater CLIv2
© Copyright Pivotal Software Inc, 2013-2018 51 1.10
TroubleshootingErrorsStarthereifyou’rerespondingtoaspecificerrororerrormessages.
FailedInstall
1. Certificateissues:Theon-demandbroker(ODB)requiresvalidcertificates.Ensurethatyourcertificatesarevalidandgeneratenewones ifnecessary.
2. Deployfails:Deployscanfailforavarietyofreasons.ViewthelogsusingOpsManagertodeterminewhythedeployisfailing.
3. Networkingproblems:
CloudFoundrycannotreachtheRabbitMQforPCFservicebrokerCloudFoundrycannotreachtheserviceinstancesTheservicenetworkcannotaccesstheBOSHdirector
4. Registerbrokererrandfails.
5. Thesmoketesterrandfails.
6. Resourcesizingissues:TheseoccurwhentheresourcesizesselectedforagivenplanarelessthantheRabbitMQforPCFservicerequirestofunction.CheckyourresourceconfigurationinOpsManagerandensurethattheconfigurationmatchesthatrecommendedbytheservice.
7. Otherservice-specificissues.
CannotCreateorDeleteServiceInstances
Ifdevelopersreporterrorssuchasthefollowing:
Instance provisioning failed: There was a problem completing your request. Please contact your operations team providing the following information: service: redis-acceptance, service-instance-guid: ae9e232c-0bd5-4684-af27-1b08b0c70089, broker-request-id: 63da3a35-24aa-4183-aec6-db8294506bac, task-id: 442, operation: create
1. IftheBOSHerrorshowsaproblemwiththedeploymentmanifest:
a. Downloadthemanifestfortheon-demandserviceinstancebyrunning:bosh download manifest service-instance_SERVICE-INSTANCE-GUID MY-SERVICE.yml.
b. Checkthemanifestforconfigurationerrors.
2. Tocontinuetroubleshooting,LogintoBOSH andtargettheRabbitMQforPCFserviceinstanceusingtheinstructionsonparsingaCloudFoundryerrormessage.
3. RetrievetheBOSHtaskIDfromtheerrormessageandrunoneofthefollowingcommandsdependingonyourOpsManagerversion:
OpsManagerVersion BOSHCommand
1.10andearlier bosh task TASK-ID
1.11 bosh2 task TASK-ID
1.12andlater bosh task TASK-ID
4. Ifyouneedmoreinformation,accessthebrokerlogsandusethe broker-request-id fromtheerrormessageabovetosearchthelogsformoreinformation.Checkfor:
Note:ThiserrordoesnotapplyifyouareusingBOSHCLIv2.Inthatcase,totroubleshootpossibleproblemswiththemanifest,openitinatexteditorandinspectthemanifestthere.
© Copyright Pivotal Software Inc, 2013-2018 52 1.10
AuthenticationerrorsNetworkerrorsQuotaerrors
BrokerRequestTimeoutsIfdevelopersreporterrorssuchas:
Server error, status code: 504, error code: 10001, message: The request to the service broker timed out: https://BROKER-URL/v2/service_instances/e34046d3-2379-40d0-a318-d54fc7a5b13f/service_bindings/aa635a3b-ef6d-41c3-a23f-55752f3f651b
1. ConfirmthatCloudFoundry(CF)isconnectedtotheservicebroker.
2. ChecktheBOSHqueuesize:
a. LogintoBOSHasanadmin.b. RunoneofthesecommandsdependingonyourOpsManagerversion:
1.10andearlier: bosh tasks1.11: bosh2 tasks1.12andlater: bosh tasks
3. Iftherearealargenumberofqueuedtasks,thesystemmaybeundertoomuchload.BOSHisconfiguredwithtwoworkersandonestatusworker,whichmaynotbesufficientresourcesforthelevelofload.Adviseappdeveloperstotryagainoncethesystemisunderlessload.
CannotBindtoorUnbindfromServiceInstances
InstanceDoesNotExist
Ifdevelopersreporterrorssuchas:
Server error, status code: 502, error code: 10001, message: Service broker error: instance does not exist`
Followthesesteps:
1. Type cf service MY-INSTANCE --guid .ThisconfirmsthatthetheRabbitMQforPCFserviceinstanceexistsinBOSHandCF,andreturnsaGUID.
2. UsingtheGUIDobtainedabove,runoneofthefollowingBOSHCLIcommandsdependingonyourOpsManagerversion:
OpsManagerVersion BOSHCommand
1.10andearlier bosh vms service-instance_GUID
1.11 bosh2 -d service-instance_GUID vms
1.12andlater bosh -d service-instance_GUID vms
IftheBOSHdeploymentisnotfound,ithasbeendeletedfromBOSH.ContactPivotalsupportforfurtherassistance.
OtherErrors
Ifdevelopersreporterrorssuchas:
Server error, status code: 502, error code: 10001, message: Service broker error: There was a problem completing your request. Please contact your operations team providing the following information: service: example-service, service-instance-guid: 8d69de6c-88c6-4283-b8bc-1c46103714e2, broker-request-id: 15f4f87e-200a-4b1a-b76c-1c4b6597c2e1, operation: bind
© Copyright Pivotal Software Inc, 2013-2018 53 1.10
Tofindouttheexactissuewiththebindingprocess:
1. Accesstheservicebrokerlogs.
2. Searchthelogsforthe broker-request-id stringlistedintheerrormessageabove.
3. ContactPivotalsupportforfurtherassistanceifyouareunabletoresolvetheproblem.
4. Checkfor:
AuthenticationerrorsNetworkerrors
CannotConnecttoaServiceInstanceIfdevelopersreportthattheirappcannotuseserviceinstancesthattheyhavesuccessfullycreatedandbound:
Asktheusertosendapplicationlogsthatshowtheconnectionerror.Iftheerrorisoriginatingfromtheservice,thenfollowRabbitMQforPCF-specificinstructions.Iftheissueappearstobenetwork-related,then:
1. Checkthatapplicationsecuritygroups areconfiguredcorrectly.Accessshouldbeconfiguredfortheservicenetworkthatthetileisdeployedto.
2. EnsurethatthenetworkthePCFElasticRuntimetileisdeployedtohasnetworkaccesstotheservicenetwork.YoucanfindthenetworkdefinitionforthisservicenetworkintheOpsManagerDirectortile.
3. InOpsManagergointotheservicetileandseetheservicenetworkthatisconfiguredinthenetworkstab.
4. InOpsManagergointotheERTtileandseethenetworkitisassignedto.Makesurethatthesenetworkscanaccesseachother.
UpgradeAllInstancesFailsIfthe upgrade-all-service-instances errandfails,lookattheerrandoutputintheOpsManagerlog.
Ifaninstancefailstoupgrade,debugandfixitbeforerunningtheerrandagaintopreventanyfailureissuesfromspreadingtootheron-demandinstances.
OncetheOpsManagerlognolongerliststhedeploymentas failing ,re-runtheerrandtoupgradetherestoftheinstances.
MissingLogsandMetricsIfnologsarebeingemittedbytheon-demandbroker,checkthatyoursyslogforwardingaddressiscorrectinOpsManager.
1. Ensureyouhaveconfiguredsyslogforthetile.
2. Ensurethatyouhavenetworkconnectivitybetweenthenetworksthatthetileisusingandthesyslogdestination.Ifthedestinationisexternal,youneedtousethepublicip VMextensionfeatureavailableinyourOpsManagertileconfigurationsettings.
3. VerifythattheFirehoseisemittingmetrics:
a. Installthe cf nozzle plugin
b. Run cf nozzle -f ValueMetric | grep --line-buffered "on-demand-broker/MY-SERVICE" tofindlogsfromyourserviceinthecf nozzle output.
Ifnometricsappearwithinfiveminutes,verifythatthebrokernetworkhasaccesstotheLoggregatorsystemonallrequiredports.
ContactPivotalsupportifyouareunabletoresolvetheissue.
© Copyright Pivotal Software Inc, 2013-2018 54 1.10
FailedDeploymentonUpgradeorafterApplyChangesIfthedeploymentfailsaftereditingtheAssignAZsandNetworkspaneoftheRabbitMQforPCFtile,itmightbeduetoachangetotheIPaddressesassignedtothe RabbitMQ Server job.RabbitMQforPCFrequiresthattheseIPaddressesdonotchangeonceassigned.Ifyouchangethem,thedeploymentfails.Thisincludeschangesmadetoyourcurrentinstallationorduringanupgrade.Todiagnoseandsolvethisissue,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .
TroubleshootingComponentsGuidanceoncheckingforandfixingissuesinon-demandservicecomponents.
BOSHproblems
MissingBOSHDirectorUUID
IfusingtheBOSHCLIv1,re-addthe director_uuid tothemanifest:
1. Run bosh status --uuid andrecordthe director_uuid valuefromtheoutput.
2. Editthemanifestandaddthe director_uuid: DIRECTOR-UUID fromthelaststepatthetopofthemanifest.
Formore,seeDeploymentIdentification intheBOSHdocs.
LargeBOSHQueue
On-demandservicebrokersaddtaskstotheBOSHrequestqueue,whichcanbackupandcausedelayunderheavyloads.AnappdeveloperwhorequestsanewRabbitMQforPCFinstancesees create in progress intheCloudFoundryCommandLineInterface(cfCLI)untilBOSHprocessesthequeuedrequest.
OpsManagercurrentlydeploystwoBOSHworkerstoprocessitsqueue.FutureversionsofOpsManagerwillletusersconfigurethenumberofBOSHworkers.
Configuration
Serviceinstancesinfailingstate
YoumayhaveconfiguredaVM/DisktypeintileplanpageinOpsManagerthatisinsufficientlylargefortheRabbitMQforPCFserviceinstancetostart.Seetile-specificguidanceonresourcerequirements.
Note:ThiserrordoesnotoccurifyouareusingBOSHCLIv2
© Copyright Pivotal Software Inc, 2013-2018 55 1.10
Authentication
UAAChanges
IfyouhaverotatedanyUAAusercredentialsthenyoumayseeauthenticationissuesintheservicebrokerlogs.
Toresolvethis,redeploytheRabbitMQforPCFtileinOpsManager.Thisprovidesthebrokerwiththelatestconfiguration.
NetworkingCommonissuesinclude:
1. NetworklatencywhenconnectingtotheRabbitMQforPCFserviceinstancetocreateordeleteabinding.
Solution:Tryagainorimprovenetworkperformance
2. NetworkfirewallrulesareblockingconnectionsfromtheRabbitMQforPCFservicebrokertotheserviceinstance.
Solution:OpentheRabbitMQforPCFtileinOpsManagerandcheckthetwonetworksconfiguredintheNetworkspane.Ensurethatthesenetworksallowaccesstoeachother.
3. NetworkfirewallrulesareblockingconnectionsfromtheservicenetworktotheBOSHdirectornetwork.
Solution:EnsurethatserviceinstancescanaccesstheDirectorsothattheBOSHagentscanreportin.
4. Appscannotaccesstheservicenetwork.
Solution:ConfigureCloudFoundryapplicationsecuritygroupstoallowruntimeaccesstotheservicenetwork.
5. ProblemsaccessingBOSH’sUAAortheBOSHdirector.
Solution:FollownetworktroubleshootingandcheckthattheBOSHdirectorisonline.
ValidateServiceBrokerConnectivitytoServiceInstances1. Tovalidateyoucan bosh2 ssh ontotheRabbitMQforPCFservicebroker:
WithBOSHCLIv2:Targetthedeployment,andreachtheserviceinstance.WithBOSHCLIv1:Downloadthebrokermanifestandtargetthedeployment,thentrytoreachtheserviceinstance.
2. IfnoBOSH task-id appearsintheerrormessage,lookinthebrokerlogusingthe broker-request-id fromthetask.
ValidateAppAccesstoServiceInstance
Use cf ssh toaccesstotheappcontainer,thentryconnectingtotheRabbitMQforPCFserviceinstanceusingthebindingincludedintheVCAP_SERVICES environmentvariable.
Quotas
Note:YoumustensurethatanychangestoUAAcredentialsarereflectedintheOpsManager credentials taboftheElasticRuntimetile.
© Copyright Pivotal Software Inc, 2013-2018 56 1.10
PlanQuotaissues
Ifdevelopersreporterrorssuchas:
Message: Service broker error: The quota for this service plan has been exceeded. Please contact your Operator for help.
1. Checkyourcurrentplanquota.
2. Increasetheplanquota.
3. LogintoOpsManager.
4. Reconfigurethequotaontheplanpage.
5. Deploythetile.
6. Findwhoisusingtheplanquotaandtaketheappropriateaction.
GlobalQuotaIssues
Ifdevelopersreporterrorssuchas:
Message: Service broker error: The quota for this service has been exceeded. Please contact your Operator for help.
1. Checkyourcurrentglobalquota.
2. Increasetheglobalquota.
3. LogintoOpsManager.
4. Reconfigurethequotaontheon-demandsettingspage.
5. Deploythetile.
6. Findoutwhoisusingthequotaandtaketheappropriateaction.
Failingjobsandunhealthyinstances
TodeterminewhetherthereisanissuewiththeRabbitMQforPCFservicedeployment,inspecttheVMs.Todoso,runoneofthefollowingcommands:
OpsManagerVersion BOSHCommand
1.10orearlier bosh vms --vitals service-instance_GUID
1.11 bosh2 -d service-instance_GUID vms --vitals
1.12andlater bosh -d service-instance_GUID vms --vitals
Foradditionalinformation,runoneofthefollowingcommands:
OpsManagerVersion BOSHCommand
1.10andearlier bosh instances --ps --vitals
1.11 bosh2 instances --ps --vitals
1.12andlater bosh instances --ps --vitals
© Copyright Pivotal Software Inc, 2013-2018 57 1.10
IftheVMisfailing,followtheservice-specificinformation.Anyunadvisedcorrectiveactions(suchasrunningBOSH restart onaVM)cancauseissuesintheserviceinstance.
TechniquesforTroubleshootingThissectioncontainsinstructionsoninteractingwiththeon-demandservicebrokerandon-demandserviceinstanceBOSHdeployments,andonperforminggeneralmaintenanceandhousekeepingtasks.
ParseaCloudFoundry(CF)ErrorMessageFailedoperations(create,update,bind,unbind,delete)resultinanerrormessage.YoucanretrievetheerrormessagelaterbyrunningthecfCLIcommand cf service INSTANCE-NAME .
$ cf service myservice
Service instance: myserviceService: super-dbBound apps:Tags:Plan: dedicated-vmDescription: Dedicated InstanceDocumentation url:Dashboard:
Last OperationStatus: create failedMessage: Instance provisioning failed: There was a problem completing your request. Please contact your operations team providing the following information: service: redis-acceptance, service-instance-guid: ae9e232c-0bd5-4684-af27-1b08b0c70089, broker-request-id: 63da3a35-24aa-4183-aec6-db8294506bac, task-id: 442, operation: createStarted: 2017-03-13T10:16:55ZUpdated: 2017-03-13T10:17:58Z
Usetheinformationinthe Message fieldtodebugfurther.ProvidethisinformationtoPivotalSupportwhenfilingaticket.
The task-id fieldmapstotheBOSHtaskID.FormoreinformationonafailedBOSHtask,usethe bosh task TASK-ID .
The broker-request-guid mapstotheportionoftheOn-DemandBrokerlogcontainingthefailedstep.Accessthebrokerlogthroughyoursyslogaggregator,oraccessBOSHlogsforthebrokerbytyping bosh logs broker 0 .Ifyouhavemorethanonebrokerinstance,repeatthisprocessforeachinstance.
AccessBrokerandInstanceLogsandVMsBeforefollowingtheproceduresbelow,logintothecfCLI andtheBOSHCLI .
AccessBrokerLogsandVM(s)
YoucanaccesslogsusingOpsManager byclickingontheLogstabinthetileanddownloadingthebrokerlogs.
ToaccesslogsusingtheBOSHCLI,dothefollowing:
1. Identifytheon-demandbroker(ODB)deploymentbyrunningoneofthefollowingcommands,dependingonyourOpsManagerversion:
OpsManagerVersion BOSHCommand
1.10andearlier bosh deployments
1.11 bosh2 deployments
© Copyright Pivotal Software Inc, 2013-2018 58 1.10
1.12andlater bosh deploymentsOpsManagerVersion BOSHCommand
2. ForBOSHCLIv1only:
a. Run bosh download manifest ODB-DEPLOYMENT-NAME odb.yml todownloadtheODBmanifest.b. SelecttheODBdeploymentusing bosh deployment odb.yml .
3. ViewVMsinthedeploymentusingoneofthefollowingcommands:
OpsManagerVersion BOSHCommand
1.10andearlier bosh instances
1.11 bosh2 -d DEPLOYMENT-NAME instances
1.12andlater bosh -d DEPLOYMENT-NAME instances
4. SSHontotheVMbyrunningoneofthefollowingcommands:
OpsManagerVersion BOSHCommand
1.10andearlier bosh ssh service-instance_GUID
1.11 bosh2 -d service-instance_GUID ssh
1.12andlater bosh -d service-instance_GUID ssh
5. Downloadthebrokerlogsbyrunningoneofthefollowingcommands:
OpsManagerVersion BOSHCommand
1.10andearlier bosh logs service-instance_GUID
1.11 bosh2 -d service-instance_GUID logs
1.12andlater bosh -d service-instance_GUID logs
ThearchivegeneratedbyBOSHorOpsManagerincludesthefollowinglogs:
LogName Description
broker.logRequeststotheon-demandbrokerandtheactionsthebrokerperformswhileorchestratingtherequest(e.g.generatingamanifestandcallingBOSH).Startherewhentroubleshooting.
broker_ctl.log Controlscriptlogsforstartingandstoppingtheon-demandbroker.
post-start.stderr.log
Errorsthatoccurduringpost-startverification.
post-start.stdout.log
Post-startverification.
drain.stderr.log Errorsthatoccurwhilerunningthedrainscript.
AccessServiceInstanceLogsandVMs1. Totargetanindividualserviceinstancedeployment,retrievetheGUIDofyourserviceinstancewiththecfCLIcommand cf service MY-SERVICE --guid .
2. ForBOSHCLIv1only:
a. Run bosh status --uuid toretrievetheBOSHDirectorGUID.
b. TodownloadyourBOSHmanifestfortheservice,run bosh download manifest service-instance_BOSH-DIRECTOR-GUID MANIFEST.ymlusingtheGUIDyoujustobtainedandafilenameyouwanttosavethemanifestas.
c. Editthefollowinglineintheserviceinstancemanifestthatyoujustsaved,toincludethecurrentBOSHDirectorGUID:
director_uuid: BOSH-DIRECTOR-GUID
d. Run bosh deployment MANIFEST.yml toselectthedeploymentusingtheDirectorUUID.
Note:“GUID”and“UUID”meanthesamething.
© Copyright Pivotal Software Inc, 2013-2018 59 1.10
3. ViewVMsinthedeploymentusingoneofthefollowingcommands:
OpsManagerVersion BOSHCommand
1.10andearlier bosh instances
1.11 bosh2 -d DEPLOYMENT-NAME instances
1.12andlater bosh -d DEPLOYMENT-NAME instances
4. SSHontoaVMbyrunningoneofthefollowingcommands:
OpsManagerVersion BOSHCommand
1.10andearlier bosh ssh service-instance_GUID
1.11 bosh2 -d service-instance_GUID ssh
1.12andlater bosh -d service-instance_GUID ssh
5. Downloadtheinstancelogsbyrunningoneofthefollowingcommands:
OpsManagerVersion BOSHCommand
1.10andearlier bosh logs service-instance_GUID
1.11 bosh2 -d service-instance_GUID logs
1.12andlater bosh -d service-instance_GUID logs
RunServiceBrokerErrandstoManageBrokersandInstancesFromtheBOSHCLI,youcanrunservicebrokererrandsthatmanagetheservicebrokersandperformmassoperationsontheserviceinstancesthatthebrokerscreated.Theseservicebrokererrandsinclude:
register-broker registersabrokerwiththeCloudControllerandlistsitintheMarketplace
deregister-broker deregistersabrokerwiththeCloudControllerandremovesitfromtheMarketplace
upgrade-all-service-instances upgradesexistinginstancesofaservicetoitslatestinstalledversion
delete-all-service-instances deletesallinstancesofservice
orphan-deployments detects“orphan”instancesthatarerunningonBOSHbutnotregisteredwiththeCloudController
Torunerrands:
1. ForBOSHCLIv1only:Selectthebrokerdeploymentbyrunningthiscommand:bosh deployment BOSH_MANIFEST.yml
2. RunoneofthefollowingcommandsdependingonyourOpsManagerversion:
OpsManagerVersion BOSHCommand
1.10andearlier bosh run errand ERRAND_NAME
1.11 bosh2 -d DEPLOYMENT_NAME run-errand ERRAND_NAME
1.12andlater bosh -d DEPLOYMENT_NAME run-errand ERRAND_NAME
Examples:bosh run errand deregister-broker bosh2 -d DEPLOYMENT-NAME run-errand deregister-broker
RegisterBroker
The register-broker errandregistersthebrokerwithCloudFoundryandenablesaccesstoplansintheservicecatalog.Runthiserrandwheneverthebrokerisre-deployedwithnewcatalogmetadatatoupdatetheCloudFoundrycatalog.
Planswithdisabledserviceaccessarenotvisibletonon-adminCloudFoundryusers(includingOrgManagersandSpaceManagers).AdminCloudFoundry
© Copyright Pivotal Software Inc, 2013-2018 60 1.10
userscanseeallplansincludingthosewithdisabledserviceaccess.
Theerranddoesthefollowing:
RegisterstheservicebrokerwithCloudController.
Enablesserviceaccessforanyplansthathavetheradiobuttonsetto enabled inthetileplanpage.
Disablesserviceaccessforanyplansthathavetheradiobuttonsetto disabled inthetileplanpage.
Doesnothingforanyforanyplansthathavetheradiobuttonsetto manual .
Toruntheerrand,dothefollowing:
1. ForBOSHCLIv1only:Selectthebrokerdeploymentbyrunningthiscommand:bosh deployment BOSH_MANIFEST.yml
2. RunoneofthefollowingcommandsdependingonyourOpsManagerversion:
OpsManagerVersion BOSHCommand
1.10andearlier bosh run errand register-broker
1.11 bosh2 -d DEPLOYMENT-NAME run-errand register-broker
1.12andlater bosh -d DEPLOYMENT-NAME run-errand register-broker
DeregisterBroker
ThiserrandderegistersabrokerfromCloudFoundry.
Theerranddoesthefollowing:
DeletestheservicebrokerfromCloudController
Failsifthereareanyserviceinstances,withorwithoutbindings
UsetheDeleteAllServiceInstanceserrandtodeleteanyexistingserviceinstances.
Toruntheerrand,dothefollowing:
1. ForBOSHCLIv1only:Selectthebrokerdeploymentbyrunningthecommand:bosh deployment BROKER_MANIFEST.yml .
2. RunoneofthefollowingcommandsdependingonyourOpsManagerversion:
OpsManagerVersion BOSHCommand
1.10andearlier bosh run errand deregister-broker
1.11 bosh2 -d DEPLOYMENT-NAME run-errand deregister-broker
1.12andlater bosh -d DEPLOYMENT-NAME run-errand deregister-broker
UpgradeAllServiceInstances
IfyouhavemadechangestotheplandefinitionoruploadedanewtileintoOpsManager,youmaywanttoupgradealltheRabbitMQforPCFserviceinstancestothelatestsoftware/plandefinition.
The upgrade-all-service-instances erranddoesthefollowing:
Collectsalloftheserviceinstancestheon-demandbrokerhasregistered.
Foreachinstancetheerrandserially:
Issuesanupgradecommandtotheon-demandbroker.Re-generatestheserviceinstancemanifestbasedonitslatestconfigurationfromthetile.Deploysthenewmanifestfortheserviceinstance.
© Copyright Pivotal Software Inc, 2013-2018 61 1.10
Waitsforthisoperationtocomplete,thenproceedstothenextinstance.
AddstoaretrylistanyinstancesthathaveongoingBOSHtasksatthetimeofupgrade.
Retriesanyinstancesintheretrylistuntilallareupgraded.
Ifanyinstancefailstoupgrade,theerrandfailsimmediately.Thispreventssystemicproblemsfromspreadingtotherestofyourserviceinstances.Runtheerrandbyfollowingeitheroftheproceduresbelow.
Toruntheerrand,youcaneitherselecttheerrandthroughtheOpsManagerUIandhaveitrunwhenyouclick Apply Changes ,ordothefollowing:
1. ForBOSHCLIv1only:Selectthebrokerdeploymentbyrunningthiscommand:bosh deployment BOSH_MANIFEST.yml
2. RunoneofthefollowingcommandsdependingonyourOpsManagerversion:
OpsManagerVersion BOSHCommand
1.10andearlier bosh run errand upgrade-all-service-instances
1.11 bosh2 -d DEPLOYMENT-NAME run-errand upgrade-all-service-instances
1.12andlater bosh -d DEPLOYMENT-NAME run-errand upgrade-all-service-instances
DeleteAllServiceInstances
Thiserranddeletesallserviceinstancesofyourbroker’sserviceofferingineveryorgandspaceofCloudFoundry.ItusestheCloudControllerAPItodothis,andthereforeonlydeletesinstancestheCloudControllerknowsabout.ItwillnotdeleteorphanBOSHdeployments.
OrphanBOSHdeploymentsdon’tcorrespondtoaknownserviceinstance.Whilerare,orphandeploymentscanoccur.Usethe orphan-deployments errandtoidentifythem.
Theerranddoesthefollowing:
Unbindsallapplicationsfromtheserviceinstances.
Deletesallserviceinstancessequentially.
Checksifanyinstanceshavebeencreatedwhiletheerrandwasrunning.
Ifnewly-createdinstancesaredetected,theerrandfails.
Toruntheerrand,dothefollowing:
1. ForBOSHCLIv1only:Selectthebrokerdeploymentbyrunningthecommand:bosh deployment BROKER_MANIFEST.yml .
2. RunoneofthefollowingcommandsdependingonyourOpsManagerversion:
OpsManagerVersion BOSHCommand
1.10andearlier bosh run errand delete-all-service-instances
1.11 bosh2 -d service-instance_GUID delete-deployment
1.12andlater bosh -d service-instance_GUID delete-deployment
DetectOrphanedInstancesServiceInstances
Aserviceinstanceisdefinedas‘orphaned’whentheBOSHdeploymentfortheinstanceisstillrunning,buttheserviceisnolongerregisteredinCloudFoundry.
The orphan-deployments errandcollatesalistofservicedeploymentsthathavenomatchingserviceinstancesinCloudFoundryandreturnthelisttotheoperator.ItisthenuptotheoperatortoremovetheorphanedBOSHdeployments.
WARNING:Useextremecautionwhenrunningthiserrand.Youshouldonlyuseitwhenyouwanttototallydestroyalloftheon-demandserviceinstancesinanenvironment.
© Copyright Pivotal Software Inc, 2013-2018 62 1.10
Toruntheerrand,dothefollowing:
1. ForBOSHCLIv1only:Selectthebrokerdeploymentbyrunningthecommand:bosh deployment BROKER_MANIFEST.yml
2. RuntheerrandusingoneofthefollowingcommandsdependingonyourOpsManagerversion:
OpsManagerVersion BOSHCommand
1.10andearlier bosh run errand orphan-deployments
1.11 bosh2 -d DEPLOYMENT-NAME run-errand orphan-deployments
1.12andlater bosh -d DEPLOYMENT-NAME run-errand orphan-deployments
Iforphandeploymentsexist,theerrandscriptwill:
Exitwithexitcode10
Outputalistofdeploymentnamesundera [stdout] header
Provideadetailederrormessageundera [stderr] header
Forexample:
[stdout][{"deployment_name":"service-instance_80e3c5a7-80be-49f0-8512-44840f3c4d1b"}]
[stderr]Orphan BOSH deployments detected with no corresponding service instance in Cloud Foundry. Before deleting any deployment it is recommended to verify the service instance no longer exists in Cloud Foundry and any data is safe to delete.
Errand 'orphan-deployments' completed with error (exit code 10)
ThesedetailswillalsobeavailablethroughtheBOSH /tasks/ APIendpointforuseinscripting:
$ curl 'https://bosh-user:bosh-password@bosh-url:25555/tasks/task-id/output?type=result' | jq .{ "exit_code": 10, "stdout": "[{"deployment_name":"service-instance_80e3c5a7-80be-49f0-8512-44840f3c4d1b"}]\n", "stderr": "Orphan BOSH deployments detected with no corresponding service instance in Cloud Foundry. Before deleting any deployment it is recommended to verify the service instance no longer exists in Cloud Foundry and any data is safe to delete.\n", "logs": { "blobstore_id": "d830c4bf-8086-4bc2-8c1d-54d3a3c6d88d" }}
Ifnoorphandeploymentsexist,theerrandscriptwill:
Exitwithexitcode0
Stdoutwillbeanemptylistofdeployments
Stderrwillbe None
[stdout][]
[stderr]None
Errand 'orphan-deployments' completed successfully (exit code 0)
Iftheerrandencountersanerrorduringrunningitwill:
Exitwithexit1
Stdoutwillbeempty
Anyerrormessageswillbeunderstderr
Tocleanuporphanedinstances,runthefollowingcommandoneachinstance:
© Copyright Pivotal Software Inc, 2013-2018 63 1.10
OpsManagerVersion BOSHCommand
1.10andearlier bosh delete deployment service-instance_SERVICE-INSTANCE-GUID
1.11 bosh2 delete-deployment service-instance_SERVICE-INSTANCE-GUID
1.12andlater bosh delete-deployment service-instance_SERVICE-INSTANCE-GUID
SelecttheBOSHDeploymentforaServiceInstance
ThisisanadditionaltroubleshootingoptionforBOSHCLIv1only.ItdoesnotapplytotheBOSHCLIv2.
1. RetrievetheGUIDofyourserviceinstancewiththecommand cf service YOUR-SERVICE-INSTANCE --guid .
2. TodownloadyourBOSHmanifestfortheservice,run bosh download manifest service-instance_SERVICE-INSTANCE-GUID myservice.yml usingtheGUIDyoujustobtainedandafilenameyouwanttousewhensavingthemanifest.
3. Run bosh deployment MY-SERVICE.yml toselectthedeployment.
GetAdminCredentialsforaServiceInstance
Toretrievetheadminandread-onlyadmincredentialsforaserviceinstance,performthefollowingsteps:
1. IdentifytheservicedeploymentbyGUID.
2. LogintoBOSH .
3. DownloadthemanifestfortheserviceinstanceandaddtheGUIDifusingtheBOSHCLIv1.
4. Lookinthemanifestforthe admin and roadmin credentials.
ReinstallaTile
Toreinstallatileinthesameenvironmentwhereitwaspreviouslyuninstalled:
1. Ensurethattheprevioustilewascorrectlyuninstalledasfollows:
a. Loginasanadminwith cf login .b. Use cf m toconfirmthattheMarketplacedoesnotlistRabbitMQforPCF.c. DependingonwhichversionoftheBOSHCLIyouareusing,followoneofthestepsbelowtologintoBOSHasanadmin:
i. ForBOSHCLIv2:Use bosh2 log-in .ii. ForBOSHCLIv1:Use bosh login .
d. DependingonwhichversionoftheBOSHCLIyouareusing,followoneofthestepsbelowtodisplayyourBOSHdeploymentstoconfirmthattheoutputdoesnotshowaRabbitMQforPCFdeployment:
i. ForBOSHCLIv2:Use bosh2 deployments .ii. ForBOSHCLIv1:Use bosh deployments .
e. Runthe“delete-all-service-instances”errandtodeleteeveryinstanceoftheservice.f. Runthe“deregister-broker”errandtodeletetheservicebroker.g. DependingonwhichversionoftheBOSHCLIyouareusing,followoneofthestepsbelow:
WARNING:RunningthiscommandmayleaveIaaSresourcesinanunusablestate.
SkipthisstepifyouareusingtheBOSHCLIv2.YoucannotdownloadthemanifestwiththeBOSHCLIv2.Openitinatexteditorinstead.
© Copyright Pivotal Software Inc, 2013-2018 64 1.10
i. ForBOSHCLIv2:Use bosh2 delete-deployment BROKER-DEPLOYMENT-NAME todeletetheservicebrokerBOSHdeployment.ii. ForBOSHCLIv1:Use bosh delete deployment BROKER-DEPLOYMENT-NAME todeletetheservicebrokerBOSHdeployment.
h. Reinstallthetile.
ViewResourceSaturationandScaling
BOSHCLIv2:Viewingstatistics
Toviewusagestatisticsforanyservicedothefollowing:
1. ForBOSHCLIv1only:Selectthebrokerdeploymentbyrunningthiscommand:bosh deployment BOSH_MANIFEST.yml
2. RunthefollowingcommandsdependingonyourOpsManagerversion:
OpsManagerVersion BOSHCommands
v1.10andearlier
RuntheBOSHCLIv1command bosh vms --vitals .Toviewprocess-levelinformation,run bosh instances --ps .
v1.11RuntheBOSHCLIv2command bosh2 -d DEPLOYMENT-NAME vms --vitals .Toviewprocess-levelinformation,runbosh2 -d DEPLOYMENT-NAME instances --ps
v1.12andlaterRuntheBOSHCLIv2command bosh -d DEPLOYMENT-NAME vms --vitals .Toviewprocess-levelinformation,runbosh2 -d DEPLOYMENT-NAME instances --ps
IdentifyServiceInstanceOwner
IfyouwanttoidentifywhichappsareusingaspecificserviceinstancefromtheBOSHdeploymentsname,youcanrunthefollowingsteps:
1. Takethedeploymentnameandstripthe service-instance_ leavingyouwiththeGUID.
2. LogintoCFasanadmin.
3. Obtainalistofallservicebindingsbyrunningthefollowing: cf curl /v2/service_instances/GUID/service_bindings
4. Theoutputfromtheabovecurlgivesyoualistof resources ,witheachitemreferencingaservicebinding,whichcontainsthe APP-URL .Tofindthename,org,andspacefortheapp,runthefollowing:
a. cf curl APP-URL andrecordtheappnameunder entity.nameb. cf curl SPACE-URL toobtainthespace,usingthe entity.space_url fromtheabovecurl.Recordthespacenameunder entity.namec. cf curl ORGANIZATION-URL toobtaintheorg,usingthe entity.organization_url fromtheabovecurl.Recordtheorganizationname
under entity.name
MonitorQuotaSaturationandServiceInstanceCount
QuotasaturationandtotalnumberofserviceinstancesareavailablethroughODBmetricsemittedtoLoggregator.Themetricnamesareshownbelow:
MetricName Description
on-demand-broker/SERVICE-NAME-MARKETPLACE/quota_remaining globalquotaremainingforallinstancesacrossallplans
Note:Whenrunning cf curl ensurethatyouqueryallpages,becausetheresponsesarelimitedtoacertainnumberofbindingsperpage.Thedefaultis50.Tofindthenextpagecurlthevalueunder next_url
© Copyright Pivotal Software Inc, 2013-2018 65 1.10
on-demand-broker/SERVICE-NAME-MARKETPLACE/PLAN-NAME/quota_remaining quotaremainingforaparticularplan
on-demand-broker/SERVICE-NAME-MARKETPLACE/total_instances totalinstancescreatedacrossallplans
on-demand-broker/SERVICE-NAME-MARKETPLACE/PLAN-NAME/total_instances totalinstancescreatedforagivenplan
MetricName Description
FrequentlyAskedQuestions
WhatshouldIcheckbeforedeployinganewversionofthetile?
EnsurethatallnodesintheclusterarehealthyviatheRabbitMQManagementUI,orhealthmetricsexposedviathefirehose.
DonotrelyonBOSH instances ,toexecutethistaskcorrectly.ThatoutputreflectsthestateoftheErlangVMusedbyRabbitMQ,nottheRabbitMQapplication.
WhatisthecorrectwaytostopandstartRabbitMQinPCF?
OnlyBOSHcommandsshouldbeusedbytheoperatortointeractwiththeRabbitMQapplication.ExamplesforbothversionsoftheCLIaregivenbelow.
ForOpsManagerv1.10orearlier:bosh stop rabbitmq-server bosh start rabbitmq-server
ForOpsManagerv1.11orlater:bosh2 stop rabbitmq-server bosh2 start rabbitmq-server
ThereareBOSHjoblifecyclehookswhichareonlyfiredwhenrabbitmq-serverisstoppedthroughBOSH.Youcanalsostopindividualinstancesbyrunningoneofthefollowingcommands:
ForOpsManagerv1.10orearlier: bosh stop JOB [index]
ForOpsManagerv1.11orlater: bosh2 stop JOB [index]
WhathappenswhenIstoptheRabbitMQserver?
YoucanstoptheRabbitMQserverwithBOSH stop .
BOSHstartstheshutdownsequencefromthebootstrapinstance.
ThiscommandtellstheRabbitMQapplicationtoshutdownandthenshutdowntheErlangVMinwhichitisrunning.Ifthissucceeds,runthefollowingcheckstoensurethattheRabbitMQapplicationandErlangVMhavestopped:
1. If /var/vcap/sys/run/rabbitmq-server/pid exists,checkthatthePIDinsidethisfiledoesnotpointtoarunningErlangVMprocess.ThisprocesscheckstheErlangPIDandnottheRabbitMQPID.
2. Checkthat rabbitmqctl doesnotreturnanErlangVMPID.
Oncethiscompletesonthebootstrapinstance,BOSHwillcontinuethesamesequenceonthenextinstance.AllremainingRabbitMQserversstopsequentiallyafterthat.
Note:Quotametricsarenotemittedifnoquotahasbeenset.
Note:Donotuse monit stop rabbitmq-server
.Thiscommanddoesnotcallthedrainscripts.
© Copyright Pivotal Software Inc, 2013-2018 66 1.10
WhathappenswhentheRabbitMQserverfailstostop?
IfBOSH stop fails,youwilllikelygetanerrorsayingthatthedrainscriptfailedwith:
result: 1 of 1 drain scripts failed. Failed Jobs: rabbitmq-server.
WhatdoIdowhentheRabbitMQserverfailstostop?
Thedrainscriptlogsto /var/vcap/sys/log/rabbitmq-server/drain.log .Ifyouhavearemotesyslogconfigured,thiswillappearasthe rmq_server_drain program.
1. BOSH ssh intothefailingrabbitmq-serverinstanceandstarttherabbitmq-serverjobbyrunning monit start rabbitmq-server ).
2. Whentherabbitmq-serverjobisrunning(confirmthisvia monit status ),run DEBUG=1 /var/vcap/jobs/rabbitmq-server/bin/drain .Thiswilltellyouexactlywhyit’sfailing.
HowcanImanuallybackupthestateoftheRabbitMQcluster?
YoucanbackupthestateofaRabbitMQclusterforboththeon-demandandpre-provisionedservicesusingtheRabbitMQManagementAPI.Backupsincludevhosts,exchanges,queuesandusers.
BackupManually1. LogintotheRabbitMQManagementUIastheadminuseryoucreated.
2. Selectexportdefinitionsfromthemainpage.
BackupandRestorewithaScript
UsetheAPItorunscriptswithcodesimilartothefollowing:
1. Forthebackup:
curl -u "$USERNAME:$PASSWORD" "http://$RABBIT_ADDRESS:15672/api/definitions"-o "$BACKUP_FOLDER/rabbit-backup.json"
2. Fortherestore:
curl -u "$USERNAME:$PASSWORD" "http://$RABBIT_ADDRESS:15672/api/definitions"-X POST -H "Content-Type: application/json" -d"@$BACKUP_FOLDER/rabbit-backup.json"
Whatpre-upgradechecksshouldIdo?
BeforedoinganyupgradeofRabbitMQ,Pivotalrecommendscheckingthefollowing:
Note:YouwillnotbeabletostartthejobwithBOSH start asthiscommandalwaysrunsthedrainscriptfirst.Itwillfailsincethedrainscriptisfailing.
© Copyright Pivotal Software Inc, 2013-2018 67 1.10
1. InOperationsManagercheckthatthestatusofalloftheinstancesishealthy.
2. LogintotheRabbitMQManagementUIandcheckthatnoalarmshavebeentriggeredandthatallnodesarehealthy,thatis,theyshoulddisplayasgreen.
3. Checkthatthesystemisnotclosetohittingeitherthememoryordiskalarm.DothisbylookingatwhathasbeenconsumedbyeachnodeintheRabbitMQManagmentUI.
KnowledgeBase(Community)FindtheanswertoyourquestionandbrowseproductdiscussionsandsolutionsbysearchingthePivotalKnowledgeBase .
FileaSupportTicketYoucanfileasupporttickethere .Besuretoprovidetheerrormessagefrom cf service YOUR-SERVICE-INSTANCE .
Tohelpexpeditetroubleshooting,alsoprovideyourservicebrokerlogs,yourserviceinstancelogsandBOSHtaskoutput,ifyourcf service YOUR-SERVICE-INSTANCE outputincludesa task-id .
© Copyright Pivotal Software Inc, 2013-2018 68 1.10
UsingOn-DemandRabbitMQforPCFThistopicprovidesinstructionsfordevelopersusingtheon-demandRabbitMQservicefortheirPivotalCloudFoundry(PCF)apps.RabbitMQenablesmessagingbetweencloud-basedservers,appsanddevices.
TheseproceduresusetheCloudFoundryCommand-LineInterface(cfCLI).YoucanalsouseAppsManager toperformthesametasksusingagraphicalUI.
Forgeneralinformation,seeManagingServiceInstanceswiththecfCLI .
PrerequisitesTouseon-demandRabbitMQforPCFwithyourPCFapps,youneed:
APCFinstallationwithRabbitMQforPCF installedandlistedintheMarketplace
ASpaceDeveloper orAdminaccountonthePCFinstallation
Alocalmachinewiththefollowinginstalled:
abrowserashelltheCloudFoundryCommand-LineInterface (cfCLI)theLinuxwatch command
Tologinto theorgandspacecontainingyourapp
DeveloperGuide
EntriesintheVCAP_SERVICESEnvironmentVariableAppsrunninginCloudFoundrygainaccesstotheboundserviceinstancesviaanenvironmentvariablecredentialshashcalledVCAP_SERVICES .Anexamplehashisshownbelow:
{ "p-rabbitmq": [{ "credentials": { "dashboard_url": "http://pivotal-rabbitmq.your.pcf.example.com/#/login/b5d0ad14-4352-48e8-8982-d5b1d257029f/tavk86pnnns1ddiqpsdtbchurn", "username": "b5d0ad14-4352-48e8-8982-d5b1d257029f", "password": "tavk86pnnns1ddiqpsdtbchurn", "protocols": { "amqp": { "password": "tavk86pnnns1ddiqpsdtbchurn", "username": "b5d0ad14-4352-48e8-8982-d5b1d257029f", "uris": [ "amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7", "amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7" ] } } } }]}
Youcanreadmoredetailsabouttheenvironmentvariable VCAP_SERVICES here.
TheCreate-BindProcessBecauseeveryappandserviceinPCFisscopedtoaspace ,anappcanonlyuseaserviceifaninstanceoftheserviceexistsinthesamespace.
TouseRabbitMQinaPCFapp:
1. UsethecfCLI orAppsManager tologintotheorgandspacethatcontainstheapp.
© Copyright Pivotal Software Inc, 2013-2018 69 1.10
2. MakesureaninstanceoftheRabbitMQforPCFserviceexistsinthesamespaceastheapp.
IfthespacedoesnotalreadyhaveaRabbitMQforPCFinstance,createone.IfthespacealreadyhasaRabbitforPCFinstance,youcanbindyourapptotheexistinginstanceorcreateanewinstancetobindtoyourapp.
3. BindtheapptotheRabbitMQforPCFserviceinstance,toenabletheapptouseRabbitMQ.
ConfirmServiceAvailabilityForanapptouseaservice,1)theservicemustbeavailableintheMarketplaceforitsspaceand2)aninstanceoftheservicemustexistinitsspace.
YoucanconfirmbothoftheseusingthecfCLIasfollows.
1. TofindoutifOn-DemandRabbitMQforPCFserviceisavailableintheMarketplace:
a. Enter cf marketplaceb. Iftheoutputlists ondemand-rabbitmq inthe service column,on-demandRabbitMQforPCFisavailable.Ifitisnotavailable,askyour
operatortoinstallit.
$ cf marketplaceGetting services from marketplace in org my-org / space my-space as [email protected] plans description[...]ondemand-rabbitmq Solo RabbitMQ Service[...]
2. ToconfirmthatanOn-DemandRabbitMQforPCFinstanceisrunninginthespace
a. Enter cf servicesb. Any ondemand-rabbitmq listingsinthe service columnareserviceinstancesofon-demandRabbitMQinthespace.
$ cf servicesGetting services in org my-org / space my-space as [email protected] service plan bound apps last operationmy-instance ondemand-rabbitmq Solo create succeeded
Youcanbindyourapptoanexistinginstanceorcreateanewinstancetobindtoyourapp.
CreateaServiceInstanceUnlikepre-provisionedservices,on-demandservicesarecreatedasynchronously,notimmediately.The watch commandshowsyouwhenyourserviceinstanceisreadytobindanduse.
Tocreateaninstanceoftheon-demandRabbitMQforPCFservice,run cf create-service
:
1. Enter cf create-service ondemand-rabbitmq Solo SERVICE_INSTANCE
Where SERVICE_INSTANCE isanameyouchoosetoidentifytheserviceinstance.Thisnamewillappearunder service [sic]inoutputfrom cf
services .
2. Enter watch cf services andwaitforthe last operation foryourinstancetoshowas create succeeded .
$ cf create-service ondemand-rabbitmq Solo my-instance
Creating service my-instance in org my-org / space my-space as [email protected]
$ watch cf services
Getting services in org my-org / space my-space as [email protected] service plan bound apps last operationmy-instance ondemand-rabbitmq Solo create succeeded
Ifyougetanerror,seeTroubleshootingInstances.
© Copyright Pivotal Software Inc, 2013-2018 70 1.10
BindaServiceInstancetoYourAppForanapptouseaservice,youmustbindittoaserviceinstance.Dothisafteryoupushorre-pushtheappusing cf push .
TobindanapptoaRabbitMQinstancerun $ cf bind-service
.
1. Enter cf bind-service APP SERVICE_INSTANCE
Where APP istheappyouwanttousetheRabbitMQserviceinstanceand SERVICE_INSTANCE isthenameyousuppliedwhenyouran cf create-
service .
$ cf bind-service my-app my-instance
Binding service mydb to my-app in org my-org / space test as [email protected]: Use 'cf push' to ensure your env variable changes take effect
UsetheRabbitMQServiceinYourAppToaccesstheRabbitMQservicefromyourapp:
1. Run cf env APP_NAME withthenameoftheappboundtotheRabbitMQforPCFinstance.
2. Intheoutput,notetheconnectionstringslistedinthe VCAP_SERVICES > credentials objectfortheapp.
3. Inyourappcode,calltheRabbitMQserviceusingtheconnectionstrings.
ForhowtocodeyourapptouseRabbitMQmessaging,seeAboutUsingPivotalRabbitMQ>ClientDocumentationintheRabbitMQdocumentation .
RestageanAppwithaNewServiceInstanceIfaserviceinstancehasbeenupdatedbasedonanewversionoftheservice,youneedtorun cf restage torestageyourapptousethenewinstance.
1. Enter cf restage-app APP
Where APP istheappyouwanttousetheupdatedserviceinstance.
$ cf restage-app my-app
Whenanewversionofaserviceobsoletesanolderversion,theplatformoperatormayaskyoutoupdateyourinstancesoftheserviceandrestageanyappsboundtotheserviceinstances.
Pushingnewversionofanappautomaticallyrestagestheapponanyserviceinstancesitisboundto.
UnbindaServiceInstancetoYourAppTostopanappfromusingaserviceitnolongerneeds,unbinditfromtheserviceinstanceusing cf unbind-
service.
1. Enter cf unbind-service APP SERVICE_INSTANCE
Where APP istheappyouwanttostopusingtheRabbitMQserviceinstanceand SERVICE_INSTANCE isthenameyousuppliedwhenyouran cf
create-service .
$ cf unbind-service my-app my-instance
Unbinding app my-app from service my-instance in org my-org / space my-space as [email protected]
DeleteaServiceInstance
© Copyright Pivotal Software Inc, 2013-2018 71 1.10
Todeleteaserviceinstance,run cf delete-service
.
1. Enter cf delete-service SERVICE_INSTANCE
Where SERVICE_INSTANCE isthenameoftheservicetodelete.
$ cf delete-service my-instance
Are you sure you want to delete the service my-instance ? yDeleting service my-service in org my-org / space my-space as [email protected]
2. Enter watch cf service SERVICE_INSTANCE andwaitfora Service instance not found errorindicatingthattheinstancenolongerexists.
Youcannotdeleteaserviceinstancethatanappisboundto.
CreateanAdminUserforaServiceInstanceIfyouwanttogetadminprivilegestotheRabbitMQManagementUI,youcancreateanadminuserforaserviceinstance,andobtainusercredentialsthatyoucansharewithotherappdevelopers.
Bothoperatorsandappdeveloperscanusethisprocedure.Forinstructions,seeCreateanAdminUserforaServiceInstance.
FederateExchangesandQueuesYoucanfederateexchangesandqueuesinRabbitMQforPCF,asyouwouldinanyRabbitMQdeployment.
Tofederateexchangesandqueues,dothefollowing:
1. CreateaservicekeybyfollowingtheinstructionsinCreateanAdminUserforaServiceInstance.
Theoutputoftheaboveprocedurereturnsadminusercredentials,alongwithotherdata.
2. Intheoutputfromtheabovestep,lookforthe uris array.Itwillhavethispattern:
{..."uri": "amqp://USERNAME:PASSWORD@IP_ADDRESS/VHOST","uris": ["amqp://USERNAME:PASSWORD@IP_ADDRESS/VHOST"],...}
Forexample:
{..."uri": "amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f: [email protected]:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7","uris": ["amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f: [email protected]:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7","amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f: [email protected]:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7"]},...}
3. Setupfederationasyounormallywould,usingtheRabbitMQManagementUIorAPI,withtheURIsfoundinthe uris arrayyougotfromthestepabove.
Forinstructionsonfederation,seetheRabbitMQdocumentation .
ShovelExchangesandQueues
© Copyright Pivotal Software Inc, 2013-2018 72 1.10
YoucanshovelexchangesandqueuesinRabbitMQforPCF,asyouwouldinanyRabbitMQdeployment.
Toshovelexchangesandqueues,dothefollowing:
1. CreateaservicekeybyfollowingtheinstructionsinCreateanAdminUserforaServiceInstance.
Theoutputoftheaboveprocedurereturnsadminusercredentials,alongwithotherdata.
2. Intheoutputfromtheabovestep,lookforthe uris array.Itwillhavethispattern:
{..."uri": "amqp://USERNAME:PASSWORD@IP_ADDRESS/VHOST","uris": ["amqp://USERNAME:PASSWORD@IP_ADDRESS/VHOST"],...}
Forexample:
{..."uri": "amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f: [email protected]:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7","uris": ["amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f: [email protected]:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7","amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f: [email protected]:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7"]},...}
3. Setupshovelasyounormallywould,usingtheRabbitMQManagementUIorAPI,withtheURIsfoundinthe uris arrayyougotfromthestepabove.
Forshovelinstructions,seetheRabbitMQdocumentation .
© Copyright Pivotal Software Inc, 2013-2018 73 1.10
RabbitMQ®EnvironmentVariablesThistopicprovidesareferencefortheenvironmentvariablesthatCloudFoundrystoresforRabbitMQforPCFserviceinstances.Thesevariablesincludethecredentialsthatappsusetoaccesstheserviceinstances.
VCAP_SERVICESApplicationsrunninginCloudFoundrygainaccesstotheboundserviceinstancesviaanenvironmentvariablecredentialshashcalledVCAP_SERVICES .Anexamplehashisshowbelow:
{ "p-rabbitmq": [{ "label": "p-rabbitmq", "name": "my-rabbit-service-instance", "plan": "standard", "tags": ["rabbitmq", "messaging", "message-queue", "amqp", "pivotal"], "credentials": { "dashboard_url": "http://pivotal-rabbitmq.your.pcf.example.com/#/login/b5d0ad14-4352-48e8-8982-d5b1d257029f/tavk86pnnns1ddiqpsdtbchurn", "username": "b5d0ad14-4352-48e8-8982-d5b1d257029f", "vhost": "62e5ab21-7b38-44ac-b139-6aa97af01cd7", "password": "#passwordexample123456789", "ssl": false, "hostname": "10.0.0.41", "hostnames": [ "10.0.0.41", "10.0.0.51"], "uri": "amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]/62e5ab21-7b38-44ac-b139-6aa97af01cd7", "uris": [ "amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]/62e5ab21-7b38-44ac-b139-6aa97af01cd7", "amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]/62e5ab21-7b38-44ac-b139-6aa97af01cd7"], "http_api_uri": "http://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:15672/api", "http_api_uris": [ "http://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:15672/api", "http://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:15672/api"], "protocols": { "amqp": { "password": "passwordexample123456789", "port": 5672, "ssl": false, "username": "b5d0ad14-4352-48e8-8982-d5b1d257029f", "vhost": "62e5ab21-7b38-44ac-b139-6aa97af01cd7", "host": "10.0.0.41", "hosts": [ "10.0.0.41", "10.0.0.51"], "uri": "amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7", "uris": [ "amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7", "amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7"]}, "management": { "username": "b5d0ad14-4352-48e8-8982-d5b1d257029f", "password": "passwordexample123456789", "path": "/api", "port": 15672, "ssl": false, "host": "10.0.0.41", "hosts": [ "10.0.0.41", "10.0.0.51"], "uri": "http://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:15672/api", "uris": [ "http://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:15672/api", "http://b5d0ad14-4352-48e8-8982-d5b1d257029f:[email protected]:15672/api"]}}}}]}
Youcansearchforyourservicebyits name ,givenwhencreatingtheserviceinstance,ordynamicallyviathe tags or label properties.The credentials
propertycanbeusedasfollows:
Thetoplevelproperties uri , uris , vhost , username , password , hostname and hostnames provideaccesstotheAMQP0.9.1protocol.
Amoreflexibleapproachisprovidedbythe credentials.protocols property,whichhasakeyperenabledprotocol.Thepossiblekeysare amqp ,management , mqtt ,and stomp .IfSSLisenabled,thenthekeyswillbe amqp+ssl , management+ssl , mqtt+ssl ,and stomp+ssl respectively.
Thevaluesassociatedwitheachofthesekeysgivesaccesscredentialsspecifictoeachprotocol.Inallcases,URIsareprovided,alongwiththeindividualcomponents.
© Copyright Pivotal Software Inc, 2013-2018 74 1.10
ChangingEnabledPluginsandProtocols
IfyouadjustthepluginsandprotocolsenabledforRabbitMQ,youmayneedtoforceallapp’s VCAP_SERVICES environmentvariabletoberegenerated.Addingandremovingthefollowingpluginsrequireboundapplicationstoberestaged:
rabbitmq_management
rabbitmq_stomp
rabbitmq_mqtt
rabbitmq_amqp1_0
IncommonwithallservicesinPivotalCloudFoundry (PCF),the VCAP_SERVICES environmentvariableforanapplicationisonlymodifiedwhentheapplicationisboundtoaserviceinstance.Userswillneedto cf unbind-
service, cf bind-
serviceand cf restage theirappinthisscenario.
Note:Removingoraddingplugins/protocolsmaycauseappsboundwithRabbitMQtobreak.
© Copyright Pivotal Software Inc, 2013-2018 75 1.10
TroubleshootingInstancesThistopicprovidesbasicinstructionsforappdeveloperstroubleshootingOn-DemandRabbitMQ®forPCF.
ErrorsYoumayseeanerrorwhenusingtheCloudFoundryCommand-LineInterface(cfCLI)toperformbasicoperationsonaRabbitMQforPCFserviceinstance:
cf create
cf update
cf bind
cf unbind
cf delete
ParseaCloudFoundry(CF)ErrorMessageFailedoperations(create,update,bind,unbind,delete)resultinanerrormessage.YoucanretrievetheerrormessagelaterbyrunningthecfCLIcommand cf service INSTANCE-NAME .
$ cf service myservice
Service instance: myserviceService: super-dbBound apps:Tags:Plan: dedicated-vmDescription: Dedicated InstanceDocumentation url:Dashboard:
Last OperationStatus: create failedMessage: Instance provisioning failed: There was a problem completing your request. Please contact your operations team providing the following information: service: redis-acceptance, service-instance-guid: ae9e232c-0bd5-4684-af27-1b08b0c70089, broker-request-id: 63da3a35-24aa-4183-aec6-db8294506bac, task-id: 442, operation: createStarted: 2017-03-13T10:16:55ZUpdated: 2017-03-13T10:17:58Z
Usetheinformationinthe Message fieldtodebugfurther.ProvidethisinformationtoPivotalSupportwhenfilingaticket.
The task-id fieldmapstotheBOSHtaskID.FormoreinformationonafailedBOSHtask,usethe bosh task TASK-ID .
The broker-request-guid mapstotheportionoftheOn-DemandBrokerlogcontainingthefailedstep.Accessthebrokerlogthroughyoursyslogaggregator,oraccessBOSHlogsforthebrokerbytyping bosh logs broker 0 .Ifyouhavemorethanonebrokerinstance,repeatthisprocessforeachinstance.
RetrieveServiceInstanceInformation1. Logintothespacecontainingtheinstanceorfailedinstance.
$ cf login
2. Ifyoudonotknowthenameoftheserviceinstance,run cf services toseealistingofallserviceinstancesinthespace.Theserviceinstancesarelistedinthe name column.
© Copyright Pivotal Software Inc, 2013-2018 76 1.10
$ cf servicesGetting services in org my-org / space my-space as [email protected] service plan bound apps last operationmy-instance ondemand-rabbitmq Solo create succeeded
3. Run cf service SERVICE-INSTANCE-NAME toretrievemoreinformationaboutaspecificinstance.
4. Run cf service SERVICE-INSTANCE-NAME --guid toretrievetheGUIDoftheinstance,whichisusefulfordebugging.
RetrieveRabbitMQInstanceCredentialsIfyouwanttoaccesstheManagementDashboardortheRabbitMQserverfortroubleshooting,youcancreateanewservice-keytoretrieveRabbitMQinstancecredentials.Pivotalrecommendsthatyouusethiskeyfortroubleshootingonly,andthatyoudeletethekeyaftertroubleshooting.Toretrievethecredentials,dothefollowing:
1. Createaservice-keyforyourRabbitMQinstanceusingthecommand cf create-service-key INSTANCE-NAME SERVICE-KEY-NAME .
2. Retrievethecredentialsusingthecommand cf service-key INSTANCE-NAME SERVICE-KEY-NAME .
Forexample:
$ cf create-service-key my-rmq-instance my-keyCreating service key my-key for service instance my-rmq-instance as admin...OK$ cf service-key my-rmq-instance my-keyGetting key my-key for service instance my-rmq-instance as admin...{ "host": "10.0.8.4", "password": "", "port": 6379}
SelecttheBOSHDeploymentforaServiceInstanceThisisanadditionaltroubleshootingoptionforBOSHCLIv1only.ItdoesnotapplytotheBOSHCLIv2.
1. RetrievetheGUIDofyourserviceinstancewiththecommand cf service YOUR-SERVICE-INSTANCE --guid .
2. TodownloadyourBOSHmanifestfortheservice,run bosh download manifest service-instance_SERVICE-INSTANCE-GUID myservice.yml usingtheGUIDyoujustobtainedandafilenameyouwanttousewhensavingthemanifest.
3. Run bosh deployment MY-SERVICE.yml toselectthedeployment.
KnowledgeBase(Community)FindtheanswertoyourquestionandbrowseproductdiscussionsandsolutionsbysearchingthePivotalKnowledgeBase .
FileaSupportTicketYoucanfileasupporttickethere .Besuretoprovidetheerrormessagefrom cf service YOUR-SERVICE-INSTANCE .
Tohelpexpeditetroubleshooting,ifpossiblealsoprovideyourservicebrokerlogs,serviceinstancelogs,andBOSHtaskoutput.Yourcloudoperatorshouldbeabletoobtainthesefromyourerrormessage.
DeleteRabbitMQInstanceOn-DemandBrokerprovidesaBOSHcommandtodeletealltheOn-DemandBrokerdeployedinstances.Todeletetheinstances,dothefollowingprocedure:
© Copyright Pivotal Software Inc, 2013-2018 77 1.10
1. ForBOSHCLIv1only:SetthedeploymenttotheOn-DemandBrokerinBOSH.
2. RunoneofthefollowingcommandstodeleteallinstancesoftheOn-DemandBroker,dependingonyourOpsManagerversion:
Forv1.10orearlier: bosh run errand delete-sub-deploymentsForv1.11orlater: bosh2 run-errand delete-sub-deployments
WARNING:Thiscommanddeletesdeploymentinstancesserially.Itisverydestructiveandtheoperationcannotbeundone.
© Copyright Pivotal Software Inc, 2013-2018 78 1.10
InstallingandConfiguringthePre-ProvisionedServiceThistopicprovidesinstructionstoPivotalCloudFoundry(PCF)operatorsabouthowtoinstall,configure,anddeploytheRabbitMQforPCFtiletoprovideapre-provisionedservice.
TheRabbitMQopensourceproductprovidesadditionaldocumentation.FormoreinformationaboutgettingstartedwithRabbitMQandensuringproductionreadiness,seetheProductionChecklistintheRabbitMQDocumentation .
DownloadandInstalltheTile1. DownloadtheproductfilefromPivotalNetwork .
2. NavigatetotheOpsManagerInstallationDashboardandclickImportaProducttouploadtheproductfile.
3. UndertheImportaProductbutton,click+nexttotheversionnumberofRabbitMQforPCF.Thisaddsthetiletoyourstagingarea.
4. ClickthenewlyaddedRabbitMQforPCFtile.Thisletsyoubeginconfiguringthetile.Theinstallationiscompletewhenyouapplythechangesfromtheconfiguration.
ConfigurePre-ProvisionedRabbitMQforPCFTheconfigurationscreenbelowappearswhenyouclicktheRabbitMQforPCFtileinOpsManager.Anorangecirclebesideatabindicatesthatyoumustcompleteaconfigurationinthetab.Agreencheckmarkindicatesthatthetabispreconfiguredandyoumayoptionallychangeitssettings.
Note:Forinstructionsabouthowtoinstall,configure,anddeploytheRabbitMQforPCFtileasanon-demandservice,seeInstallingandConfiguringRabbitMQforPCFasanOn-DemandService.
© Copyright Pivotal Software Inc, 2013-2018 79 1.10
WhichSettingsTabstoConfigureforthePre-ProvisionedServiceConfigurethefollowingtabsforthepre-provisionedservice:
RabbitMQSettingsTab Instructions
AssignAZsandNetworks AssignAZsandNetworks
RabbitMQ RabbitMQ
Syslog Syslog
DedicatedInstance:SingleNodePlan DedicatedInstance:SingleNodePlan
Errands Errands
Stemcell Stemcell
AssignAZsandNetworksFollowthestepsbelowtoconfiguretheAZsandnetworks.
1. IntheSettingsscreen,clickAssignAZsandNetworks.
2. ConfigurethefieldsontheAssignAZsandNetworksasfollows.Allfieldsarerequired,thoughsomedonotapplytothepre-provisionedservice.
RequiredFields Instructions
Placesingletonjobsin
Selectaregion.Thisselectiononlyaffectstheon-demandservice.
Balanceotherjobsin
Selectadditionalregion(s).Thisselectiononlyaffectsthepre-provisionedservice.
Network
SelectanetworkfortheRabbitMQBroker.
ThisshouldbeaseparatenetworkfromtheoneyouselectforServiceNetwork.ThisnetworkisrepresentedbytheDefaultNetwork,describedinDefaultNetworkandServiceNetwork.Typically,youselectthenetworkusedforthePivotalElasticRuntimecomponents.
ServiceNetwork
Selectanetwork.Thisselectiononlyaffectstheon-demandservice.
3. ClickSave.
RabbitMQToconfigurethefollowingsectionsintheRabbitMQtab,intheSettingsscreen,clickRabbitMQ.
RabbitMQAdminUserCredentialsEnteranadminusernameandpasswordforRabbitMQ.ThisgrantsyoufulladminaccesstotheRabbitMQManagementUI.
EnterthedesiredcredentialsinthissectionoftheRabbitMQtab:
Important:YoucannotchangetheregionsornetworksafteryouhaveclickedApplyChangesinthefinalstepbelow.
WARNING:ChangingtheNetworkafteryouhaveconfiguredit,orchangingtheIPconfiguration,resultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .
© Copyright Pivotal Software Inc, 2013-2018 80 1.10
PluginsChoosewhichpluginsyouwanttoenableinthissectionoftheRabbitMQtab.
Youmustleavetherabbitmq_managementpluginenabledforthisproducttowork.
FormoreinformationaboutRabbitMQplugins,seetheRabbitMQdocumentation .
HAProxyPortsEntertheportsHAProxyshouldloadbalancetotheRabbitMQnodesinthissectionoftheRabbitMQtab:
Allthedefaultportsofalltheavailablepluginsareload-balancedbydefault.However,ifyouinstallextraprotocolplugins,orprovideacustomconfigurationthatchangestheportsRabbitMQlistenson,thenyoumustupdatethelistofload-balancedports.
Youmustleavethemanagementpluginlisteningonport 15672 andloadbalancethatport.
IfyouchangethetopologyofyourRabbitMQcluster,theHAProxyisautomaticallyreconfiguredduringthedeployment.
SSL(Optional)ProvideSSLcertificatesandkeysforusebytheRabbitMQclusterinthissectionoftheRabbitMQtab:
Note:Torotateyouradministratorcredentials,enteranewusernameandpassword,saveyouroptions,andredeploybyreturningtotheOpsManagerInstallationDashboardandclickingApplyChanges.
© Copyright Pivotal Software Inc, 2013-2018 81 1.10
SSLissimultaneouslyprovidedontheAMQPSport(5671)andthemanagementport(15672).NootherpluginsareautomaticallyconfiguredforusewithSSL.
IfyouprovideSSLkeysandcertificatesnon-SSLsupportisdisabled.
SSLsettingsareappliedequallyacrossallVMsinthecluster.
FormoreinformationaboutSSLsupport,seetheRabbitMQdocumentation .
ErlangCookie(Optional)ProvideanErlangcookietobeusedbytheclusterinthissectionoftheRabbitMQtab.ThisisusefulifyouwanttoconnectdirectlytotheRabbitMQcluster,forexamplewith rabbitmqctl ,ortoconnectotherVMsrunningErlang.
KnownIssueswithErlangCookie
TherearetwoknownissuesassociatedwiththeErlangcookiewhenyoudothefollowing:
ClusterScaling
ChangingtheErlangCookieValue
ClusterScalingKnownIssue
IfyouhavenotsettheErlangcookieandyouwanttoscaleoutyourclustersizewithoutdowntime,followthesesteps:
1. Followthestepsinoneofthelinksbelow,uptoandincludingthesectionLogintotheBOSHDirector:
© Copyright Pivotal Software Inc, 2013-2018 82 1.10
ForOpsManagerv1.10orearlier:AdvancedTroubleshootingwiththeBOSHCLI
ForOpsManagerv1.11orlater:PreparetoUsetheBOSHCLI
2. RunoneofthefollowingcommandsdependingonyourOpsManagerversion:
ForOpsManagerv1.10orearlier: bosh ssh rabbitmq-server/0ForOpsManagerv1.11orlater: bosh2 ssh rabbitmq-server/0
3. Runthefollowingcommands:
sudo -iecho $(cat /var/vcap/store/rabbitmq/.erlang.cookie)
4. PastethevaluereturnedfromthelastcommandintotheErlangcookiefieldintheRabbitMQtab.Thisfieldisshownabove.
5. Toincreasethesizeofyourcluster,navigatetotheResourceConfigtaband,inthefirstrow,raisethevalueforthenumberofInstancesoftheRabbitMQnode.
6. ReturntotheOpsManagerInstallationDashboard,andclickApplyChanges.
ChangingtheErlangCookieValueKnownIssue
ChangingtheErlangcookievaluerequiresclusterdowntime.Pivotalstronglyrecommendsthatyoudonotchangeanythingelseduringthistime,becauseitispossiblefortheconfigurationtobeinconsistentlyappliedduringthisprocess.
Thedeploymentmayfailafterthisprocess.Ifso,redeployingfixestheissue.
RabbitMQConfiguration(Optional)Provideafull rabbitmq.config filebypastingitscontentsintheRabbitMQconfigurationfieldintheRabbitMQtab.This rabbitmq.config fileisthenprovidedtoallthenodesinthecluster.
TheinputinthisfieldmustbeBase64encoded.
Forexample,supposeyouwanttoconfigurethe rates_mode ofthe rabbitmq_management statsbelow:
[ {rabbitmq_management, [ {rates_mode, detailed} ]}].
1. EncodethefileintoBase64:
WwogIHtyYWJiaXRtcV9tYW5hZ2VtZW50LCBbCiAgICB7cmF0ZXNfbW9kZSwgZGV0YWlsZWR9CiAgXX0KXS4K
2. PastetheaboveintotheRabbitMQconfigurationfield:
Note:BOSHtellsyouthatthecookiehaschanged—thisisbecausethedefaultvalueinthemanifestisempty,whichresultsinanauto-generatedcookie.However,thevalueofthecookieontheserverremainsthesame,sotheknownissuebelowdoesnotapply.
© Copyright Pivotal Software Inc, 2013-2018 83 1.10
Youcanseeanexample rabbitmq.config filehere .FormoreinformationabouttheRabbitMQconfiguration,seetheRabbitMQdocumentation .
TLSSupportConfigureTLSinthissectionoftheRabbitMQtab:
TLSv1.0isdisabledbydefault,duetosecurityissues.
TLSv1.1andv1.2areenabledbydefaultandcanbeturnedonandoff.
ExternalLoadBalancer(Optional)EnteraDNSnameorIPaddressofanexternalloadbalancertobereturnedinthebindingcredentials( VCAP_SERVICES )toapplicationdevelopers.EnterthisinthissectionoftheRabbitMQtab:
Ifyouconfigureanexternalloadbalancer,toavoidanunnecessaryVMdeployment,intheResourceConfigtabsettheHAProxyforRabbitMQinstancecountto0.
MetricsPollingIntervalThemetricspollingintervalissetinthissectionoftheRabbitMQtab:
Thedefaultsettingis30secondsforalldeployedcomponents.Pivotalrecommendsthatyoudonotchangethisinterval.Inordertoavoidoverwhelmingcomponents,donotsetthisbelow10seconds.
Changingthissettingaffectsalldeployedinstances.
DiskFreeAlarmLimitChoosehowmuchdiskspaceRabbitMQattemptstokeepfreeatanygiventimeinthissectionoftheRabbitMQtab:
© Copyright Pivotal Software Inc, 2013-2018 84 1.10
RabbitMQperiodicallychecksifthereissufficientfreespaceondisk.Ifthereisnot,RabbitMQtemporarilystopsacceptingnewmessages.Thisgivesyourappstimetoconsumeexistingmessages,andthusfreeupsomediskspace.TheRabbitMQtileprovidesfouroptionsforthisvalue:
50MBistheminimumvalue.(NotRecommended)Selecting50MBisnotrecommendedandcancausedataloss.Formoreinformation,seeDangersofSettingThisValueTooLowandWhentoUsethe50MBValuebelow.
100%MemoryensuresthatatthetimewhenRabbitMQcheckstheavailabledisk,theremustbeenoughspaceforRabbitMQtopageallmemory-basedmessagesouttodisk.
150%Memoryisrecommended.Thisisbecauseitispossiblethatinbetweendisk-spacechecks,RabbitMQmay:
Writepersistentmessagestodisk(usingupsomediskspace).Acceptmorememory-basedmessagesintovariousqueues.Pageallmemory-basedmessagestodisk.
Intheabovesituations,RabbitMQmightrequiremorefreediskthanithasmemory.
200%MemoryisaconservativevalueusedwhentheoperatorwantshigherconfidencethatRabbitMQneverrunsoutofdiskspace.
Formoreinformationaboutdiskalarms,seetheRabbitMQdocumentation .
DangersofSettingThisValueTooLow
IfthediskofagivenRabbitMQnodecompletelyfillswhileRabbitMQisrunning,thatnodecrashes.Thiscanleadtodataloss,andlossofavailability.
RabbitMQreservestherighttopageanyandallmessagesinmemory(eventransientmessages)todiskatanytime.YoumustsetyourDiskfreealarmlimithighenoughtoensurethatRabbitMQalwayshasatleastenoughspacetodothis.
DisadvantagesofSettingThisValueTooHigh
IfyousetyourDiskfreealarmlimittoavaluelargerthanthesizeofyourpersistentdisk,thenRabbitMQisnotabletofreeupenoughdiskspacetoacceptnewmessages.EnsurethatyouhavealargeenoughdisktopersistallthemessagesyouintendtopersistwhilealsoleavingenoughspacefreetosatisfytheDiskfreealarmlimitthatyouchoose.
WhentoUsethe50MBValue
Pivotaldoesnotrecommendusingthisvalueinproduction.However,ifyouareexperimentingwithadevelopmentenvironmentyoumaywanttouseasmalldisktokeepdowncosts,thoughthisincreasesthepossibilitythatRabbitMQmaycrashandlosedata.
SyslogSpecifyasyslogdestinationintheSyslogtab.
ToenablemonitoringforRabbitMQforPCF,operatorsforwardthesyslogbydesignatinganexternalsyslogendpointforRabbitMQcomponentlogmessages.ThisendpointservesastheinputtoamonitoringplatformsuchasDatadog,Papertrail,orSumoLogic.
TospecifythedestinationforRabbitMQforPCFlogmessages,dothefollowing:
1. FromtheOpsManagerInstallationDashboard,clicktheRabbitMQtile.
2. IntheRabbitMQtile,clicktheSettingstab.
© Copyright Pivotal Software Inc, 2013-2018 85 1.10
3. ClickSyslog.
4. ConfigurethefieldsontheSyslogpaneasfollows:
Option Description
Syslogaddress IPorDNSaddressofthesyslogserver
Syslogport Portofthesyslogserver
Transportprotocol
Transportprotocolofthesyslogserver.Oneof udp , tcp , relp .
Formatforlogs Formatforlogs.Pivotalrecommends RFC 5424 ,but Legacy Format canbeusedforcompatibilityreasons.
EnableTLS EnableTLStothesyslogserver.
Ifthereareseveralpeerserversthatcanrespondtoremotesyslogconnections,thenyoumayprovideawildcardinthe
© Copyright Pivotal Software Inc, 2013-2018 86 1.10
PermittedPeer domain,suchas *.example.com .
CustomCACertificate
Iftheservercertificateisnotsignedbyaknownauthority,forexample,aninternalsyslogserver,providetheCAcertificateofthelogmanagementserviceendpoint.
5. ClickSave.
6. ReturntotheOpsManagerInstallationDashboardandclickApplyChangestoredeploywiththechanges.
DedicatedInstance:SingleNodePlanThistabonlyappliestotheon-demandservice.However,youmustcompletethefieldsonthistabevenifyouarenotusingtheon-demandservice.Therefore,ifyouarenotusingtheon-demandservice:
Selectorenteranyvaluesintherequiredfields,selecttheAcknowledgecheckbox,andclickSave.
Forinformationonconfiguringtheon-demandservice,seeInstallingandConfiguringtheOn-DemandService.
Errands(Optional)IntheErrandstab,choosethedefaultsforwhenerrandsrun.
Errandscanbethoughtofastasks.Forexample,whendeployingorupdatingRabbitMQforPCF,OpsManagercanoptionallyrunaseriesofpost-deployerrands.AnexampleistheSmokeTestserrand,whichchecksthehealthoftheRabbitMQclusterafteradeployorupgrade.
Youcandecidewhethertorunpost-deployerrandsbytogglingthemonoroffbeforeyouclickApplyChangestoupdateaconfigurationintheOpsManagerInstallationDashboard:
Thisisaone-timeactionbeforeanupdate.YoucanchangetheabovedefaultsintheErrandstab,aswellasthedefaultsforpre-deleteerrands.
Formoreinformationonerrandrunrules,seeErrandRunRules .
Post-DeployErrands
Errand Description
BrokerRegistrar
Makesthepre-provisionedRabbitMQserviceplansavailableintheMarketplace
SmokeTestsChecksthatapre-provisionedRabbitMQserviceinstancecanbeboundtoaCloudFoundryapp,andthattheappcanpublishandsubscribetoaRabbitMQcluster.SeePre-ProvisionedInstanceSmokeTestsbelow.
RegisterOn-Demand Makestheon-demandRabbitMQserviceplansavailableintheMarketplace.IfyouchangetheServicePlanConfiguration,youmustrun
Important:InRabbitMQforPCFv1.9.0andlater,allpost-deployerrandsareonbydefault.Pivotalrecommendskeepingthesedefaults,becausethesmoketestscanencounterunexpectedissues,andon-demandinstancesofRabbitMQforPCFmayfallbehindiftheUpgradeAllServiceInstanceserrandisnotonbydefault.
© Copyright Pivotal Software Inc, 2013-2018 87 1.10
ServiceBroker
thiserrandinorderforthechangestobereflectedintheMarketplace.
DedicatedInstanceSmokeTests
Checksthaton-demandRabbitMQserviceinstancescanbeboundtoaCloudFoundryapp,andthattheappcanpublishandsubscribetoaRabbitMQcluster.SeeOn-DemandInstanceSmokeTests.
UpgradeAllServiceInstances
On-demandinstancesareupdatedandredeployediftherearechangestotheDedicatedInstancesettingsorthetileisupgraded.IfthiserrandissettoOfforWhenChanged,updatestoDedicatedInstancesettingswillnotbeappliedtoexistingserviceinstances.Pivotalstronglyrecommendsthatthiserrandisconfiguredtoalwaysrun.
Pre-ProvisionedInstanceSmokeTestsSmoketestsrunasapost-deploymenterrand.Formoreinformationaboutthesmoketestsprocess,seeSmokeTests.
Pre-DeleteErrandsPre-deleteerrandsrunafteranoperatorchoosestodeleteaproductintheOpsManagerInstallationDashboard,butbeforeOpsManagerfinishesdeletingtheproduct.
Errand Description
BrokerDeregistrarRemovesthepre-provisionedRabbitMQservicefromtheMarketplaceanddeletesallassociatedserviceinstancesandbindings
DeleteAllServiceInstancesUnbindsanddeletesexistingdedicatedserviceinstances.Thedurationofthiserranddependsonthenumberofdeployedon-demandinstances.
DeregisterOn-DemandServiceBroker
Removestheon-demandRabbitMQservicefromtheMarketplace
Stemcell1. ClickStemcell.
2. Ifprompted,importanewstemcellversion.Formoreinformation,seetheinformationaboutimportingthestemcellforyourIaaS:AWS ,Azure
,GCP ,orvSphere .
ApplyConfigurationandCompletetheInstallationReturntotheOpsManagerInstallationDashboardandclickApplyChangestocompletetheinstallationofRabbitMQforPCF.
OtherConfigurationTopics
ConnectingtoaHighlyAvailableRabbitMQClusterTheRabbitMQtile,allowsforahighlyavailableclusterthroughmultipleHAProxynodes.The hostnames , uris and hosts propertieshavebeenaddedandshouldbeusedinpreferenceovertheequivalentsingularproperties.Thesingularpropertiesaremaintainedforbackwardscompatibilityandalwayscontainthefirstvaluefromtheequivalentpluralproperty.Thesingularpropertieswilleventuallybedeprecated.
Forexample,withtwoHAProxyjobsdeployed,thefollowingpropertieswillbepresent:
"hostname": "10.0.0.41","hostnames": [ "10.0.0.41", "10.0.0.51"]
© Copyright Pivotal Software Inc, 2013-2018 88 1.10
Porttoprotocolmappings15672=Managementdashboard
5672=RabbitMQ
5671=RabbitMQSSL
1883=MQTT
8883=MQTTSSL
61613=STOMP
61614=STOMPSSL
15674=WebSTOMP
4567=RabbitMQServiceBroker
3457-3459=CFLoggregator
SecurityGroupsToenableaccesstotheRabbitMQtileservice,youmustensureyoursecuritygroupallowsaccesstotheHAProxyandRabbitMQServiceBrokerVMsconfiguredinyourdeployment.YoucanobtaintheIPaddressesforthesefromtheOpsManagerStatuspagefortheRabbitMQtile.EnsurethefollowingportsareenabledforthoseVMs:
15672
5672
5671
1883
8883
61613
61614
15674
4567
3457-3459
ThefollowingisatemplateforconfiguringyourCloudFoundrysecuritygroups:[ {"protocol":"tcp","destination":"<haproxy-node-IP-addresses>","ports":"5671,5672,1883,8883,61613,61614,15672,15674"}, {"protocol":"tcp","destination":"<service-broker-node-IP-addresses>","ports":"4567"} ]
ApplicationSecurityGroupsToallowthisservicetohavenetworkaccess,youmustcreateApplicationSecurityGroups (ASGs).
ApplicationContainerNetworkConnectionsApplicationcontainersthatuseinstancesoftheRabbitMQservicerequirethefollowingoutboundnetworkconnections:
Destination Ports Protocol Reason
HAProxy IPs 5672 tcp ApplicationcontainersusingAMQP
HAProxy IPs 5671 tcp ApplicationcontainersusingAMQPoverSSL
HAProxy IPs 1883 tcp ApplicationcontainersusingMQTT
HAProxy IPs 8883 tcp ApplicationcontainersusingMQTToverSSL
HAProxy IPs 61613 tcp ApplicationcontainersusingSTOMP
HAProxy IPs 61614 tcp ApplicationcontainersusingSTOMPoverSSL
Note:TheserviceisunusablewithoutApplicationSecurityGroups.
© Copyright Pivotal Software Inc, 2013-2018 89 1.10
HAProxy IPs 61613 tcp ApplicationcontainersusingWebSTOMPDestination Ports Protocol Reason
CreateanASGnamed rabbitmq-app-containers withtheaboveconfigurationandbindittoeither:
Theappropriatespace
The default-running ASGsetifyouwanttoprovideaccesstoallstartedapps.Thenrestartyourapps.
Ifyouareusinganexternalloadbalancer,orhavemorethanoneIPaddressforHAProxy,youmustalsocreateegressrulesforthese.Forexample:
[ { "ports": "5671-5672", "protocol": "tcp", "destination": "10.10.10.10/32" }]
AssignedIPsRabbitMQforPCFdoesnotsupportchangingtheIPaddresseswhichhavebeenassignedtotheRabbitMQdeployments.Forexample,youcannotchangethesubnetintowhichtheRabbitMQclusterwasoriginallyprovisioned.Doingsocausesthedeploymenttofail.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .
PreservingDynamicallyAssignedIPsYoucannotswitchfromdynamicallyassignedIPaddressestoadifferentsetofstaticIPaddresses.However,youcanconfigureOpsManagersothecurrentsetofdynamicallyassignedIPaddressesalwayscontinuetobeused.Thismaybeusefulwhenupgrading.
Todothis,followthesesteps:
1. GototheStatuspageintheRabbitMQtile.
2. TakenoteoftheIPaddressesfortheRabbitMQServerandHAProxyforRabbitMQjobs,intheordernodesappearintheUI.
3. GototheSettingspage,andclickNetworking.
4. EntertheIPaddressesyougotfromtheStatuspageasacomma-separatedlist.
5. ClickSave.
RabbitMQServerSettingsthatCannotbeOverwrittenInallcases:
rabbit halt_on_upgrade_failure false
rabbitmq_mqtt subscription_ttl 1800000
log_levels [{connection,info}]
halt_on_upgrade_failure false
{rabbit, [ {collect_statistics_interval, 60000}] }
{rabbitmq_management, [ {rates_mode, none}] }
© Copyright Pivotal Software Inc, 2013-2018 90 1.10
WhenSSLisenabled:
rabbit tcp_listeners []
rabbit ssl_listeners [5671]
rabbitmq_management listener [{port,15672},{ssl,false}]
rabbitmq_mqtt ssl_listeners [8883]
rabbitmq_stomp ssl_listeners [61614]
© Copyright Pivotal Software Inc, 2013-2018 91 1.10
CreatingIsolationwiththeRabbitMQforPCFReplicator
OverviewRabbitMQforPCFReplicatorisatoolthatallowsyoutoinstallmultipleRabbitMQforPivotalCloudFoundry(PCF)tilesinasingleOpsManagerenvironment.Thisletsyourunmultiplepre-provisionedRabbitMQclustersthatareisolatedfromeachother.
Forexample,youmaywanttoisolatetheclusterservingSpringCloudServices(SCS)fromtheclusterservingappsintheMarketplace.Oryoumaywanttogiveacertainteamtheirowndedicated,pre-provisionedclusterthatyoumanageforthem.Forinformationonhowtoaccomplishthesescenarios,seeCommonUseCases.
CommonUseCasesTheimagebelowillustrateshowtoisolateSCSonRabbitMQforPCFfromclusteredandsinglenodeserviceinstancesdedicatedtodifferentteams.Inthisusecase:
TheunreplicatedRabbitMQforPCFtiledeploystwotypesofservices:
Apre-provisionedserviceisusedasabackingserviceforSCSAnon-demandserviceisusedtocreatethreecompletelyisolatedsinglenodeserviceinstances
Tworeplicatilesareusedtocreatetwodedicatedpre-provisionedclusters,witheachonededicatedtoaspecificteam.
ThesescenariosareexplainedbelowinRunningSCSonaDedicatedRabbitMQClusterandProvidingaPre-ProvisionedDedicatedCluster.
© Copyright Pivotal Software Inc, 2013-2018 92 1.10
RunningSCSonaDedicatedRabbitMQClusterReplicaRabbitMQtilescannotbeusedtoprovideabackingserviceforSpringCloudServices(SCS)becauseSCSexpectsthattheserviceiscalled
p-rabbitmq .Therefore,ifyouwanttoisolatetheRabbitMQclusterthatisusedbySCSfromothertenants,youcanreservetheunreplicatedRabbitMQforPCFtileforSCS,asshowninthediagrambelow.YoucanthenaddreplicaRabbitMQclustersforusebyappsintheMarketplace.
PivotalrecommendsthatyouusetheunreplicatedRabbitMQforPCFtilesolelyforSCStoavoidcontentionbetweenappsusingSCS,andappsusingRabbitMQforPCFintheMarketplace.
ToreservetheunreplicatedtileforSCS,turnofftheBrokerRegistrarerrandtopreventthebrokerfrombeingexposedintheMarketplace.Formoreinformation,seeErrands .
ToofferRabbitMQasacloudmessagingserviceintheMarketplace,createoneorseveralreplicas,installtheminOpsManager,andeitherallowthebrokerregistrarerrandtorun,orregistertheservicemanuallyusingCF .
ProvidingaPre-ProvisionedDedicatedClusterToreserveaRabbitMQclusterforusebyaspecificteam,allyouneedtodoisdisablethebrokerregistrarerrandinOpsManager .ThispreventsserviceregistrationintheMarketplace.Afteryoudeploythetile,manuallyexposetheservicebrokertoyourdesiredorgsandspaces.Forinstructions,seeRegisteraBroker .
UsingReplicasWhileOfferingtheOn-DemandRabbitMQServiceTheOn-Demandserviceisnotofferedinreplicatiles,sincethepurposeofthereplicatoristocreateadditionalpre-provisionedclusters.Ifyouwishtoofferon-demandserviceplans,usetheunreplicatedRabbitMQforPCFtileasshowninthediagramabove.
Blue-GreenUpgrades(Advanced)Inordertodoblue-greenstyleupgradestominimizedowntime,youcanstandupanewclusterandmigratedataandusersovertothenewclusteroveraperiodoftime.SpeakwithyourPlatformArchitectabouthowtoenablethisworkflow.
GeneratingReplicaTilesThistopicdescribeshowtoinstallthereplicatorandgeneratereplicatilesofRabbitMQforPCF.
PrerequisitesRabbitMQforPCFv1.8.xorv1.9.x
2.5GBoffreediskspace
DownloadtheReplicatorTheRabbitMQforPCFReplicatoriscurrentlyavailablefromPivotalNetwork .Searchforanddownloadthisarchive,andthenruntheenclosedbinary.
GenerateReplicaTilesThefollowingisthesyntaxforgeneratingareplicatile:
./rabbitmq-replicator-darwin\--name YOUR_DESIRED_TILE_NAME\--path PATH_TO_TILE\--output DESIRED_FILE_NAME.pivotal
Thefollowingaretheparametersexpectedintheabovesyntax:
© Copyright Pivotal Software Inc, 2013-2018 93 1.10
Parameter Description
nameThedesireduniqueidentifierforthereplicatile,whichisusedtogeneratemanifests,deploymentnames,andMarketplacenamefortheservice.Onlyalphanumericcharactersand - areacceptedbythetool.
path Thelocationoftheoriginal,unreplicated,RabbitMQforPCFsourcetilethatyoudownloaded
output Thedesiredfilenameandpathforthereplicatile
NamingConventionsinOriginalandReplicaTilesThetablebelowshowsthenamingconventionsforvariouscomponentsrelatedtotheoriginalRabbitMQforPCFtileandtothereplicatile.
Forthepurposesofthisexample,assumethatwhenyougenerateareplicatileasshownabove,inthe name fieldyouprovidethestring finance\ .Thentheattributesfortheoriginalandreplicatilesareasfollows:
Component NamewithOriginalTile NamewithReplicaTile
Brokername p-rabbitmq p-rabbitmq-finance
BrokerURL pivotal-rabbitmq-broker.YOUR_CF_DOMAIN pivotal-rabbitmq-broker-finance.YOUR_CF_DOMAIN
Servicename p-rabbitmq p-rabbitmq-finance
URLfortheRabbitMQManagementUIDashboard pivotal-rabbitmq.YOUR_CF_DOMAIN pivotal-rabbitmq-finance.YOUR_CF_DOMAIN
TiledisplaynameinOpsManager RabbitMQ RabbitMQ(finance)
TilenameusedinternallybyOpsManager p-rabbitmq p-rabbitmq-finance
Metrics/LoggingOrigin p-rabbitmq p-rabbitmq-finance
InstallingReplicaTilesAfteryouhavegeneratedareplicatile,youcanuploadittoOpsManagerasyouwouldanyothertile.Afteryouhaveuploadedit,followtheinstructionsforInstallingandConfiguringRabbitMQforPCFasaPre-ProvisionedService .TheOn-Demandserviceisnotofferedonreplicatiles.
LimitingAccesstoReplicaTilestoSpecificOrgsWhenyoureplicateRabbitMQforPCF,thereplicatilehastheBrokerRegistrarerrandsettoOnbydefault.ThisfieldappearsintheErrandstabinthetile:
Withanytile,iftheBrokerRegistrarerrandissettoOn,itrunsautomaticallywhenyoufinishinstallingthetileandcausesthetiletobeavailabletoallCForgs.
Ifyouwanttolimitaccesstothetiletoaspecificorg,followthesesteps:
1. SetthebrokerregistrarerrandtoOff,andapplyyourchanges.
2. ManuallyregisterthetilewithaspecificCForgusingthefollowingcommand.Seetheabovetablefor BROKER_NAME , BROKER_URL ,andSERVICE_NAME :
cf create-service-broker BROKER_NAME BROKER_USERNAME BROKER_PASSWORD BROKER_URL
© Copyright Pivotal Software Inc, 2013-2018 94 1.10
3. Togiveaccesstotheorg,usethefollowingcommandandrepeatforeachadditionalorg:
cf enable-service-access SERVICE_NAME -o ORG_NAME
UpgradingReplicaTilesYoucanupgradereplicatileslikeregulartileswithoneimportantdifference.YoumustgenerateareplicaofthenewerversionoftheRabbitMQforPCFtile,usingthereplicator,andgivethenewreplicathesame name astheexistingreplica.Thisisshownintheexampleworkflowbelow.
ExampleofanIn-PlaceUpgradeofaReplicaSupposeyouusedthereplicatortogenerateareplicaofv1oftheRabbitMQforPCFtile,withthe name trading-team,andyouinstalleditinOpsManager.Hereisthesamplereplicatorcommandyouusedfortheinitialinstallation:
./rabbitmq-replicator-darwin\--name trading-team\--path /download/p-rabbitmq-v1.pivotal\--output /output/p-rabbitmq-v1-trading-team.pivotal
Toupgradetov2,followthesesteps:
1. DownloadthenewRabbitMQforPCFv2.
2. Runthereplicatorcommand,usingthepathtothenewRabbitMQforPCFv2tile,andsupplythesame name ,trading-team,asshownbelow.
./rabbitmq-replicator-darwin\--name trading-team\--path /download/p-rabbitmq-v2.pivotal\--output /output/p-rabbitmq-v2-trading-team.pivotal
3. Afteryouhavethereplicatilep-rabbitmq-v2-trading-team.pivotal,uploadittoOpsManager.Thisupgradesthev1replicatileinplace.
Youcanthenproceedwithupgradingthecluster.
Ifyouwanttodoablue-greenstyleupgrade,seeBlue-GreenUpgrades.
LimitationsTheOn-Demandserviceisnotofferedonreplicatiles.
© Copyright Pivotal Software Inc, 2013-2018 95 1.10
DeployingtheRabbitMQPre-ProvisionedService
DefaultDeploymentDeployingRabbitMQforPivotalCloudFoundry (PCF)throughOpsManagerwilldeployaRabbitMQclusterof3nodesbydefault.
Thedeploymentincludesasingleloadbalancer haproxy whichspreadsconnectionsonallofthedefaultports,foralloftheshippedpluginsacrossallofthemachineswithinthecluster.
Thedeploymentwilloccurinasingleavailabilityzone(AZ).
Thedefaultconfigurationisfortestingpurposesonlyanditisrecommendedthatcustomershaveaminimumof3RabbitMQnodesand2HAProxynodes
ConsiderationsforthisdeploymentProvidesHAfortheRabbitMQcluster
QueuesmustbejudiciouslyconfiguredtobeHAastheyareplacedononenodebydefault
Customersshoulddecideonwhichpartitionbehaviourisbestsuitedtotheirusecase.Fortwonodes‘automatic’ispreferred
HAProxyisasinglepointoffailure(SPOF)
TheentiredeploymentisinasingleAZ,whichdoesnotprotectagainstexternalfailuresfromfailuresinhardware,networking,etc.
RecommendedDeployment
© Copyright Pivotal Software Inc, 2013-2018 96 1.10
WerecommendthatRabbitMQisdeployedacrossatleasttwoavailabilityzones.
RabbitMQservernodesshouldbescaledtoanoddnumberandshouldbegreaterthan3.
Replicationofqueuesshouldonlybeusedwhererequiredasitcanhaveabigimpactonsystemperformance.
TheHAProxyjobinstancecountshouldalsobeincreasedtomatchthenumberofAZstoensurethereisaHAProxylocatedineachAZ.ThisremovestheHAProxySPOFandprovidesfurtherredundancy.
Intheabovediagram,youcanseethatyoucannowsufferthefailureofasingleHAProxyandsingleRabbitMQnodeandstillkeepyourclusteronlineandapplicationsconnected.
ItisalsorecommendthatcustomerschoosesanoddnumberofRabbitMQservernodesofthreeormore.
UpgradingtothisdeploymentfromasingleAZdeploymentItisnotpossibletoupgradetothissetupfromthedefaultdeploymentacrossasingleAZ.
ThisisbecausetheAZsetupcannotbechangedoncethetilehasbeingdeployedforthefirsttime,thisistoprotectagainstdatalosswhenmovingjobsbetweenAZs.
UpgradingtothisdeploymentfromamultiAZdeploymentIfyouhavedeployedthetileacrosstwoAZs,butwithasingleHAProxyinstanceyoucanmigratetothissetupasfollows:
1. DeployanadditionalHAProxyinstancethroughOpsManager
2. Neworre-boundapplicationstotheRabbitMQservicewillseetheIPsofbothHAProxysimmediately
3. Existingboundapplicationswillcontinuetowork,butonlyusingthepreviouslydeployedHAProxyIPAddress.Theycanbere-boundasrequiredatyourdiscretion.
© Copyright Pivotal Software Inc, 2013-2018 97 1.10
ConsiderationsforthisdeploymentRequiresIaaSconfigurationforavailabilityzonesaheadofdeployingtheRabbitMQtile
ApplicationdeveloperswillbehandedtheIPsofeachdeployedHAProxyintheirenvironmentvariables
QueuesmustbejudiciouslyconfiguredtobeHAastheyareplacedononenodebydefault
Customersshoulddecideonwhichpartitionbehaviourisbestsuitedtotheirusecase.For3ormorenodes'pause_minority’ispreferred
AdvancedDeploymentThisdeploymentbuildsupontheaboverecommendeddeployment,sofollowsthesameupgradepaths.
ThisallowsyoutoreplacetheuseofHAProxywithyourownexternalloadbalancer.
YoumaychoosetodothistoremoveanyknowledgeofthetopologyoftheRabbitMQsetupfromapplicationdevelopers.
Advantages
ApplicationdevelopersdonotneedtohandlemultipleIPsfortheHAProxyjobsintheirapplications
Disadvantages
TheloadbalancerneedstobeconfiguredwiththeIPsoftheRabbitMQNodes.Thesewillonlybeknownoncethedeploymenthasfinished.TheIPsshouldremainthesameduringsubsequentdeploymentsbutthereisarisktheycanchange.
UpgradingtothisdeploymentfromtherecommendeddeploymentItispossibletofirstdeploywithmultipleHAProxyjobs,aspertherecommendeddeploymentanddecidedtolateruseyourownexternalloadbalancer.
Thiscanbeachievedwithoutdowntimetoyourapplications.
Thiscanbeachievedasfollows:
1. ConfigureyourexternalloadbalancertopointtotheRabbitMQNodeIPs
2. ConfiguretheDNSnameorIPaddressfortheexternalloadbalancer(ELB)ontheRabbitMQtileinOpsManager
3. Deploythechanges
© Copyright Pivotal Software Inc, 2013-2018 98 1.10
4. AnynewinstancesoftheRabbitMQserviceoranyre-boundconnectionswillusetheDNSnameorIPaddressoftheELBintheir VCAP_SERVICES
5. AnyexistinginstanceswillcontinuetousetheHAProxyIPaddressesintheir VCAP_SERVICES
6. Phasethere-bindingofexistingapplicationstohavetheirenvironmentvariablesupdated
7. Onceallapplicationsareupdated
8. Reducetheinstancecountofthe HAProxy jobinOpsManagerto1
9. Deploythechanges
Thisapproachworksasanyexistingboundapplicationshavetheir VCAP_SERVICES informationcachedinthecloudcontrollerandareonlyupdatedbyare-bindrequest.
DowngradingfromthisdeploymenttotherecommendeddeploymentIfyouarecurrentlyusinganexternalloadbalancer,thenyoucanmovebacktousingHAProxysinstead.
Youcanachievethisbyfollowingtheabovestepsinreverseorderandre-instatingtheHAProxyjobs.
ResourcerequirementsThefollowingtableshowsthedefaultresourceandIPrequirementsforinstallingthetile:
Product Resource Instances CPU Ram Ephemeral Persistent StaticIP DynamicIP
RabbitMQ RabbitMQnode 3 2 8192 16384 30720 1 0
RabbitMQ HAProxyforRabbitMQ 1 1 2048 4096 0 1 0
RabbitMQ RabbitMQservicebroker 1 1 2048 4096 0 1 0
RabbitMQ BrokerRegistrar 1 1 1024 2048 0 0 1
RabbitMQ BrokerDeregistrar 1 1 1024 2048 0 0 1
RabbitMQ SmokeTests 1 1 1024 2048 0 0 1
RabbitMQ RabbitMQon-demandbroker 1 1 1024 8192 1024 0 1
RabbitMQ RegisterOn-DemandServiceBroker 1 1 1024 2048 0 0 1
RabbitMQ DeregisterOn-DemandServiceBroker 1 1 1024 2048 0 0 1
RabbitMQ DeleteAllServiceInstances 1 1 1024 2048 0 0 1
RabbitMQ UpgradeAllServiceInstances 1 1 1024 2048 0 0 1
Notes:Thenumberof RabbitMQ Node canbeincreasedifrequired.
ChangingthenumberofRabbitMQnodeswhentheerlangcookieisnotdefinedwillrestartthecluster.Checkhereformoreinformation.
© Copyright Pivotal Software Inc, 2013-2018 99 1.10
SettingDefaultPoliciesfortheRabbitMQService
UnderstandingaRabbitMQPolicyYoucansetadefaultqueueandanexchangepolicyintheRabbitMQforPivotalCloudFoundry(PCF)tile,tobeappliedtotheirRabbitMQcluster.Afteryoudeploythetile,PivotalrecommendsthatyouusetheRabbitMQManagementInterfacetomakeconfigurationchanges.
FormoreinformationaboutRabbitMQpolicies,seetheRabbitMQdocumentation .
RulesforPoliciesSetintheTileThefollowingrulesapplytopolicessetthroughtheRabbitMQforPCFtile:
Anewpolicy,oranupdatetoapolicy,onlyappliestonewinstances.Existinginstancesarenotaffectedbythepolicy.
ThepolicycanonlybedeletedmanuallyfromtheRabbitMQnodes.
PoliciescanbeaddeddynamicallyusingtheRabbitMQManagementInterface.Forgranularpolicysettings,PivotalrecommendsusingtheRabbitMQManagementUI.Seta priority number lowerthan 50 ,thedefault priority number appliedthroughtheOpsManagerconfiguration.
AnExamplePolicy:MirroronTwoNodesHereisanexamplepolicythatensuresmessagesaremirroredontwonodes:
{ "ha-mode": "exactly", "ha-params": 2, "ha-sync-mode": "automatic"}
Operatorsshouldconsidersomeoftheperformanceimplicationsofmakingqueuesandexchangeshighlyavailable.Formoreinformationabouthighlyavailablequeues,seetheRabbitMQdocumentation .
BestPracticeforSyncingQueuesWhenaqueuesyncsallitsmessages,theyareloadedintomemory.Whenqueuesaresyncing,theycanuseasmuchmemoryasthetotalsizeofallmessages.Thisappliestobothnodes—thenodewherethequeueleaderruns(fromnode)andthenodewherethequeuefollowerruns(tonode),butonlyappliestonewlycreatedqueuefollowers.
Thisbehaviorisespeciallyrelevantwhenanychangeaffectsthedeployment,forexample:stemcellupdates,deploymentconfigurationchanges,andnetworkchanges.Verifythatyouhaveenoughmemoryanddiskavailabletosupportallmessages.
Forexample:
Thereare5GBofmessagesinamirroredqueuethatissettoautomaticsync.Whenthisqueueneedstosync,thenodewherethequeueleaderrunscanuseupto5GBofextramemory.Thesameappliestothenodewherethenewqueuefolloweriscreated.
SettingorChangingthePolicyTosettheRabbitMQpolicy,dothefollowing:
1. FromtheOpsManagerInstallationDashboard,clicktheRabbitMQforPCFtile,thenclickRabbitMQPolicy.
© Copyright Pivotal Software Inc, 2013-2018 100 1.10
2. SelectEnablecustompolicyonnewinstances.
3. InthePolicyfornewinstancesfield,pastethepolicy.ThepolicymustbevalidJSONandshouldmeetvalidRabbitMQpolicycriteria.
4. IntheSelectthenetworkpartitionbehavioroftheRabbitMQcluster,choosethedesiredbehavior:pause_minorityorautoheal.
Formoreinformationabouttheseoptions,andRabbitMQclustersandnetworkpartitions,seetheRabbitMQdocumentation .
Forproductionpurposes,PivotalrecommendsthatcustomershaveatleastthreeRabbitMQservernodesandtwoHAProxiesspreadacrosslowlatencyavailabilityzones.
ViewingPoliciesintheRabbitMQManagementDashboardYoucanviewRabbitMQpoliciesintheRabbitMQManagementDashboard,shownbelow.TheexamplepolicyweenteredintheRabbitMQforPCFtileaboveisappliedtoallqueuesandgivenaPriorityof50.Thisallowsyoutooverrideitbydefininganotherpolicywithahigherpriority.
IntheQueuessectionshownbelow,youcanseethatanynewqueuescreatedhavethepolicyautomaticallyapplied.
Note:Nopolicyvalidationoccursduringthedeployment,anderrorscancausethedeploymenttofailorpoliciestobeappliedincorrectly.
© Copyright Pivotal Software Inc, 2013-2018 101 1.10
Note:DeveloperscanobtaintheURLofthepolicyfrom VCAP_SERVICES forapplicationdevelopers.
© Copyright Pivotal Software Inc, 2013-2018 102 1.10
MonitoringandKPIsforPre-ProvisionedRabbitMQforPCFThistopicexplainshowtomonitorthehealthofthepre-provisionedversionoftheRabbitMQforPivotalCloudFoundry(PCF)serviceusingthelogs,metrics,andKeyPerformanceIndicators(KPIs)generatedbyRabbitMQforPCFcomponentVMs.
Pre-provisionedRabbitMQforPCFcomponentsgeneratemanyofthesamemetricsastheon-demandRabbitMQservicecomponents.
SeeLoggingandMetrics forgeneralinformationaboutloggingandmetricsinPCF.
SettingupSyslogForwardingOperatorscanenablelogforwardingbyconfiguringanexternalsyslogendpointforRabbitMQcomponentlogmessages.Forinstructionsonsettingupsyslogforwarding,seeSyslog.
Ifsyslogforwardingisenabled,logentrieswithtimestampsareavailablelocallyin /var/log/messages .Logsareavailableunder /var/vcap/sys/log/ whethersyslogforwardingisenabledornot.
LoggingFormatsWithpre-provisionedRabbitMQforPCFloggingconfigured,threetypesofcomponentgeneratelogs:theRabbitMQmessageservernodes,theservicebroker,andHAProxy.IfyouhavemultipleserverorHAProxynodes,youcanidentifylogsfromindividualnodesbytheirindex,whichcorrespondstotheindexoftheRabbitMQVMinstancesdisplayedinOpsManager:
ThelogsforRabbitMQservernodesfollowtheformat [job=rabbitmq-server-partition-GUID index=X]
ThelogsforHAProxynodesfollowtheformat [job=rabbitmq-haproxy-partition-GUID index=X]
ThelogsfortheRabbitMQservicebrokerfollowtheformat [job=rabbitmq-broker-partition-GUID index=X]
RabbitMQandHAProxyserverslogatthe info levelandcaptureerrors,warnings,andinformationalmessages.
Forusersfamiliarwithdocumentationforpreviousversionsofthetile,thetagweusedtocallthe app_name isnowcalledthe program_name .
Thegenericlogformatisasfollows:
<PRI>TIMESTAMP IP_ADDRESS PROGRAM_NAME [job=NAME index=JOB_INDEX id=JOB_ID] MESSAGE
Therawlogslooksimilartothefollowing:
<7>2017-06-28T16:06:10.733560+00:00 10.244.16.133 vcap.agent [job=rmq index=0 id=e37ecdca-5b10-4141-abd8-e1d777dfd8b5] 2017/06/28 16:06:10 CEF:0|CloudFoundry|BOSH|1|agent_api|ssh|1|duser=director.be5a66bb-a9b4-459f-a0d3-1fc5c9c3ed79.be148cc6-91ef-4eed-a788-237b0b8c63b7 src=10.254.50.4 spt=4222 shost=5ae233e0-ecc5-4868-9ae0-f9767571251b<86>2017-06-28T16:06:16.704572+00:00 10.244.16.133 useradd [job=rmq index=0 id=e37ecdca-5b10-4141-abd8-e1d777dfd8b5] new group: name=bosh_ly0d2rbjr, GID=1003<86>2017-06-28T16:06:16.704663+00:00 10.244.16.133 useradd [job=rmq index=0 id=e37ecdca-5b10-4141-abd8-e1d777dfd8b5] new user: name=bosh_ly0d2rbjr, UID=1001, GID=1003, home=/var/vcap/bosh_ssh/bosh_ly0d2rbjr, shell=/bin/bash<86>2017-06-28T16:06:16.736932+00:00 10.244.16.133 usermod [job=rmq index=0 id=e37ecdca-5b10-4141-abd8-e1d777dfd8b5] add 'bosh_ly0d2rbjr' to group 'admin'<86>2017-06-28T16:06:16.736964+00:00 10.244.16.133 usermod [job=rmq index=0 id=e37ecdca-5b10-4141-abd8-e1d777dfd8b5] add 'bosh_ly0d2rbjr' to group 'vcap'
LogssenttoexternalloggingtoolssuchasPapertrailmaybepresentedinadifferentformat.
Thefollowingtabledescribestheloggingtagsusedinthistemplate:
Tag Description
PRI Thisisavaluewhichinfuturewillbeusedtodescribetheseverityofthelogmessageandwhichfacilityitcamefrom.
TIMESTAMPThisisthetimestampofwhenthelogisforwarded,forexample, 2016-08-24T05:14:15.000003Z .Thetimestampvalueistypicallyslightlyafterwhenthelogmessagewasgenerated.
IP_ADDRESS TheinternalIPaddressofserveronwhichthelogmessageoriginated
PROGRAM_NAMEProcessnameoftheprogramthegeneratedthemessage.Sameas app_name beforev1.9.0.Formoreinformationaboutprogramname,seeRabbitMQProgramNamesbelow.
NAME TheBOSHinstancegroupname(forexample, rabbitmq_server )
JOB_INDEX BOSHjobindex.Usedtodistinguishbetweenmultipleinstancesofthesamejob.
JOB_IDBOSHVMGUID.ThisisdistinctfromtheCIDdisplayedintheOpsManagerStatustab,whichcorrespondstotheVMIDassignedbytheinfrastructureprovider.
© Copyright Pivotal Software Inc, 2013-2018 103 1.10
MESSAGE Thelogmessagethatappears
RabbitMQProgramNames
ProgramName Description
rabbitmq_server_cluster_check ChecksthattheRabbitMQclusterishealthy.Runsaftereverydeploy.
rabbitmq_server_node_check ChecksthattheRabbitMQnodeishealthy.Runsaftereverydeploy.
rabbitmq_route_registrar_stderr RegisterstherouteforthemanagementAPIwiththeGorouterinyourElasticRuntimedeployment.
rabbitmq_route_registrar_stdout RegisterstherouteforthemanagementAPIwiththeGorouterinyourElasticRuntimedeployment.
rabbitmq_server TheErlangVMandRabbitMQapps.Logsmayspanmultiplelines.
rabbitmq_server_drain ShutsdowntheErlangVMandRabbitMQapps.RunsaspartoftheBOSHlifecycle.
rabbitmq_server_http_api_access AccesstotheRabbitMQmanagementUI.
rabbitmq_server_init StartstheErlangVMandRabbitMQ.
rabbitmq_server_post_deploy_stderr Runsthenodecheckandclustercheck.Runsaftereverydeploy.
rabbitmq_server_post_deploy_stdout Runsthenodecheckandclustercheck.Runsaftereverydeploy.
rabbitmq_server_pre_start Runsbeforetherabbitmq-serverjobisstarted.
rabbitmq_server_sasl Supervisor,progress,andcrashreportingfortheErlangVMandRabbitMQapps.
rabbitmq_server_shutdown_stderr StopstheRabbitMQappandErlangVM.
rabbitmq_server_shutdown_stdout StopstheRabbitMQappandErlangVM.
rabbitmq_server_startup_stderr StartstheRabbitMQappandErlangVM,thenconfiguresusersandpermissions.
rabbitmq_server_startup_stdout StartstheRabbitMQappandErlangVM,thenconfiguresusersandpermissions.
rabbitmq_server_upgrade ShutsdownErlangVMandRabbitMQappifrequiredduringanupgrade.
MetricsMetricsareregularly-generatedlogmessagesthatreportmeasuredcomponentstates.Themetricspollingintervaldefaultsto30seconds.ThemetricspollingintervalisaconfigurationoptionontheRabbitMQtile(Settings>RabbitMQ).Theintervalsettingappliestoallcomponentsdeployedbythetile.
Metricsarelong,singlelinesoftextthatfollowtheformat:
origin:"p-rabbitmq" eventType:ValueMetric timestamp:1441188462382091652 deployment:"cf-rabbitmq" job:"cf-rabbitmq-node" index:"0" ip:"10.244.3.46" valueMetric: < name:"/p-rabbitmq/rabbitmq/system/memory" value:1024 unit:"MB">
PartitionIndicatorAnewmetrichasbeenintroducedtohelptoidentifynetworkpartitions.Essentiallyitexposeshowmanynodeseachnodeknows.Whenanodeisinpartitiontheonlynodethatitrecognizesisitselfandthatisagoodindicationthatthatnodemightbeinapartition.
Anexampleofthatmetricsis:
origin:"p-rabbitmq" eventType:ValueMetric timestamp:1441188462382091652 deployment:"cf-rabbitmq" job:"cf-rabbitmq-node" index:"0" ip:"10.244.3.46" valueMetric: < name:"/p-rabbitmq/rabbitmq/erlang/reachable_nodes" value:3 unit:"count">
Monitorscanbecreatedtoemitalertsincaseaclusterseemstobeinapartition.Ametricsisemittedforeachnodeinthecluster.Forexample:inathree-nodeclusteramonitorcanexpecttohaveatotalof9(nine)sinceeachnodeisexpectedtoemit3(2reachablenodesanditself).Otherwise,analertcanbesenttotheteam.
RecoveringfromanetworkpartitionPleaserefertotheoficialRabbitMQguidetounderstandhowtorecoverfromanetworkpartition:https://www.rabbitmq.com/partitions.html
© Copyright Pivotal Software Inc, 2013-2018 104 1.10
KeyPerformanceIndicatorsKeyPerformanceIndicators(KPIs)forRabbitMQforPCFaremetricsthatoperatorsfindmostusefulformonitoringtheirRabbitMQservicetoensuresmoothoperation.KPIsarehigh-signal-valuemetricsthatcanindicateemergingissues.KPIscanberawcomponentmetricsorderivedmetricsgeneratedbyapplyingformulastorawmetrics.
PivotalprovidesthefollowingKPIsasgeneralalertingandresponseguidancefortypicalRabbitMQforPCFinstallations.Pivotalrecommendsthatoperatorscontinuetofine-tunethealertmeasurestotheirinstallationbyobservinghistoricaltrends.Pivotalalsorecommendsthatoperatorsexpandbeyondthisguidanceandcreatenew,installation-specificmonitoringmetrics,thresholds,andalertsbasedonlearningfromtheirowninstallations.
ForalistofallRabbitMQforPCFrawcomponentmetrics,seeComponentMetricsReference.
ComponentHeartbeatsKeyRabbitMQforPCFcomponentsperiodicallyemitheartbeatmetrics:theRabbitMQservernodes,HAProxynodes,andtheServiceBroker.TheheartbeatsareBooleanmetrics,where 1 meansthesystemisavailable,and 0 ortheabsenceofaheartbeatmetricmeanstheserviceisnotrespondingandshouldbeinvestigated.
ServiceBrokerHeartbeat
p-rabbitmq.service_broker.heartbeat
Description
RabbitMQServiceBroker is alive poll,whichindicatesifthecomponentisavailableandabletorespondtorequests.
Use:IftheServiceBrokerdoesnotemitheartbeats,thisindicatesthatitisoffline.TheServiceBrokerisrequiredtocreate,update,anddeleteserviceinstances,whicharecriticalfordependenttilessuchasSpringCloudServicesandSpringCloudDataFlow.
Origin:Doppler/FirehoseType:booleanFrequency:30s(default),10s(configurableminimum)
Recommendedmeasurement Averageoverlast5minutes
RecommendedalertthresholdsYellowwarning:N/ARedcritical:<1
Recommendedresponse
ChecktheRabbitMQServiceBrokerlogsforerrors.YoucanfindthisVMbytargetingyourRabbitMQdeploymentwithBOSHandrunningoneofthefollowingcommandsdependingonyourOpsManagerVersion:
ForOpsManagerv1.10orearlier:bosh vms service-instance_GUID
ForOpsManagerv1.11orlater:bosh2 -d service-instance_GUID vms
HAProxyHeartbeat
p-rabbitmq.haproxy.heartbeat
Description
RabbitMQHAProxy is alive poll,whichindicatesifthecomponentisavailableandabletorespondtorequests.
Use:IftheHAProxydoesnotemitheartbeats,thisindicatesthatitisoffline.Tobefunctional,serviceinstancesrequireHAProxy.
Origin:Doppler/FirehoseType:booleanFrequency:30s(default),10s(configurableminimum)
Recommendedmeasurement Averageoverlast5minutes
© Copyright Pivotal Software Inc, 2013-2018 105 1.10
Recommendedalertthresholds Yellowwarning:N/ARedcritical:<1
Recommendedresponse
ChecktheRabbitMQHAProxylogsforerrors.YoucanfindtheVMbytargetingyourRabbitMQdeploymentwithBOSHandrunningoneofthefollowingcommands,whichlists HAProxy_GUID :
ForOpsManagerv1.10orearlier:bosh vms service-instance_GUID
ForOpsManagerv1.11orlater:bosh2 -d service-instance_GUID vms
ServerHeartbeat
p-rabbitmq.rabbitmq.heartbeat
Description
RabbitMQServer is alive poll,whichindicatesifthecomponentisavailableandabletorespondtorequests.
Use:Iftheserverdoesnotemitheartbeats,thisindicatesthatitisoffline.Tobefunctional,serviceinstancesrequireRabbitMQServer.
Origin:Doppler/FirehoseType:booleanFrequency:30s(default),10s(configurableminimum)
Recommendedmeasurement Averageoverlast5minutes
RecommendedalertthresholdsYellowwarning:N/ARedcritical:<1
Recommendedresponse
ChecktheRabbitMQServerlogsforerrors.YoucanfindtheVMbytargetingyourRabbitMQdeploymentwithBOSHandrunningoneofthefollowingcommands,whichlists rabbitmq :
ForOpsManagerv1.10orearlier:bosh vms service-instance_GUID
ForOpsManagerv1.11orlater:bosh2 -d service-instance_GUID vms
RabbitMQServerKPIsThefollowingKPIsfromtheRabbitMQservercomponent:
FileDescriptors
p-rabbitmq.rabbitmq.system.file_descriptors
Description
Filedescriptorsconsumed.
Use:Ifthenumberoffiledescriptorsconsumedbecomestoolarge,theVMmaylosetheabilitytoperformdiskIO,whichcancausedataloss.
Origin:Doppler/FirehoseType:countFrequency:30s(default),10s(configurableminimum)
Recommendedmeasurement Averageoverlast10minutes
RecommendedalertthresholdsYellowwarning:>50000Redcritical:>55000
Thedefault ulimit forRabbitMQforPCFv1.6andlateris60000.Ifthismetricismetorexceededforan
Note:Thisassumesnon-persistentmessagesarehandledbyretriesorsomeotherlogicbytheproducers.
© Copyright Pivotal Software Inc, 2013-2018 106 1.10
Recommendedresponse extendedperiodoftime,consideroneofthefollowingactions:ScalingtherabbitnodesinthetileResourceConfigpane.
Increasingthe ulimit
ErlangProcesses
p-rabbitmq.rabbitmq.system.erlang_processes
Description
Erlang processesconsumedbyRabbitMQ,whichrunsonanErlangVM.
Use:Thisisthekeyindicatoroftheprocessingcapabilityofanode.
Origin:Doppler/FirehoseType:countFrequency:30s(default),10s(configurableminimum)
Recommendedmeasurement Averageoverlast10minutes
RecommendedalertthresholdsYellowwarning:>900000Redcritical:>950000
RecommendedresponseThedefaultErlangprocesslimitinRabbitMQforPCFv1.6andlateris1,048,816.Ifthismetricmeetsorexceedstherecommendedthresholdsforextendedperiodsoftime,considerscalingtheRabbitMQnodesinthetileResourceConfigpane.
BOSHSystemHealthMetricsTheBOSHlayerthatunderliesPCFgenerates healthmonitor metricsforallVMsinthedeployment.However,thesemetricsarenotincludedintheLoggregatorFirehosebydefault.Togetthesemetrics,doeitherofthefollowing:
TosendBOSHHMmetricsthroughtheFirehose,installtheopen-sourceHMForwarder .
ToretrieveBOSHhealthmetricsoutsideoftheFirehose,installtheJMXBridge forPCFtile.
InafuturereleasetheBOSHsystemhealthmetricswillbeavailabledirectlyfromtheFirehose.
AllBOSH-deployedcomponentsgeneratethefollowingsystemhealthmetrics.ComingfromRabbitMQforPCFcomponents,thesesystemmetricsserveasKPIsfortheRabbitMQforPCFservice.
RAM
system.mem.percent
Description
RAMbeingconsumedbythe p-rabbitmq VM.
Use:RabbitMQisconsideredtobeinagoodstatewhenithaslittleornomessages.Inotherwords,“anemptyrabbitisahappyrabbit.”Alertingonthismetriccanindicatethattherearetoofewconsumersorappsthatreadmessagesfromthequeue.
HealthmonitorreportswhenRabbitMQusesmorethan40%ofitsRAMforthepasttenminutes.
Origin:JMXBridgeorBOSHHMType:percentFrequency:30s(default),10s(configurableminimum)
Recommendedmeasurement Averageoverlast10minutes
RecommendedalertthresholdsYellowwarning:>40Redcritical:>50
Recommendedresponse Addmoreconsumerstodrainthequeueasfastaspossible.
© Copyright Pivotal Software Inc, 2013-2018 107 1.10
CPU
system.cpu.percent
Description
CPUbeingconsumedbythe p-rabbitmq VM.
Use:AnodethatexperiencescontextswitchingorhighCPUusagewillbecomeunresponsive.Thisalsoaffectstheabilityofthenodetoreportmetrics.
HealthmonitorreportswhenRabbitMQusesmorethan40%ofitsCPUforthepasttenminutes.
Origin:JMXBridgeorBOSHHMType:percentFrequency:30s(default),10s(configurableminimum)
Recommendedmeasurement Averageoverlast10minutes
RecommendedalertthresholdsYellowwarning:>60Redcritical:>75
Recommendedresponse Rememberthat“anemptyrabbitisahappyrabbit”.Addmoreconsumerstodrainthequeueasfastaspossible.
EphemeralDisk
system.disk.percent
Description
EphemeralDiskbeingconsumedbythe p-rabbitmq VM.
Use:Ifsystemdiskfillsup,therearetoofewconsumers.
HealthmonitorreportswhenRabbitMQusesmorethan40%ofitsCPUforthepasttenminutes.
Origin:JMXBridgeorBOSHHMType:percentFrequency:30s(default),10s(configurableminimum)
Recommendedmeasurement Averageoverlast10minutes
RecommendedalertthresholdsYellowwarning:>60Redcritical:>75
Recommendedresponse Rememberthat“anemptyrabbitisahappyrabbit”.Addmoreconsumerstodrainthequeueasfastaspossible.
PersistentDisk
persistent.disk.percent
Description
PersistentDiskbeingconsumedbythe p-rabbitmq VM.
Use:Ifsystemdiskfillsup,therearetoofewconsumers.
HealthmonitorreportswhenRabbitMQusesmorethan40%ofitsCPUforthepasttenminutes.
Origin:JMXBridgeorBOSHHMType:percentFrequency:30s(default),10s(configurableminimum)
Recommendedmeasurement Averageoverlast10minutes
RecommendedalertthresholdsYellowwarning:>60Redcritical:>75
Recommendedresponse Rememberthat“anemptyrabbitisahappyrabbit”.Addmoreconsumerstodrainthequeueasfastaspossible.
© Copyright Pivotal Software Inc, 2013-2018 108 1.10
ComponentMetricReferenceRabbitMQforPCFcomponentVMsemitthefollowingrawmetrics.Thefullnameofthemetricfollowstheformat: /p-rabbitmq/COMPONENT/METRIC-NAME
RabbitMQServerMetricsRabbitMQforPCFmessageservercomponentsemitthefollowingmetrics.
FullName Unit Description
/p-rabbitmq.rabbitmq.heartbeat boolean IndicateswhethertheRabbitMQserverisavailableandabletorespondtorequests
/p-rabbitmq/rabbitmq/erlang/erlang_processes
count ThenumberofErlangprocesses
/p-rabbitmq/rabbitmq/system/memory MB ThememoryinMBusedbythenode
/p-rabbitmq/rabbitmq/system/mem_alarm boolean Indicatesifthememoryalarmhasgoneoff
/p-rabbitmq/rabbitmq/system/disk_free_alarm boolean Indicatesifthediskfreealarmhasgoneoff
/p-rabbitmq/rabbitmq/connections/count count Thetotalnumberofconnectionstothenode
/p-rabbitmq/rabbitmq/consumers/count count Thetotalnumberofconsumersregisteredinthenode
/p-rabbitmq/rabbitmq/messages/delivered count Thetotalnumberofmessageswiththestatus deliver_get onthenode
/p-rabbitmq/rabbitmq/messages/delivered_no_ack
count Thenumberofmessageswiththestatus deliver_no_ack onthenode
/p-rabbitmq/rabbitmq/messages/delivered_rate
rate Therateatwhichmessagesarebeingdeliveredtoconsumersorclientsonthenode
/p-rabbitmq/rabbitmq/messages/published count Thetotalnumberofmessageswiththestatus publish onthenode
/p-rabbitmq/rabbitmq/messages/published_rate
rate Therateatwhichmessagesarebeingpublishedbythenode
/p-rabbitmq/rabbitmq/messages/redelivered count Thetotalnumberofmessageswiththestatus redeliver onthenode
/p-rabbitmq/rabbitmq/messages/redelivered_rate
rate Therateatwhichmessagesaregettingthestatus redeliver onthenode
/p-rabbitmq/rabbitmq/messages/got _no_ack count Thenumberofmessageswiththestatus get_no_ack onthenode
/p-rabbitmq/rabbitmq/messages/get_no_ack_rate
rate Therateatwhichmessagesgetthestatus get_no_ack onthenode
/p-rabbitmq/rabbitmq/messages/pending count Thenumberofmessageswiththestatus messages_unacknowledged onthenode
/p-rabbitmq/rabbitmq/messages/depth countThenumberofmessageswiththestatus messages_unacknowledged ormessages_ready onthenode
/p-rabbitmq/rabbitmq/system/filedescriptors
count Thenumberofopenfiledescriptorsonthenode
/p-rabbitmq/rabbitmq/exchanges/count count Thetotalnumberofexchangesonthenode
/p-rabbitmq/rabbitmq/messages/available count Thetotalnumberofmessageswiththestatus messages_ready onthenode
/p-rabbitmq/rabbitmq/queues/count count Thenumberofqueuesonthenode
/p-rabbitmq/rabbitmq/channels/count count Thenumberofchannelsonthenode
/p-rabbitmq/rabbitmq/queues/VHOST-NAME/QUEUE-NAME/consumers
count Thenumberofconsumerspervirtualhostperqueue
/p-rabbitmq/rabbitmq/queues/VHOST-NAME/QUEUE-NAME/depth
countThenumberofmessageswiththestatus messages_unacknowledged ormessages_ready pervirtualhostperqueue
HAProxyMetricsRabbitMQforPCFHAProxycomponentsemitthefollowingmetrics.
NameSpace Unit Description
/p-rabbitmq.haproxy.heartbeat booleanIndicateswhethertheRabbitMQHAProxycomponentisavailableandabletorespondtorequests
© Copyright Pivotal Software Inc, 2013-2018 109 1.10
/p-rabbitmq/haproxy/health/connections
count Thetotalnumberofconcurrentfront-endconnectionstotheserver
/p-rabbitmq/haproxy/backend/qsize/amqp
size ThetotalsizeoftheAMQPqueueontheserver
/p-rabbitmq/haproxy/backend/retries/amqp
count ThenumberofAMQPretriestotheserver
/p-rabbitmq/haproxy/backend/ctime/amqp
time ThetotaltimetoestablishtheTCPAMQPconnectiontotheserver
© Copyright Pivotal Software Inc, 2013-2018 110 1.10
ManagingtheRabbitMQ®Service
RabbitMQManagementDashboard
AdminUserTogainaccesstothemanagementdashboardasthe admin user,visit http://pivotal-rabbitmq.<cf sys
domain>,wherecanbefoundat“PivotalElasticRuntime”
configurationon“Domains”section:“SystemDomain”field.
TheusernameandpasswordistheusernameandpasswordyouprovidedintheRabbitMQconfigurationinOpsManager,whichisalsoshownintheCredentialstab.
ApplicationDeveloperUsersofCloudFoundrywhocreateinstancesviatheAppsManagerorthecfCLIalsogetaccesstotheManagementUI.Thisisdoneusingcredentialsthatprovideaccessonlytotheirparticularvhost.
TheappropriateURLisaccessibleviatheManagebuttonwithintheAppsManager.
© Copyright Pivotal Software Inc, 2013-2018 111 1.10
Oritisalsoinjectedintothe VCAP_SERVICES environmentvariableprovidedtoappsrunningonCloudFoundry.ThiscanalsobefoundviatheCLIusingcf env <your appname>
.
LoggingATCPSyslogendpointcanbeconfiguredinOpsManager.LogsarecurrentlyonlyforwardedfortheRabbitMQcluster.
RabbitMQCLIIfyouwishtoruncommandssuchas rabbitmqctl thenyouhavetwooptions:
SSHintooneofthemachinesrunningtherabbitmq-server.IPscanbefoundfromtheStatustabandaccesscredentialsfromtheCredentialstabwithintheRabbitMQcomponentoftheinstaller.FromthereyouneedtobringRabbitMQandErlangintoyourenvironmentandfromthereyoucanuse
rabbitmqctl :
bash-4.1# export PATH=$PATH:/var/vcap/packages/rabbitmq-server/binbash-4.1# export PATH=$PATH:/var/vcap/packages/erlang/binbash-4.1# rabbitmqctl cluster_statusCluster status of node rabbit@node0 ...[{nodes,[{disc,[rabbit@node0,rabbit@node1,rabbit@node2,rabbit@node3]}]}, {running_nodes,[rabbit@node3,rabbit@node2,rabbit@node1,rabbit@node0]}, {partitions,[]}]...done.
Alternatively,installRabbitMQandErlangonamachineofyourchoice.Besuretomatchversionsofbothtothecluster:theManagementUIshowsboththeversionofRabbitMQandErlang.
Thensetyour ~/.erlang.cookie tomatchthecookieusedinthecluster(youmayhavesuppliedthisaspartoftheinstallation;seeabove).
Youwillneedtosetupyour /etc/hosts filetomatchtheRabbitMQnodes.
© Copyright Pivotal Software Inc, 2013-2018 112 1.10