RabbitMQ for PCF - resources.docs.pivotal.io · RabbitMQ for PCF About RabbitMQ for PCF RabbitMQ...

RabbitMQ for PCF®

Version 1.10

User's Guide

14182130313339404951697476799296

100103111

Table of ContentsTable of ContentsRabbitMQ for PCFRabbitMQ® for PCF Release NotesUnlocking the Power of On-Demand RabbitMQ for PCFOn-Demand Service ArchitectureInstalling and Configuring the On-Demand ServiceSmoke TestsUpgrading RabbitMQ for PCFSetting Limits for On-Demand Service InstancesControlling Access to Service Plans by OrgMonitoring and KPIs for On-Demand RabbitMQ for PCFClustering and Network PartitionsTroubleshooting On-Demand RabbitMQ for PCFUsing On-Demand RabbitMQ for PCFRabbitMQ® Environment VariablesTroubleshooting InstancesInstalling and Configuring the Pre-Provisioned ServiceCreating Isolation with the RabbitMQ for PCF ReplicatorDeploying the RabbitMQ Pre-Provisioned ServiceSetting Default Policies for the RabbitMQ ServiceMonitoring and KPIs for Pre‑Provisioned RabbitMQ for PCFManaging the RabbitMQ® Service

RabbitMQforPCF

AboutRabbitMQforPCFRabbitMQforPivotalCloudFoundry(PCF)enablesPCFappdeveloperstoprovisionandusetheRabbitMQmessagebrokerwithasinglecommand.

Asofversion1.8,RabbitMQforPCFsupportstwotypesofservice,anon-demandserviceandapre-provisionedservice.Thistablesummarizesthemaindifferencesbetweenthetwo:

AvailableSince VMsitRunsOn HowVMsareCreated MetricsName

Prefix

On-DemandService v1.8

DedicatedVMthatservesasingleserviceinstance.Seethistopicfordetails.

PCFcreateseachVMon-demandwhenappdevelopercreatesserviceinstance

p.rabbitmq

(withadot)

Pre-ProvisionedService

v1.2Multi-tenantVMssharedbyappsacrossPCFdeployment

PCFcreatesallVMswhenoperatordeploysorupdatesservice

p-rabbitmq

(withadash)

ThisRabbitMQforPCFdocumentationdescribesbothservicetypes.DocumentationforRabbitMQforPCFv1.7andearlieronlydescribesapre-provisionedservice.

WhatareOn-DemandInstancesInRabbitMQforPCFversionsbeforev1.8.0,theRabbitMQserviceinstancescorrespondtoauniqueRabbitMQVhostonthemulti-tenantRabbitMQcluster.RabbitMQforPCFv1.8.0introducedOn-DemandBroker(ODB) support.Thatmeansthatanew,single-tenant,clustercanbecreatedanddedicatedtoasingleapp.

Formoreinformation,seeUnlockingthePowerofOn-DemandRabbitMQforPCFandOn-DemandServiceArchitecture.

AboutRabbitMQRabbitMQisafastanddependableopen-sourcemessageserver,whichsupportsawiderangeofusecasesincludingreliableintegration,content-basedroutingandglobaldatadelivery,andhigh-volumemonitoringanddataingestion.

Emergingasthedefactostandardforcloudmessaging,RabbitMQisusedforefficientcommunicationbetweenservers,appsanddevices,andcreateslastingvaluebyenablingrapiddevelopmentofmoderndecentralizedappanddataarchitecturesthatcanscalewithyourbusinessneeds.

ProductSnapshotThefollowingtableprovidesversionandversion-supportinformationaboutRabbitMQforPCF.

Element Details

Version v1.10.20

Releasedate July13,2018

Softwarecomponentversion RabbitMQOSSv3.6.16

CompatibleOpsManagerversion(s) v1.11.xandv1.12.x

CompatibleElasticRuntimeversion(s) v1.11.xandlater

IaaSsupport AWS,Azure,GCP,OpenStack,andvSphere

IPsecsupport No

Note:RabbitMQforPCFv1.10isnolongersupported.Thesupportperiodforv1.10hasexpired.Tostayup-to-datewiththelatestsoftwareandsecurityupdates,upgradetoRabbitMQforPCFv1.12orlater.

Features

On-DemandProvisionon-demandsinglenodededicatedinstancesofRabbitMQwithauniqueRabbitMQVhost(virtualhost)

Provisionon-demandclustereddedicatedinstancesofRabbitMQ

Bindappstoaninstanceoftheplan,providinguniquecredentialsforeachbinding

Managementdashboardaccesstoappdevelopers

Deploymentintoanavailabilityzonespecifiedbytheplan

AutomatedupgradesofRabbitMQformajor,minor,andpatchreleases(seereleasenotesfordowntimerequirements)

RabbitMQSyslogforwardingconfigurationinheritedfromthepre-provisionedconfiguration

RabbitMQmetricsareexposedonthefirehose

Runsmoketestsforon-demandplans

Formoreinformation,seeUnlockingthePowerofOn-DemandRabbitMQforPCF.

Pre-ProvisionedProvisionaninstanceoftheRabbitMQservice,whichcorrespondstoauniqueRabbitMQVhost(virtualhost)

Bindappstoaninstanceoftheplan,providinguniquecredentialsforeachbinding

ManagementdashboardaccesstoPCFOperatorsandappdevelopers

Deploymentacrossmultipleavailabilityzones,withnodesstripedacrosstheAZsautomatically

EnableSSL(SecureSocketsLayer)fortheAMQP,MQTT,STOMPprotocols

HAProxyloadbalanceracrossallnodestobalanceconnections

Pluginconfigurationcanbeeasilychangedatanytimeandtheclusterredeployedandupdated

Theclustertopologycanbechangedandeasilyscaledout

AutomatedupgradesofRabbitMQformajor,minor,andpatchreleases(seeDowntimeWhenUpgradingfordowntimerequirements)

ConfiguretheendpointfortheRabbitMQSyslog

RabbitMQandHAProxymetricsareexposedonthefirehose

Syslogforwardingonbydefault

ReleaseNotesandKnownIssuesCheckthereleasenotesforyourreleaseversionforimportantinformationandknownissues.Toseereleasenotesforanotherversion,selecttheversionfromthedropdownlistatthetopofthepage.

RabbitMQforPCFandOtherPCFServicesSomePCFservicesofferon-demandserviceplans.Theseplansletdevelopersprovisionserviceinstanceswhentheywant.

Thesecontrastwiththemorecommonpre-provisionedserviceplans,whichrequireoperatorstoprovisiontheserviceinstancesduringinstallationandconfigurationthroughtheservicetileUI.

ThefollowingPCFservicesofferon-demandserviceplans:

MySQLforPCFv2.0andlater

RabbitMQforPCF

RedisforPCF

PivotalCloudCache(PCC)

Theseservicespackageanddelivertheiron-demandserviceofferingsdifferently.Forexample,someservices,likeRedisforPCF,haveonetile,andyouconfigurethetiledifferentlydependingonwhetheryouwanton-demandserviceplansorpre-provisionedserviceplans.

Forotherservices,likePCC,youinstallonetileforon-demandserviceplansandadifferenttileforpre-provisionedserviceplans.

ThefollowingtablelistsandcontraststhedifferentwaysthatPCFservicespackageon-demandandpre-provisionedserviceofferings.

PCFservicetile Standaloneproductrelatedtotheservice Versionssupportingondemand Versionssupportingpre-provisioned

RabbitMQforPCF PivotalRabbitMQ v1.8andlater Allversions

RedisforPCF Redis v1.8andlater Allversions

MySQLforPCF MySQLv2.x(basedonPerconaServer)

v1.x(basedonMariaDBandGalera)

PCC PivotalGemFire Allversions NA

GemFireforPCF PivotalGemFire NA Allversions

Pleaseprovideanybugs,featurerequests,orquestionstothePCFFeedbacklist.

RabbitMQ®forPCFReleaseNotes

UpgradetotheLatestVersionPivotalrecommendsthatyouupgradetothelatestversionofyourcurrentminorline,thenupgradetothelatestavailableversionofthenewminorline.Forexample,ifyouuseanolderv1.7.xversion,upgradetothelatestv1.7.xversionbeforeupgradingtothelatestv1.8.xversion.

SeetheProductCompatibilityMatrix forproductversionsandupgradepaths.

v1.10.20ReleaseDate:July13,2018

FeaturesRequiresstemcell3445.51

KnownIssuesSmoketestsmightfailbecausetime-outvaluesinthesmoketestsaretoolow.Formoreinformationabouttroubleshootingsmoketests,seeSmokeTests.

ClusterscalingorchangingtheErlangCookievaluerequiresclusterdowntime,andmightresultinfaileddeployments.Formoreinformation,seeClusterScalingKnownIssueandChangingtheErlangCookieValueKnownIssue.

Changingnetworksand/orIPaddressesforthe RabbitMQ Server jobresultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .

WhenerrandrunrulesaresettoWhenChanged,OpsManagermaynotruntheerrandswhenthetilehasrelevantchanges.Formoreinformation,seeManagingErrandsinOpsManager .PivotalrecommendsleavingthedefaultrunrulesettoOn.

PackagesOSSRabbitMQv3.6.16

Erlangv20.3.8.1

HAProxyv1.6.13

v1.10.19ReleaseDate:June5,2018

cf-cligolangversionreducedtov1.9.5fromv1.10becauseofanissueparsingx509certificates.Formoreinformation,seetheGolangissuecrypto/x509:CANotAuthorizedForExtKeyUsageispremature .

Changingnetworksand/orIPaddressesforthe RabbitMQ Server jobresultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIP

AddressesResultsinaFailedDeployment .

Erlangv19.3.6.4

HAProxyv1.6.13

v1.10.18ReleaseDate:May16,2018

Timestampsaddedtologentriesthatdidnothavethem.

Golangversionreducedtov1.9.5fromv1.10becauseofanissueparsingx509certificates.Formoreinformation,seetheGolangissuecrypto/x509:CANotAuthorizedForExtKeyUsageispremature .

Erlangv19.3.6.4

HAProxyv1.6.13

v1.10.17ReleaseDate:March28,2018

Updatesthefollowing:

Javapackagetoversion1.9On-DemandSevicesSDKtov0.20.0

Loggregatorreleasetov102.1

Routingreleasetov0.174.0

KnownIssuesClusterscalingorchangingtheErlangCookievaluerequiresclusterdowntime,andmightresultinfaileddeployments.Formoreinformation,seeClusterScalingKnownIssueandChangingtheErlangCookieValueKnownIssue.

Erlangv19.3.6.4

HAProxyv1.6.13

v1.10.14ReleaseDate:February26,2018

Features[SecurityFix]Requiresstemcell3445.28 .

FixedIssuesUpgradesnolongerfailwhenSYSLOGisnotconfigured.

Specialcharacterssuchas []^_!"#$%&()*+,-./\:;<=>`' nolongerleadtoissuesloggingintotheManagementUI.Youcannowusespecialcharacters,withtheexceptionof ` and ' ,inRabbitMQusernamesandpasswords.

Metricsarenowemittedwhenusingspecialcharactersinpasswords.

IfyouchangetheRabbitMQadminusername,theusernolongerlosesaccesstopreexistingvhosts.

KnownissuesClusterscalingorchangingtheErlangCookievaluerequiresclusterdowntimeandcanresultinfaileddeployments.Formoreinformation,seeClusterScalingKnownIssueandChangingtheErlangCookieValueKnownIssue.

ChangingnetworksorIPaddressesforthe RabbitMQ Server jobresultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .

WhenerrandrunrulesaresettoWhenChanged,OpsManagermightnotruntheerrandswhenthetilehasrelevantchanges.Formoreinformation,seeManagingErrandsinOpsManager .PivotalrecommendsleavingthedefaultrunrulesettoOn.

Erlangv19.3.6.4

HAProxyv1.6.13

v1.10.13ReleaseDate:January30,2018

FeaturesUpdateRabbitMQtov3.6.15.Formoreinformation,seeRabbitMQ3.6.15ReleaseNotes .

TheRabbitMQadministratorpasswordmustbeacombinationofuppercaseandlowercasealphanumerics.Specialcharacterssuchas []^_!"#$%&()*+,-

./\:;<=>`' canleadtoissuesloggingintotheManagementUIasthatuser.

Erlangv19.3.6.4

HAProxyv1.6.13

v1.10.12-WithdrawnReleaseDate:January24,2018WithdrawalDate:February5,2018

ThisreleasewaswithdawnduetoanissueupgradingwhentherearespecialcharactersintheRabbitMQadministratorcredentials.Skipthisversionandupgradetov1.10.13orlater.

Features[SecurityFix]Requiresstemcell3445.24 toaddressSpectrevulnerabilities.

Erlangv19.3.6.4

HAProxyv1.6.13

Features[SecurityFix]Requiresstemcell3445.23 toaddressGNUCLibraryvulnerabilities.

Knownissues

ClusterscalingorchangingtheErlangCookievaluerequiresclusterdowntimeandcanresultinfaileddeployments.Formoreinformation,seeClusterScalingKnownIssueandChangingtheErlangCookieValueKnownIssue.

Erlangv19.3.6.4

HAProxyv1.6.13

Features[SecurityFix]Requiresstemcell3445.22 toaddresstheMeltdownsecurityissue.FormoreinformationaboutMeltdown,seePivotalVulnerabilityReport:MeltdownandSpectreAttacks

Servicemetricsnowusesadrainscripttopreventmonittimeoutissues

On-Demand

smoke-tests errandisrenamedto on-demand-broker-smoke-tests

SomeCFtimeoutsforsmoketestshaveincreased

Erlangv19.3.6.4

HAProxyv1.6.13

v1.10.8ReleaseDate:December12,2017

KnownIssues

ClusterscalingorchangingtheErlangCookievaluerequireclusterdowntime,andmightresultinfaileddeployments.Formoreinformation,seeClusterScalingKnownIssueandChangingtheErlangCookieValueKnownIssue.

Erlangv19.3.6.4

HAProxyv1.6.13

v1.10.7ReleaseDate:December8,2017

UpdatetoRabbitMQv3.6.14(releasenotesavailablehere )

UpdatetoErlang19.3.6.4

Goupdate(securityfixes)

MinorupdatesinPre-Provisionedservice(multi-tenantbroker):

FixmisspellinginlogmessageImproveserviceofferingdescription

KnownIssuesClusterscalingorchangingtheErlangCookievaluerequireclusterdowntime,andmightresultinfaileddeployments.Formoreinformation,seeClusterScalingKnownIssueandChangingtheErlangCookieValueKnownIssue.

Erlangv19.3.6.4

HAProxyv1.6.13

v1.10.5ReleaseDate:November3,2017

Features

On-Demand

Requiresstemcell3445.16

Addsqueuedepth,memoryalarm,anddiskalarmmetrics

Pre-Provisioned

IncreasesmaxHAProxyconnectionsto64000

RemovesthestatsUIonHAProxyinstances

AllowsformoregranularTLSprotocolversionselection

Addsqueuedepth,memoryalarm,anddiskalarmmetrics

Fixesknownissuewheresomespecialcharactersintheadministratorpasswordcausedlogonproblems

KnownIssuesClusterscalingorchangingtheErlangCookievaluerequireclusterdowntime,andmightresultinfaileddeployments.Formoreinformation,seeClusterScalingKnownIssueandChangingtheErlangCookieValueKnownIssue.

Erlangv19.3.6.2

HAProxyv1.6.13

v1.10.3ReleaseDate:October16,2017

Features

On-Demand

On-demandRabbitMQclusters

MetricsforallinstancesautomaticallysenttotheFirehose

Singleconfigurationforsyslogforwardingforallinstances

Administratorpermissionsavailablethroughaservice-key

UpdatetoRabbitMQ3.6.12

UpdatetoRabbitMQ3.6.12forallinstancesinvolvesmajorchanges tothememorycalculations

On-demandmetricsprefixedwith p.rabbitmq tomatchtheservicename

Smoketestsrunforon-demandplans

Pre-Provisioned

UpdatetoRabbitMQ3.6.12

ThisversionofthetilecanonlybeinstalledinOpsManagerv1.11andv1.12environments

UpdatetoRabbitMQ3.6.12—thisinvolvesmajorchanges tothememorycalculations

HAProxyinstancecountcanbesettozeroifanexternalloadbalancerisused

KnownIssuesClusterscalingorchangingtheErlangCookievaluerequireclusterdowntime,andmightresultinfaileddeployments.Formoreinformation,see

ClusterScalingKnownIssueandChangingtheErlangCookieValueKnownIssue.

CertainspecialcharacterscannotbeusedwiththeRabbitMQadministratorpassword,andusingthemcanleadtoissuesloggingintotheManagementUIasthatuser.

Erlangv19.3.6.2

HAProxyv1.6.13

ViewReleaseNotesforAnotherVersionToviewthereleasenotesforanotherproductversion,selecttheversionfromthedrop-downlistatthetopofthispage.

UnlockingthePowerofOn-DemandRabbitMQforPCF

IntroductionRabbitMQforPivotalCloudFoundry(PCF)v1.10respondstothedemandsofPCFoperatorstoofferaRabbitMQon-demandclusterfortheirapplicationteams,inadditiontotheexistingsingle-nodeon-demandplan.Theon-demandclusterplanisaimedatworkloadsthatrequirethesameresiliencerequirementsasthePre-Provisionedoffering,butalsorequiretheirworkloadsbeisolated.

TheplatformoperationsteamcannowconfigureaRabbitMQforPCFclustertomeettheirbusinessrequirementsandempowerappdevelopmentteamstoself-servetheirownRabbitMQcluster.

RabbitMQforPCFv1.10alsoprovidessmoketestsfortheon-demandplanssothatoperationsteamscanvalidatetheappdeveloperworkflowforon-demandservices.SeeDedicatedInstanceSmokeTestProcess .

PlatformoperatorscannowoffertheirappdevelopersthreetypesofRabbitMQforPCFserviceplans:

Pre-provisioned—Forlighttomoderatemessagingneeds,thisserviceisfullyoperatedandmanagedbyplatformoperatorsasaservice.

On-demandsinglenode—Forapplicationteamsrequiringgreaterisolationthanprovidedbythepre-provisionedapproach.Appdevelopmentteamscanhavefullaccesstotheirownmessagebrokertoadapttheruntimeparameterstotheirrequirements.Formoreinformationontheseparameters,seeParametersandPolicies intheRabbitMQdocumentation.

On-demandcluster—Foranincreasedlevelofmessageresilienceandclusteravailability,aswellasthebenefitsofworkloadisolationmentionedabove.

Thistopicexplainshowtobenefitfromthetwoon-demandplansabove.

Forinformationaboutthepre-provisionedplan,seeDeployingtheRabbitMQPre-ProvisionedService.Forinformationonusingpre-provisionedplanstoisolateworkloads,seeCreatingIsolationwiththeTileReplicator.

On-DemandSingleNodePlanThisplanisdesignedtobesimpletoconfigure,deploy,anduse.ItgivesapplicationteamsfastaccesstothepoweroftheleadingopensourcemessagebrokerbackedbyBOSHtomeetallbutthemostdemandinghighavailabilityappmessagingrequirements.

Thisplancansuithigh-performanceworkloadsrequiringmessagingresilienceandasynchronousmessagingreplication.RabbitMQcopiesmessagestodiskforresilienceandallowsasynchronousmessagingreplicationthroughtheRabbitMQFederationplug-in.

Thisplanoffers:

FastaccesstoanisolatedinstanceofRabbitMQscopedfortheapplicationteams

OrgandSpaceAdministratoraccesstotheRabbitMQManagementUIsoapplicationteamscanhavefullcontroloverthenode

Updatesandupgradesinitiatedandcontrolledbytheoperatortokeeptheinstanceup-to-datewiththelatestsecuritypatchesandbugfixes

MessageresilienceprovidedthroughRabbitMQexchange,queueFederation,andShovelplugins.

On-DemandClusterPlanLikethesinglenodeplan,thisplanisdesignedtobesimpletoconfigure,deployanduse.ItgivesapplicationteamsfastaccesstothepoweroftheleadingOpenSourcemessagebrokerbackedbyBOSHtomeetallbutthemostdemandinghighavailabilityappmessagingrequirements.

Thisplancansuithighperformanceworkloadsrequiringmessagingresilience(copiedtodisk)andasynchronousmessagingreplicationthroughtheRabbitMQFederationplugin.Withthisplan,however,youalsoscaleoutRabbitMQforPCFtomultiplenodes.

Thisplanoffers:

Fastaccesstoanisolated,clusteredinstanceofRabbitMQscopedtotheapplicationteamOrgsandSpaces

AdministratoraccesstotheRabbitMQManagementUItogiveapplicationteamsfullcontroloverthecluster

Updatesandupgradesinitiatedandcontrolledbytheoperatortokeeptheinstanceup-to-datewiththelatestsecuritypatchesandbugfixes.

MessageresilienceprovidedbymirroringqueuesacrossRabbitMQnodes,andtheoptiontousetheFederationandShovelplugins.

GeneralPrinciplesoftheClusterPlanThefollowingaresomegeneralprinciplestobeawareofwhenconfiguringtheclusterplan:

DesignedforConsistency

RabbitMQclusteringisnotprimarilyasolutionforincreasedavailability.Instead,itisdesignedforconsistencyandpartitiontolerance,asdescribedintheCAPtheorem .RabbitMQclusteringprovidesincreasedmessageconsistencythroughqueuemirroring.Thismeansthatmessagesaccessedinonequeueareexactlythesameasinanotherqueue.Formoreinformation,seeConsistencyorAvailabilityTradeoff.

Otheroptionscanbeusedforavailabilityrequirements,suchastheuseoffederationbetweenexchangesorqueues.

ForadetaileddescriptionofdistributedRabbitMQbrokers,seetheRabbitMQdocumentation .

NumberofNodes

Everynodeintheon-demandclustermaintainsacompletedatabaseofallmetadata,andallchangestothemetadataareconfirmedbyeverynodeinthecluster.Therefore,goingbeyondsevennodescanhaveasignificantnegativeimpactonperformance.Foroptimumresilienceandperformance,Pivotalrecommendsthreenodesformostworkloads.

NetworkLatency

RabbitMQclustersareonlyrecommendedfordeploymentinlowlatencynetworks,whichnormallymeansthatitisnotadvisabletodeploytheseclustersacrossavailabilityzones(AZs).ThestabilityandperformanceoftheRabbitMQclusterisheavilyinfluencedbytheworkloadonthenodes,replicationchoices,andnetworklatency.

Forthisreason,PivotalrecommendsthatyoudeployRabbitMQclustersintoasingleOpsManagerAZ.However,wheredifferentAZsareinthesamedatacenter,withreliablelowlatencylinks,spanningAZscanbeused.

ForcloudIaaSdeployments,Pivotaldoesnotrecommendthatdeploymentsspanregions.Forexample,inAmazonWebServices(AWS)terms,deployingaRabbitMQclusteracrossAZswithinaregionshouldprovidehighenoughnetworkperformancetopreventimpactingclusterstability.However,deployingacrossAWSregionsislikelytoleadtoclusterinstability.Formoreinformation,seetheAWSdocumentation .

ConsistencyorAvailabilityTradeoffInadistributedmessagingsystem,atradeoffmustbemadebetweenavailabilityorconsistencywhenanetworkpartitioneventoccursandoneormorenodesarenotabletocommunicatewitheachother.TheclusterplanletsoperatorsdecidehowtheywanttheRabbitMQclustertoreactintheeventofanetworkpartition.

Pivotalrecommendskeepingthedefaultclusterpartitionoptionof pause_minority becausethissatisfiesmostusecases.Choosingthe pause_minority

partition-handlingstrategyfavorsmessageconsistencyoveravailability.Formoreinformationabouttheoptionsforhandlingpartitions,seetheRabbitMQdocumentation .ForadetaileddescriptionoftheoptionsavailableinRabbitMQforPCF,seeClusteringandNetworkPartitions.

Hereisanexampleofhow pause_minority works.IfyoucreateaRabbitMQclusterwiththreenodesandonenodebecomesunabletocommunicatewiththeothertwo,thisnodeisintheminority.Thenodethatisintheminorityispaused,andtheothertwonodescontinueservingtraffic.Ifeachofthenodeslosesconnectivitywiththeothertwo,thentheentireclusterispausedtopreservedatasincenomajoritycanbeestablished.Theclusterhealswhentwoormorenodesareabletocommunicatewitheachother.

RabbitMQQueueAvailabilityItisimportanttobeawarethatmessagequeueavailabilityisdifferentfromclusteravailability.So,havingclusteravailabilitydoesnotmeanthatallofthemessageswithinthequeuesarealsoavailable.

Bydefault,queueswithinaRabbitMQclusterarelocatedonasinglenode—thenodeonwhichtheywerefirstdeclared.However,queuescanbeconfiguredtomirroracrossmultiplenodes,sothatanymessagepublishedtothequeueisreplicatedtoallmirrors.Enablingmirroringcanhaveanegativeimpactonqueueperformancebecausemessagesmustbecopiedtoallmirrorsbeforebeingacknowledged.

Eachmirroredqueueconsistsofonemasterandoneormoremirrors,withtheoldestmirrorbeingpromotedtothenewmasteriftheoldmasterdisappearsforanyreason.Consumersareconnectedtothemasterregardlessofwhichnodetheyconnectto,andmirrorsdropmessagesthathavebeen

acknowledgedatthemaster.Queuemirroringenhancesqueueavailability,butdoesnotdistributeloadacrossnodesbecauseeachoftheparticipatingnodesmuststilldoallthework.

Appdevelopersmustdecideiftheywanttousequeuemirroringanddeterminethepolicytheywanttoapplytotheirqueues.Thesechoiceshavesignificantimpactontheavailabilityoftheirqueues.Formoreinformation,seetheRabbitMQdocumentation .

Unlikethepre-provisionedplan,theclusterplandoesnotshipwithadefaultloadbalancer.Therefore,developersmustconfiguretheirapptousethearrayofhostsprovidedin VCAP_SERVICES .Ifdevelopersenablequeuemirroring,theymustalsoensuretheirappshavere-trylogicandreconnectionlogicthatiteratesovertherangeofhostsprovided.MostcommonRabbitMQclientshavethislogicbuiltintothem.Formoreinformation,seetheSpringAMQPdocumentation .

Becausetheclusterplanisdesignedtoenableapplicationteamstoself-serve,nothavingaloadbalancerinfrontoftheRabbitMQclusterhasthesebenefits:

Manageresourcesbetter,asfewerVMsareneeded.

Helpwithtroubleshooting.ClientIPisnowtheIPofthesourcecontainerandnottheHAProxy.

Reducethenumberofhopsbetweenappsandbroker.Thishelpswithlatency.

Determinequeueplacement.Thismakessenseforlargerscaledeployments.

Empowerapplicationteamstomanagetheirclusterinthebestwayfortheirapp.

Requirere-trylogicinanappifitneedsHAaccesstoaqueue.Thus,allnodescanroutetoaqueueifitisavailable.

ManagingOn-DemandResourcesThroughPlansInconfiguringeachplan,thereareanumberofoperationalcontrolsthatplatformoperationsteamscanusetomanagetheresourcesconsumedbyon-demandRabbitMQ:

ControlAccess—Operatorscanchoosetheappdevelopmentorgsandspacesforwhichtheplansareavailableandvisible.Eachplancanbeenabledordisabled,andserviceaccessandvisibilitycaneitherbeglobal,orenabledperorgandspacethroughthecommandline.

Forexample,youmaydecidetoenablethesinglenodeon-demandplanacrossallapplicationteamstomeettheirdemandtoisolatetheirworkload.Youmaythenchoosetooffertheon-demandclusterplanonlytoasubsetofapplicationteamswhorequiretheextraresources.

SetQuotas—Youcansetaglobalquotaforallon-demandinstancesthattakesprecedenceovereachplanquota.Thisletsyouguardagainsttheriskofover-committingresources,butallowstheflexibilityofover-committingeachplan,soyoucanmeetthefluctuatingdemandsofyourappdevelopers.

ControlResourceConsumption—Eachplanoffersmorefine-grainedcontroloverindividualplanresourceconsumption.Atthehighestlevel,youcanusetheplanquotatocontrolthenumberofinstancesthatcanbedeployedwithinafoundation.Foreachplan,youcanalsoconfigurethenumberofnodesthatconstituteacluster(3,5,or7),theinstancetype,andpersistentdiskstoragesizetobestsuityourrequirements.

Monitor—Youcanmonitorthenumberofinstancesthathavebeendeployedagainstthequotayouhavesetsothatyoucanplanfutureresourcerequirements.

CustomizingPlanOptionsTheRabbitMQforPCFon-demandplansexposeanumberofconfigurationoptions.Inmostcases,thedefaultconfigurationsmeetmostappdemands.However,itisimportantforanoperationsteamtoconsidertheoptionstoensurethattheyprovidethebestservicetotheirappdevelopers.Thissectionexplainstheseoptions.

ConfigurationOptions

SingleNodeandClusterPlansEnable/Disableplan

Determinewhichorgsandspacescanseeandaccesstheplan

SetServiceInstanceQuota

SelectAZplacement(whereapplicable)

SetRabbitMQinstancesize(CPUandMemory)

Setpersistentdisksize(PersistedMessageStore)fortheRabbitMQinstance.Ensurethesizeofthepersistentdiskisatleasttwiceaslargeasthe

instancememory.

ClusterPlanOnlySetnumberofnodesto3,5,or7

Determinenetworkpartitionbehavior.SeeConsistencyorAvailabilityTradeoffabove.

ThingsThatArePreconfiguredThefollowingarepre-configuredforboththesinglenodeandtheclusterplans:

Metrics—EmittedtotheLoggregatorFirehoseforallon-demandinstances.ThepollingintervalissetintheOpsManager,intheMetricspollingintervalfield,intheRabbitMQtaboftheRabbitMQforPCFtile.Duetotheimpactofsomeoftheclustersettingsdetailedbelow,PivotalstronglyrecommendsthatyoumonitortheexposedmetricsandconfigurealarmsasrecommendedinMonitoringandKPIsforOn-DemandRabbitMQforPCF.SeealsoMonitoringOn-DemandRabbitMQClustersbelow.

Logs—RabbitMQon-demandinstancelogsareforwardedusingthesameconfigurationascontainedintheSyslogtaboftheRabbitMQforPCFtile.

Diskfreespacelimit—Thediskfreespacelimitissetto150%ofRAMoftheinstancetypeyouselect.Forexample,ifyouselectaninstancetypewith10GBofRAM,thediskfreespacelimitissetto15GB.Acluster-widealarmistriggerediftheamountoffreediskspacedropsbelowthis,andallpublishersareblocked.InstancesmustbeconfiguredtohavepersistentdisksthatareatleasttwicethesizeofinstanceRAM.Formoreinformation,seetheRabbitMQdocumentation .

Memorythresholdfortriggeringflowcontrol—Thresholdatwhichflowcontrolistriggeredissetto40%oftheinstanceRAM.Thismeansthatwhenthealarmistriggered,allconnectionspublishingmessagesareblockedcluster-wideuntilthealarmiscleared.

Forexample,ifyouselectaninstancetypewith10GBofRAM,whenmorethan4GBofmemoryisused,allpublishingconnectionsareblocked.Formoreinformation,seeMemoryAlarms intheRabbitMQdocumentation.

Memorypagingthreshold—ThisisthelevelatwhichRabbitMQtriestofreeupmemorybyinstructingqueuestopagetheircontentsouttodisk.Thisisdonetotrytoavoidreachingthehighwatermarkandblockingpublishers.Thisthresholdissetto50%oftheconfiguredhighwatermark,whichis20%ofconfiguredmemory.Formoreinformationonmemorycalculation,seeChangestoMemoryAllocationwhenUpgrading .

Forexample,ifyouselectaninstancetypewith10GBofRAM,whenmorethan2GBofmemoryisused,allqueuesstartwritingallqueuecontentstodisk.Formoreinformation,seetheRabbitMQdocumentation .

MonitoringOn-DemandRabbitMQClustersItisimportanttomonitorandcomparethenumberofinstancesthathavebeendeployedagainstthequotayousetviathemetricexposedontheFirehose.

Eachinstanceispre-configuredtoemitmetricstotheFirehoseandcanbeidentifiedbythe deployment tag,whichhastheserviceinstanceID.ItisimportanttomonitorthesemetricsasrecommendedinMonitoringandKPIsforOn-DemandRabbitMQforPCF.

Note:Aloadbalancer,suchasHAProxy,isnotdeployedwithon-demandclusterplans.

On-DemandServiceArchitectureThistopicdescribesthearchitectureforon-demandRabbitMQ®forPivotalCloudFoundry(PCF).

Forinformationaboutarchitectureoftheolder,pre-provisionedservice,seeDeployingtheRabbitMQ®Service.

BOSH2.0andServiceNetworksWhenyoudeployPCF,youmustcreateastaticallydefinednetworktohostthecomponentvirtualmachinesthatconstitutethePCFinfrastructure.

PCFcomponents,liketheCloudControllerandUAA,runonthisinfrastructurenetwork.InPCFv2.0andearlier,on-demandPCFservicesrequirethatyouhostthemonanetworkthatrunsseparatelyfromthisnetwork.

Cloudoperatorspre-provisionserviceinstancesfromOpsManager.Then,foreachservice,OpsManagerallocatesandrecoversstaticIPaddressesfromapre-definedblockofaddresses.

Toenableon-demandservicesinPCFv2.0andearlier,operatorsmustcreateaservicenetworksinOpsManagerDirectorandselecttheServiceNetworkcheckbox.Operatorsthencanselecttheservicenetworktohoston-demandserviceinstanceswhentheyconfigurethetileforthatservice.

DefaultNetworkandServiceNetworkOn-demandPCFservicesrelyontheBOSH2.0abilitytodynamicallydeployVMsinadedicatednetwork.Theon-demandservicebrokerusesthiscapabilitytocreatesingle-tenantserviceinstancesinadedicatedservicenetwork.

On-demandservicesusethedynamically-provisionedservicenetworktohostthesingle-tenantworkerVMsthatrunasserviceinstanceswithindevelopmentspaces.ThisarchitectureletsdevelopersprovisionIaaSresourcesfortheirserviceinstancesatcreationtime,ratherthantheoperatorpre-provisioningafixedquantityofIaaSresourceswhentheydeploytheservicebroker.

Bymakingservicessingle-tenant,whereeachinstancerunsonadedicatedVMratherthansharingVMswithunrelatedprocesses,on-demandserviceseliminatethe“noisyneighbor”problemwhenoneapplicationhogsresourcesonasharedcluster.Single-tenantservicescanalsosupportregulatorycompliancewheresensitivedatamustbecompartmentalizedacrossseparatemachines.

Anon-demandservicesplitsitsoperationsbetweenthedefaultnetworkandtheservicenetwork.Sharedcomponentsoftheservice,suchasexecutivecontrollersanddatabases,runcentrallyonthedefaultnetworkalongwiththeCloudController,UAA,andotherPCFcomponents.Theworkerpooldeployedtospecificspacesrunsontheservicenetwork.

ThediagrambelowshowsworkerVMsinanon-demandserviceinstancerunningonaseparateservicesnetwork,whileothercomponentsrunonthedefaultnetwork.

RequiredNetworkingRulesforOn-DemandServicesPriortodeployinganyservicetilethatusestheon-demandbroker(ODB),theoperatormustrequestthenetworkconnectionsneededtoallowvariouscomponentsofPivotalCloudFoundry(PCF)tocommunicatewithODB.ThespecificsofhowtoopenthoseconnectionsvariesforeachIaaS.

Thefollowingtableshowstheresponsibilitiesofthekeycomponentsinanon-demandarchitecture.

KeyComponents TheirResponsibility

BOSHDirector

CreatesandupdatesserviceinstancesasinstructedbyODB

BOSHAgentBOSHincludesanAgentoneveryVMthatitdeploys.TheAgentlistensforinstructionsfromtheDirectorandcarriesoutthoseinstructions.TheAgentreceivesjobspecificationsfromtheDirectorandusesthemtoassignarole,orJob,totheVM.

BOSHUAA AsanOAuth2provider,BOSHUAAissuestokensforclientstousewhentheyactonbehalfofBOSHusers.

ERT Containstheappsthatareconsumingservices

ODB InstructsBOSHtocreateandupdateservices,andconnectstoservicestocreatebindings

Deployedserviceinstance

Runsthegivendataservice(forexample,thedeployedRedisforPCFserviceinstancerunstheRedisforPCFdataservice)

Regardlessofthespecificnetworklayout,theoperatormustensurenetworkrulesaresetupsothatconnectionsareopenasdescribedinthetablebelow.

Thiscomponent…

Mustcommunicatewith…

DefaultTCPPort Communicationdirection(s) Notes

BOSHDirector

BOSHUAA

8443 One-way Thedefaultportsarenotconfigurable.

ODBDeployedserviceinstances

15672(RabbitMQManagementUI)

One-wayThisconnectionisforadministrativetasks.Avoidopeninggeneraluse,app-specificportsforthisconnection.

ODB ERT 8443 One-way Thedefaultportisnotconfigurable.

ErrandVMs

DeployedServiceInstances

5671-2(AMQP/AMQPS)

One-way Thedefaultportisnotconfigurable.

BOSHAgent BOSHDirector 4222 Two-way

TheBOSHAgentrunsoneveryVMinthesystem,includingtheBOSHDirectorVM.TheBOSHAgentinitiatestheconnectionwiththeBOSHDirector.Thedefaultportisnotconfigurable.

DeployedappsonERT

Deployedserviceinstances

5671-2(AMQP/AMQPS)

61613-4(STOMP/STOMPS)

1883,8883(MQTT/MQTTS)

One-way Thisconnectionisforgeneraluse,app-specifictasks.Avoidopeningadministrativeportsforthisconnection.

ERT ODB 8080 One-wayThisportmaybedifferentforindividualservices.Thisportmayalsobeconfigurablebytheoperatorifallowedbythetiledeveloper.

InstallingandConfiguringtheOn-DemandServiceThistopicprovidesinstructionstoPivotalCloudFoundry(PCF)operatorsabouthowtoinstall,configure,anddeploytheRabbitMQforPCFtiletoprovideon-demandservice.

TheRabbitMQopensourceproductprovidesadditionaldocumentation.FormoreinformationaboutgettingstartedwithRabbitMQandensuringproductionreadiness,seetheProductionChecklistintheRabbitMQDocumentation .

PrerequisitesforDeployingtheOn-DemandServiceBeforedeployingRabbitMQforPCFasanon-demandservice,youmustensurethattherequirednetworkrulesareinplacetoallowvariouscomponentstocommunicate.

SeeRequiredNetworkingRulesforOn-DemandServicesfordetailsonthenetworkconnectionsthatmustbeopentoenabletheon-demandservice.

Forinformationabouttheon-demandservicearchitecture,seeOn-DemandServiceArchitecture.

DownloadandInstallRabbitMQforPCF1. DownloadtheproductfilefromPivotalNetwork .

2. NavigatetotheOpsManagerInstallationDashboardandclickImportaProducttouploadtheproductfile.

3. UndertheImportaProductbutton,click+nexttotheversionnumberofRabbitMQforPCF.Thisaddsthetiletoyourstagingarea.

4. ClickthenewlyaddedRabbitMQforPCFtile.Thisletsyoubeginconfiguringthetile.Theinstallationiscompletewhenyouapplythechangesfromtheconfiguration.

ConfigureOn-DemandRabbitMQforPCFTheconfigurationscreenbelowappearswhenyouclicktheRabbitMQforPCFtileinOpsManager.Anorangecirclebesideatabindicatesthatyoumustcompleteaconfigurationinthetab.Agreencheckmarkindicatesthatthetabispreconfiguredandyoumayoptionallychangeitssettings.

Note:Forinstructionsonhowtoinstall,configure,anddeploytheRabbitMQforPCFtileasapre-provisionedservice,seeInstallingandConfiguringthePre-ProvisionedService.

WhichSettingsTabstoConfigurefortheOn-DemandServiceConfigurethefollowingtabsfortheon-demandservice:

RabbitMQSettingsTab Instructions

AssignAZsandNetworks ConfigureAZsandNetworks

RabbitMQ SettheMetricspollingintervalonly.SeeWhatAreMetrics.

Syslog SetupSyslogForwarding

DedicatedInstance:GlobalSettings ConfigureGlobalSettings

DedicatedInstance:SingleNodePlans ConfiguretheServicePlan

DedicatedInstance:ClusterPlan ConfiguretheServicePlan

Errands Errands

Stemcell VerifytheStemcell

ConfigureAZsandNetworksFollowthestepsbelowtoconfiguretheAZsandnetworks.

1. ClickAssignAZsandNetworks.

2. ConfigurethefieldsontheAssignAZsandNetworksasfollows:

Field Instructions

Important:YoucannotchangetheregionsornetworksafteryouhaveclickedApplyChangesintheApplyChangesfromYourConfigurationbelow.

Placesingletonjobsin

SelecttheregionthatyouwantforsingletonVMs.PCFcreatestheRabbitMQbrokerinthisAZ.

Balanceotherjobsin

Selectadditionalregion.Thisselectiondoesnotaffecttheon-demandRabbitMQforPCFservice.

Network

SelectanetworkfortheRabbitMQOn-DemandBroker.

ThisshouldbeaseparatenetworkfromtheoneyouselectforServiceNetwork.FormoreinformationabouttheDefaultNetwork,seeDefaultNetworkandServiceNetwork.

Typically,youselectthenetworkusedforthePivotalElasticRuntimecomponents.

ServiceNetwork

Selectaseparatenetworkthattheon-demandserviceinstancesrunon.

Atypicalpracticeistoputallon-demandservicesonasinglenetwork,separatefromthenetworkthatERTandtheOn-DemandBrokerrunon.ForinformationabouttheServiceNetwork,seeDefaultNetworkandServiceNetwork.

Thisfieldisalsorequiredforthepre-provisionedservice,thoughinthatcase,itdoesn’tmatterwhichnetworkyouselect.

3. ClickSave.

SettheMetricsPollingIntervalOntheRabbitMQtab,setametricspollinginterval.Formoreinformation,seeWhatareMetrics.

ConfigureLoggingtoMonitorRabbitMQforPCFPivotalrecommendsthatyouconfigureloggingtomonitorthehealthofRabbitMQforPCF.FollowSetUpSyslogForwardingtoconfigurelogging.

ConfigureGlobalSettingsFollowthestepsbelowtoconfigureglobalsettings.

1. ClickDedicatedInstance:GlobalSettingsandconfigurethefollowing:

Serviceinstancequotamin:0,max:50setthetotalnumberofdedicatedserviceinstanceswhichcanbedeployed.Formoreinformation,seeSettingLimitsforOn-DemandServiceInstances.

VMoptions:

Allowoutboundinternetaccess(IaaS-dependent).Thischeckboxmustbetickedtoallowexternallogforwarding,sendingbackupartifactstoexternaldestinations,andcommunicatingwithanexternalBOSHblobstore.

2. ClickSave.

ConfiguretheServicePlanInordertoenabletheon-demandserviceplan(s),youmustenableoneorbothoftheseplans:

DedicatedInstance:SingleNodePlan—ConfiguresasinglenodeofRabbitMQServer.Youmustcompletetherequiredfieldsonthistabevenifyoudisableaccesstothisplan(seeCFServiceAccessinthetablebelow).

DedicatedInstance:ClusterPlan—ConfiguresaclusterwiththreetosevenRabbitMQServernodes.Thisplanisdisabledbydefault.

WARNING:ChangingtheNetworkorServiceNetworkafteryouhaveconfiguredthem,orchangingtheirIPconfigurations,resultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .

Note:Iftheon-demandserviceisnotenabled,theon-demandbrokerisdeployedalongsidetheRabbitMQinstallation,butitisnotavailablein

1. Choosethededicatedserviceinstanceyouwanttoconfigure:

SingleNodePlan(completerequiredfieldsevenifyoudisablethisplan):

ClusterPlan:

theMarketplace.

2. Configurethefieldsasfollows:

Field Instructions

Enablethisplan (ClusterPlanOnly)Toenable,selectPlanEnable.

CFServiceAccess

Enableordisableaccesstothisplan,orleaveaccessunchanged.

Ifyouenabletheclusterplan,thedefaultsettingisEnableServiceAccess.Ifyouchangethisdefaultsetting,thesmoketestsfail.Therefore,settheDedicatedInstanceSmokeTestserrandtoOffifyouenabletheclusterplanandchangethisdefault.Formoreinformation,seeErrands.

EnableServiceAccess—Givesaccesstoallorgs,anddisplaystheserviceplantoalldevelopersintheMarketplace.DisableServiceAccess—Disablesaccesstoallorgs,andhidestheserviceplantoalldevelopersintheMarketplace.ThissettingcannotbeselectedatalatertimeintheUI.LeaveServiceAccessUnchanged—Keepsanyexistingaccesssettings,andonlydisplaystheserviceplanintheMarketplacetomembersoforgsthathaveaccesstotheplan.YoucanchangetheaccesssettingslaterusingthecfCLI.Forinstructions,seeControllingAccesstoServicePlansbyOrg.

Planname Acceptthedefaultorenteraname.ThisisthenamethatappearsintheMarketplace.

Plandescription

Acceptthedefaultorenteradescription.

Serviceinstancequota

Enterthemaximumnumberofon-demandserviceinstancesthatcanbeavailableatonetime.Formoreinformation,seeSettingLimitsforOn-DemandServiceInstances.

PlanAcceptthedefaultorenterinformationabouttheplantohelpyourappdevelopers.

featuresNumberofnodes

(ClusterPlanOnly)Enter3,5or7.Thissettingonlyaffectsnewserviceinstances.Previouslydeployedserviceinstancesarenotupdated.

Networkpartitionbehaviour

(ClusterPlanOnly)Selectpauseminorityorautoheal.Pivotalrecommendsusingpauseminority.Formoreinformation,seeConsistencyorAvailabilityTradeoff.

RabbitMQVMType

SelectalargeVMtype.Theplancreatesaserviceinstanceofthissize.

Persistentdisktype

ThisiswhereRabbitMQpagesmessagestodisk.ServiceinstancedeploymentsfailifthisvalueislessthantwicethevolumeofRAMoftheselectedRabbitMQVMType.Formoreinformation,seeResourceTypesandDiskSizeConcernsbelow.

AZplacement

ThisfieldisavailableafteryoucompletetheAssignAZsandNetworkspage.

Forasingle-nodeplan,selectoneormoreAZs.Foraclusterplan,selectonlyoneAZ.PivotalrecommendsdeployingaclusterplantoonlyoneAZtominimizerisksduetonetworklatencyandpartitions.SeeNetworkLatencyandConsistencyorAvailabilityTradeofffordetails.

Ifyouchangethisselectionafterdeployment,existinginstancesarenotaffectedbythechange.SeeDeterminewhichAZsaServiceInstanceUses.

3. ClickSave.

DeterminewhichAZsaServiceInstanceUses

TodeterminewhichAZsaserviceinstanceisplacedin,dooneofthefollowing:

RetrievetheserviceGUIDusingthe cf service SERVICE_INSTANCE --guid commandandthenruntheBOSH instances commandfortheservice-instance_GUID deployment.

Withsyslogforwardingenabled,inspecttheservicebrokerlogswhenrunningtheUpgradeAllServiceInstanceserrand.Foreachexistingserviceinstance,thelogmessageincludestheserviceinstanceGUIDandtheAZstheserviceinstanceisrunningin.

ResourceTypesandDiskSizeConcerns

ItispossibletoconfiguretheVMtypeforRabbitMQandthesizeofthepersistentdiskthatisgoingtobeattachedtotheRabbitMQinstances.SuggestedvalueistwicetheamountofRAMoftheselectedVMtype.Formoreinformation,seethePivotalRabbitMQ documentation.

RabbitMQraisesalarmswhendiskspacedropsbelowtheconfiguredlimit.Incorrectdisksizesmightcausethedeployedinstancenottostart.RabbitMQdeclinestostartifthereisnotenoughspaceavailableaccordingtothethreshold.

Important:IfyouchangethisconfigurationafteryouhaveselectedAZsanddeployedserviceinstances,existinginstancesarenotplacedinthenewlyconfiguredAZswhentheUpgradeAllServiceInstanceserrandisrun.Thispreventsre-creationoftheVMsindifferentAZs,whichcanleadtodataloss.Allnewserviceinstances,however,willbecreatedinthenewlyconfiguredAZs.

On-demandinstancesareconfiguredwithathresholdsettothe150%ofthememory(RAM)oftheVM.Usethefollowingtableasaguidewhenselectingthesizeofthepersistentdisk.

ThefollowingtableshowsanexampleofpossibleRAMvalues,absoluteminimalvaluebelowwhichRabbitMQdeclinestostart,andthedisksizesuggestedforanaverageusecase.

RAM Freediskalarmthreshold(1.5xRAM) Suggesteddisksize(2xRAM)

10GB 15GB 20GB

16GB 24GB 32GB

32GB 48GB 64GB

Formoreinformation,seethefollowing:

MemoryandDiskAlarms

DiskAlarms

VerifytheStemcell1. ClickStemcell.

2. Verifyand,ifnecessary,importanewstemcellversion.Formoreinformation,seetheinformationaboutimportingthestemcellforyourIaaS:AWS,Azure ,GCP ,orvSphere .

ApplyChangesfromYourConfigurationYourinstallationisnotcompleteuntilyouapplyyourconfigurationchanges.Followthestepsbelow:

1. ReturntotheOpsManagerInstallationDashboard.

2. ClickApplyChanges.

ErrandsWhendeployingorupdatingRabbitMQforPCF,OpsManagercanoptionallyrunaseriesofPost-DeployErrands.Anexampleisthe Smoke

Testserrand,

whichchecksthehealthoftheRabbitMQclusterafteradeployorupgrade.

Youcandecidewhethertorunerrandsbytogglingthemonoroffbeforeanupdate.Thisisaone-timesettingontheinstallationdashboard:

However,ifnecessary,youcanchangethesedefaultsbyclickingErrandsintheRabbitMQforPCFSettingstab.

Important:AsofRabbitMQforPCFv1.9.0,allpost-deployerrandsareonbydefault.Pivotalrecommendskeepingthesedefaults,becausethesmoketestscanencounterunexpectedissues,andon-demandinstancesofRabbitMQforPCFmayfallbehindiftheUpgradeAllServiceInstanceserrandisnotonbydefault.

Formoreinformationonerrandrunrules,seeErrandRunRules .

Post-DeployErrands

Errand Description

BrokerRegistrar

Makesthepre-provisionedRabbitMQserviceplansavailableintheMarketplace

SmokeTests

Checksthatapre-provisionedRabbitMQserviceinstancecanbeboundtoaCloudFoundryapp,andthattheappcanpublishandsubscribetoaRabbitMQcluster.SeePre-ProvisionedInstanceSmokeTests.

RegisterOn-DemandServiceBroker

Makestheon-demandRabbitMQserviceplansavailableintheMarketplace.IfyouchangetheServicePlanConfiguration,youmustrunthiserrandinorderforthechangestobereflectedintheMarketplace.

DedicatedInstanceSmokeTests

Checksthaton-demandRabbitMQserviceinstancescanbeboundtoaCloudFoundryapp,andthattheappcanpublishandsubscribetoaRabbitMQcluster.SeeDedicatedInstanceSmokeTestsbelow.

UpgradeAllServiceInstances

On-demandinstancesareupdatedandredeployediftherearechangestotheDedicatedInstancesettingsorthetileisupgraded.Ifthiserrandissetto“Off”or“WhenChanged”,updatestoDedicatedInstancesettingswillnotbeappliedtoexistingserviceinstances.Pivotalstronglyrecommendsthatthiserrandisconfiguredtoalwaysrun.

Pre-DeleteErrands

Errand Description

BrokerDeregistrarRemovesthepre-provisionedRabbitMQservicefromtheMarketplaceanddeletesallassociatedserviceinstancesandbindings

DeleteAllServiceInstancesUnbindsanddeletesexistingon-demandserviceinstances.Thedurationofthiserranddependsonthenumberofdeployedon-demandinstances.

DeregisterOn-DemandServiceBroker

Removestheon-demandRabbitMQservicefromtheMarketplace

DedicatedInstanceSmokeTestsSmoketestsonlyrunagainsttheclusterplan.Formoreinformationaboutthesmoketestsprocess,seeSmokeTests.

CreateanAdminUserforaServiceInstanceIfyouwanttogiveappdevelopersadminprivilegestotheRabbitMQManagementUI,youcancreateanadminuserforaserviceinstanceandsharetheusercredentialswithappdevelopers.

Bothoperatorsandappdeveloperscanusethisprocedure.

TocreateanadminuseronaRabbitMQinstancedothefollowing:

1. Runthiscommandtocreateaservicekey:

cf create-service-key SERVICE_INSTANCE SERVICE_KEY -c '{"tags": "administrator"}'

where:

SERVICE_INSTANCE isthenameyousuppliedwhenyouran cf create-service .SERVICE_KEY isanameyouchoosetoidentifytheservicekey.

Forexample:

$ cf create-service-key my-instance my-admin-key -c '{"tags":"administrator"}'

Creating service key my-admin-key for service instance my-instance as user@example.com...OK

2. Runthiscommandtogettheadminusercredentials:

cf service-key SERVICE_INSTANCE SERVICE_KEY wherethevariablesarethesameasabove.

ThisreturnsaDashboardURLcontainingtheadmincredentials,whichcanbeusedtoaccessthemanagementUI.Forexample:

$ cf service-key my-instance my-admin-key

Getting key my-admin-key for service instance my-instance as user@example.com...{ "dashboard_url": "https://my-instance.bosh-lite.com/#/login/admin-username/admin-password", "username": "admin-username", "password": "admin-password", ...}

RabbitMQServerSettingsThatCannotBeDisabledThefollowingpluginsareenabledbydefaultandcannotbedisabled:

rabbitmq_management

rabbitmq_mqtt

rabbitmq_stomp

rabbitmq_federation

rabbitmq_federation_management

rabbitmq_shovel

rabbitmq_shovel_management

SmokeTestsRabbitMQforPCFrunsasetofsmoketestsduringinstallationtoconfirmsystemhealth.

SmokeTestStepsThesmoketestsperformthefollowingforeachavailableserviceplan:

1. Targetstheorg system andcreatesaspacetorunthetests.

2. DeploysaninstanceoftheCFRabbitMQExampleApp tothisspace

3. CreatesaRabbitMQserviceinstanceandbindsittotheCFRabbitMQExampleApp

4. ChecksthattheCFRabbitMQExampleAppcanwritetoandreadfromtheRabbitMQserviceinstance

5. Cleansupalldeployedapplicationandallitsservicebindings.Finally,thecfspaceisdeleted.

TroubleshootingIferrorsoccurwhilethesmoketestsrun,theyaresummarizedattheendoftheerrandlogoutput.Detailedlogscanbefoundwherethefailureoccurs.

Whenencounteringanerrorwhenrunningsmoketests,itcanbehelpfultosearchthelogforotherinstancesoftheerrorsummaryprintedattheendofthetests,forexample, Failed to target Cloud Foundry .Lookoutfor TIP: ... inthelogsnexttoanyerroroutputforfurthertroubleshootinghints.

Note:Smoketestsfailunlessyouenableglobaldefaultapplicationsecuritygroups(ASGs).YoucanenableglobaldefaultASGsbybindingtheASGtothe system orgwithoutspecifyingaspace.ToenableglobaldefaultASGs,use cf bind-running-security-

groupp>

UpgradingRabbitMQforPCFThisproductenablesautomatedupgradesbetweenversionsoftheproductandisdeployedthroughOpsManager.Insomecases,youmightberequiredtotaketheclusteroffline.Whenthisisnecessary,itisclearlynotedinthereleasenotesforthatversion.

TheupgradepathsforeachversionaredetailedatPivotalNetwork-RabbitMQforPCFpage .

DowntimeWhenUpgradingAguidefordowntimeduringupgradedeploymentsisshowninthetablebelow.Insomecases,theclusterremainsavailableduringatileupgrade,butindividualqueuesonclusternodesmaybetakenoffline.

Thisisonlyaguide,sobeforeupgrading,checkthereleasenotesfortheversionyouareupgradingto.

UpgradeType WillDowntimeBeRequiredForThisUpgrade/Update

MajorTileVersion TheRabbitMQclusteristakenofflineforthedurationoftheupgrade.

MinorTileVersion TheRabbitMQclusteristakenofflineforthedurationoftheupgrade.

PatchTileVersion

Normallythesearerollingdeploymentswitheachnodebeingupdatedinturn.Inthesecasestheclusterremainsavailable,butindividualqueuesmaybetakenofflineaseachnodeisrestarted.Therearespecificmigrationpathsthatrequiredowntime,whichareidentifiedinthereleasenotesforthatversion.

Stemcell-OnlyPatchTileVersion

Wherethepatchupdateisonlyanewstemcellversionthesearerollingdeploymentswitheachnodebeingupdatedinturn.Inthesecasestheclusterremainsavailable,butindividualqueuesmaybetakenofflineaseachnodeisrestarted.

NotesontheUpgradeProcessReviewthefollowingbeforestartinganupgradeofRabbitMQforPCF:

Upgradingtoanewerversionoftheproductdoesnotcauseanylossofdataorconfiguration.

ItmaytakebusyRabbitMQnodesalongtimetoshutdownduringtheupgradeandyoumustnotinterruptthisprocess.

ThebenefityougetfromstemcellrollingupgradesdependsonhowyouhaveconfigurednetworkpartitionhandlingandtheResourceConfigtab.AnHAProxyinstancecountof2andaRabbitMQnodecountof3arerequiredforrollingstemcellupgrades.Asofv1.7.7,thesecountsarethedefault.Formoreinformation,seeClusteringandNetworkPartitions.

ThelengthofthedowntimedependsonwhetherthereisastemcellupdatetoreplacetheoperatingsystemimageoriftheexistingVMcanjusthavetheRabbitMQsoftwareupdated.StemcellupdatesincuradditionaldowntimewhiletheIaaScreatesthenewVM.

TheclusterbecomesunavailableonlywhenupgradingbetweenspecificversionsofErlangorRabbitMQ.Thisisclearlystatedinthereleasenotesforthoseversions.

OpsManagerensurestheinstancesareupdatedwiththenewpackagesandanyconfigurationchangesareappliedautomatically.

ForissueswithupgradingRabbitMQforPCF,seeTroubleshootingOn-DemandRabbitMQforPCF.

BeforeUpgradingtoaNewerVersionofRabbitMQorErlangReviewthefollowingbeforestartinganupgradethatincludesanewversionofRabbitMQorErlang:

EnsuretheclusterishealthyviatheRabbitMQManagementUI.YoucannotrelyontheBOSH instances output,thatreflectsthestateofErlangVM,notRabbitMQ.

StopRabbitMQwhenupgradingtheRabbitMQorErlangVMversion.

UpgradeRabbitMQforPCFToupgradetheproduct,followthesesteps:

1. DownloadthelatestversionoftheproductfromPivotalNetwork .

2. Uploadthenew.pivotalfiletoOpsManager.

3. Uploadthestemcellassociatedwiththeupdate(ifrequired).

4. Updateanynewmandatoryconfigurationparameters(ifrequired).

5. ClickApplychangesintheOpsManagerInstallationDashboard.Therestoftheprocessisautomated.

ReleasePolicyWhenanewversionofRabbitMQisreleased,anewversionofRabbitMQforPCFisreleasedsoonafter.

WherethereisanewversionofRabbitMQoranotherdependentsoftwarecomponent,suchasthestemcellreleasedduetoacriticalCVE,Pivotal’sgoalistoreleaseanewversionoftheproductwithin48hours.

SettingLimitsforOn-DemandServiceInstancesOn-demandprovisioningisintendedtoaccelerateappdevelopmentbyeliminatingtheneedfordevelopmentteamstorequestandwaitforoperatorstocreateaserviceinstance.However,tocontrolcosts,operationsteamsandadministratorsmustensureresponsibleuseofresources.

Thereareseveralwaystocontroltheprovisioningofon-demandserviceinstancesbysettingvariousquotasattheselevels:

Global

Afteryousetquotas,youcan:

ViewCurrentOrgandSpace-levelQuotas

MonitorQuotaUseandServiceInstanceCount

CalculateResourceCostsforOn-DemandPlans

CreateGlobal-levelQuotasEachPivotalCloudFoundry(PCF)servicehasaseparateservicebroker.Aglobalquotaattheservicelevelsetsthemaximumnumberofserviceinstancesthatcanbecreatedbyagivenservicebroker.Ifaservicehasmorethanoneplan,thenthenumberofserviceinstancesforallplanscombinedcannotexceedtheglobalquotafortheservice.

TheoperatorsetsaglobalquotaforeachPCFserviceindependently.Forexample,ifyouhaveRedisforPCFandRabbitMQforPCF,youmustsetaseparateglobalservicequotaforeachofthem.

Whentheglobalquotaisreachedforaservice,nomoreinstancesofthatservicecanbecreatedunlessthequotaisincreased,orsomeinstancesofthatservicearedeleted.

TheglobalquotaissetintheservicetileinOpsManager,shownforanexampleservicebelow.

Note:Thisisanexampleimageonly.Thefollowingscreenmaylookslightlydifferentforyourserviceorreleaseversion.

CreatePlan-levelQuotasAservicemayofferoneormoreplans.Youcansetaseparatequotaperplansothatinstancesofthatplancannotexceedtheplanquota.Foraservicewithmultipleplans,thetotalnumberofinstancescreatedforallplanscombinedcannotexceedtheglobalquotafortheservice.

Whentheplanquotaisreached,nomoreinstancesofthatplancanbecreatedunlesstheplanquotaisincreasedorsomeinstancesofthatplanaredeleted.

TheplanquotaissetintheservicetileinOpsManager,shownforanexampleserviceplanbelow.

Note:Thisisanexampleimageonly.Thefollowingscreenmaylookslightlydifferentforyourserviceorreleaseversion.

CreateandSetOrg-levelQuotasAnorg-levelquotaappliestoallPCFservicesandsetsthemaximumnumberofserviceinstancesanorganizationcancreatewithinPCF.Forexample,ifyousetyourorg-levelquotato100,developerscancreateupto100serviceinstancesinthatorgusinganycombinationofPCFservices.

Whenthisquotaismet,nomoreserviceinstancesofanykindcanbecreatedintheorgunlessthequotaisincreasedorsomeserviceinstancesaredeleted.

Tocreateandsetanorg-levelquota,dothefollowing:

1. Runthiscommandtocreateaquotaforserviceinstancesattheorglevel:

cf create-quota QUOTA-NAME -m TOTAL-MEMORY -i INSTANCE-MEMORY -r ROUTES -s SERVICE-INSTANCES --allow-paid-service-plans

wherethesevariablesare:

QUOTA-NAME —AnameforthisquotaTOTAL-MEMORY —MaximummemoryusedbyallserviceinstancescombinedINSTANCE-MEMORY —MaximummemoryusedbyanysingleserviceinstanceROUTES —MaximumnumberofroutesallowedforallserviceinstancescombinedSERVICE-INSTANCES —Maximumnumberofserviceinstancesallowedfortheorg

Forexample:cf create-quota myquota -m 1024mb -i 16gb -r 30 -s 50 --allow-paid-service-plans

2. Associatethequotayoucreatedabovewithaspecificorgbyrunningthefollowingcommand:

cf set-quota ORG-NAME QUOTA-NAME

Forexample: cf set-quota dev_org myquota

Formoreinformationonmanagingorg-levelquotas,seeCreatingandModifyingQuotaPlans .

CreateandSetSpace-levelQuotasAspace-levelservicequotaappliestoallPCFservicesandsetsthemaximumnumberofserviceinstancesthatcanbecreatedwithinagivenspaceinPCF.Forexample,ifyousetyourspace-levelquotato100,developerscancreateupto100serviceinstancesinthatspaceusinganycombinationofPCFservices.

Whenthisquotaismet,nomoreserviceinstancesofanykindcanbecreatedinthespaceunlessthequotaisupdatedorsomeserviceinstancesaredeleted.

Tocreateandsetaspace-levelquota,dothefollowing:

1. Runthefollowingcommandtocreatethequota:

cf create-space-quota QUOTA -m TOTAL-MEMORY -i INSTANCE-MEMORY -r ROUTES -s SERVICE-INSTANCES --allow-paid-service-plans

wherethesevariablesare:

QUOTA-NAME —AnameforthisquotaTOTAL-MEMORY —MaximummemoryusedbyallserviceinstancescombinedINSTANCE-MEMORY —MaximummemoryusedbyanysingleserviceinstanceROUTES —MaximumnumberofroutesallowedforallserviceinstancescombinedSERVICE-INSTANCES —Maximumnumberofserviceinstancesallowedfortheorg

Forexample: cf create-space-quota myspacequota -m 1024mb -i 16gb -r 30 -s 50 --allow-paid-service-plans

2. Associatethequotayoucreatedabovewithaspecificspacebyrunningthefollowingcommand:

cf set-space-quota SPACE-NAME QUOTA-NAME

Forexample:cf set-space-quota myspace myspacequota

Formoreinformationonmanagingspace-levelquotas,seeCreatingandModifyingQuotaPlans .

ViewCurrentOrgandSpace-levelQuotasTovieworgquotas,runthefollowingcommand.

cf org ORG-NAME

Toviewspacequotas,runthefollowingcommand:

cf space SPACE-NAME

Formoreinformationonmanagingorgandspace-levelquotas,seetheCreatingandModifyingQuotaPlans .

MonitorQuotaUseandServiceInstanceCountService-levelandplan-levelquotause,andtotalnumberofserviceinstances,areavailablethroughtheon-demandbrokermetricsemittedtoLoggregator.Thesemetricsarelistedbelow:

MetricName Description

on-demand-broker/SERVICE-NAME/quota_remaining Quotaremainingforallinstancesacrossallplans

on-demand-broker/SERVICE-NAME/PLAN-NAME/quota_remaining Quotaremainingforaspecificplan

on-demand-broker/SERVICE-NAME/total_instances Totalinstancescreatedacrossallplans

on-demand-broker/SERVICE-NAME/PLAN-NAME/total_instances Totalinstancescreatedforaspecificplan

CalculateResourceCostsforOn-DemandPlansOn-demandplansusededicatedVMs,disks,andvariousotherresourcesfromanIaaS,suchasAWS.Tocalculatemaximumresourcecostforplansindividuallyorcombined,youmultiplythequotabythecostofVMandPersistentDisktypesselectedintheplanconfiguration(s).ThespecificcostsdependonyourIaaS.

TheimagebelowshowsanexampleoftheVMtypeandpersistentdiskselected,aswellasthequotaforthisplan.

CalculateMaximumResourceCostPerOn-DemandPlanTocalculatethemaximumcostofVMsandpersistentdiskforeachplan,dothefollowingcalculation:

planquotaxcostofselectedresources

Forexample,ifyouselectedtheoptionsintheaboveimage,youhaveselectedaVMtypemicro.cpuandapersistentdisktype20GB,andtheplanquotais15.TheVMandpersistentdisktypeshaveanassociatedcostfortheIaaSyouareusing.Therefore,tocalculatethemaximumcostofresourcesforthisplan,multiplythecostoftheresourcesselectedbytheplanquota:

(15xcostofmicro.cpuVMtype)+(15xcostof20GBpersistentdisk)

CalculateMaximumResourceCostforAllOn-DemandPlansTocalculatethemaximumcostforallplanscombined,addtogetherthemaximumcostsforeachplan.Thisassumesthatthesumofyourindividualplanquotasislessthantheglobalquota.

Hereisanexample:

(plan1quotaxplan1resourcecost)+(plan2quotaxplan2resourcecost)=maxcostforallplans

Note:Quotametricsarenotemittedifnoquotahasbeenset.

Important:Althoughoperatorscanlimiton-demandinstanceswithplanquotasandaglobalquota,asdescribedintheabovetopics,IaaSresourceusagestillvariesbasedonthenumberofon-demandinstancesprovisioned.

CalculateActualResourceCostofallOn-DemandPlansTocalculatethecurrentactualresourcecostacrossallyouron-demandplans:

1. Findthenumberofinstancescurrentlyprovisionedforeachactiveplanbylookingatthe total_instance metricforthatplan.

2. Multiplythe total_instance countforeachplanbythatplan’sresourcecosts.Recordthecostsforeachplan.

3. AddupthecostsnotedinStep2togetyourtotalcurrentresourcecosts.

Forexample:

(plan1totalinstancesxplan1resourcecost)+(plan2totalinstancesxplan2resourcecost)=currentcostforallplans

ControllingAccesstoServicePlansbyOrgIfyouwanttorestrictaccesstoaserviceplantoaspecificorg,followtheinstructionsbelow.

Youcanalsolimitthenumberofserviceinstancesbysettingquotas—forinstructions,seeSettingLimitsforOn-DemandInstances.

ChangeAccesstoServicePlans

Torestrictaccesstoaplanforaspecificorg,runthiscommand:

cf enable-service-access p.rabbitmq -p PLAN_NAME -oORG_NAME

Forexample:

$ cf enable-service-access p.rabbitmq -p my-cluster-plan -o my-dev-org

Formoreinformationontheabovecommand,seeAccessControl .

Note:Iftheplanyouarerestrictingiscurrentlyenabledforallorgs,youmustfirstDisableServiceAccessfortheplan,thengrantaccesstotheplantospecificorgs.UsetheCFServiceAccessfieldtodisableaccessintheserviceplanconfiguration.

MonitoringandKPIsforOn-DemandRabbitMQforPCFThistopicexplainshowtomonitorthehealthoftheon-demandversionoftheRabbitMQforPivotalCloudFoundry(PCF)serviceusingthelogs,metrics,andKeyPerformanceIndicators(KPIs)generatedbyRabbitMQforPCFcomponentVMs.

On-demandRabbitMQforPCFcomponentsgeneratemanyofthesamemetricsasthepre-provisionedRabbitMQservicecomponents.Forinformationaboutmetricsforthepre-provisionedservice,seeMonitoringandKPIsforPre-ProvisionedRabbitMQforPCF.

SeeLoggingandMetrics forgeneralinformationaboutloggingandmetricsinPCF.

SetupSyslogForwardingAsofRabbitMQforPCFv1.9.0,syslogforwardingispreconfiguredandsettoonbydefault.Pivotalrecommendsthatyoukeepthedefaultsettingbecauseitisgoodoperationalpractice.However,youcanoptoutbyselectingNoforDoyouwanttoconfiguresyslog?intheOpsManagerSettingstab.

ToenablemonitoringforRabbitMQforPCF,operatorsdesignateanexternalsyslogendpointforRabbitMQcomponentlogmessages.TheendpointservesastheinputtoamonitoringplatformsuchasDatadog,Papertrail,orSumoLogic.

TospecifythedestinationforRabbitMQforPCFlogmessages,dothefollowing:

1. FromtheOpsManagerInstallationDashboard,clicktheRabbitMQtile.

2. IntheRabbitMQtile,clicktheSettingstab.

Note:On-demandservicemetricsareprefixedwith p.rabbitmq (withadot),todistinguishthemfromthepre-provisionedservicemetrics.

3. ClickSyslog.

4. ConfigurethefieldsontheSyslogpaneasfollows:

Option Description

Syslogaddress IPorDNSaddressofthesyslogserver

Syslogport Portofthesyslogserver

Transportprotocol

Transportprotocolofthesyslogserver.Oneof udp , tcp , relp .

Formatforlogs Formatforlogs.Pivotalrecommends RFC 5424 ,but Legacy Format canbeusedforcompatibilityreasons.

EnableTLS EnableTLStothesyslogserver.

Ifthereareseveralpeerserversthatcanrespondtoremotesyslogconnections,thenyoumayprovideawildcardinthe

PermittedPeer domain,suchas *.example.com .

CustomCACertificate

Iftheservercertificateisnotsignedbyaknownauthority,forexample,aninternalsyslogserver,providetheCAcertificateofthelogmanagementserviceendpoint.

5. ClickSave.

6. ReturntotheOpsManagerInstallationDashboardandclickApplyChangestoredeploywiththechanges.

LoggingFormatWithon-demandRabbitMQforPCFloggingconfigured,twotypesofcomponentsgeneratelogs:theservernodesandtheservicebroker:

ThelogsforRabbitMQservernodesfollowtheformat [job=rabbitmq-server-partition-GUID index=0]

ThelogsfortheRabbitMQservicebrokerfollowtheformat [job=rabbitmq-broker-partition-GUID index=0]

TheRabbitMQVMslogatthe info levelandcaptureerrors,warnings,andinformationalmessages.

Forusersfamiliarwithdocumentationforpreviousversionsofthetile,thetagweusedtocallthe app_name isnowcalledthe program_name .

Thegenericlogformatisasfollows:

<PRI>TIMESTAMP IP_ADDRESS PROGRAM_NAME [job=NAME index=JOB_INDEX id=JOB_ID] MESSAGE

Therawlogslooksimilartothefollowing:

<7>2017-06-28T16:06:10.733560+00:00 10.244.16.133 vcap.agent [job=rmq index=0 id=e37ecdca-5b10-4141-abd8-e1d777dfd8b5] 2017/06/28 16:06:10 CEF:0|CloudFoundry|BOSH|1|agent_api|ssh|1|duser=director.be5a66bb-a9b4-459f-a0d3-1fc5c9c3ed79.be148cc6-91ef-4eed-a788-237b0b8c63b7 src=10.254.50.4 spt=4222 shost=5ae233e0-ecc5-4868-9ae0-f9767571251b<86>2017-06-28T16:06:16.704572+00:00 10.244.16.133 useradd [job=rmq index=0 id=e37ecdca-5b10-4141-abd8-e1d777dfd8b5] new group: name=bosh_ly0d2rbjr, GID=1003<86>2017-06-28T16:06:16.704663+00:00 10.244.16.133 useradd [job=rmq index=0 id=e37ecdca-5b10-4141-abd8-e1d777dfd8b5] new user: name=bosh_ly0d2rbjr, UID=1001, GID=1003, home=/var/vcap/bosh_ssh/bosh_ly0d2rbjr, shell=/bin/bash<86>2017-06-28T16:06:16.736932+00:00 10.244.16.133 usermod [job=rmq index=0 id=e37ecdca-5b10-4141-abd8-e1d777dfd8b5] add 'bosh_ly0d2rbjr' to group 'admin'<86>2017-06-28T16:06:16.736964+00:00 10.244.16.133 usermod [job=rmq index=0 id=e37ecdca-5b10-4141-abd8-e1d777dfd8b5] add 'bosh_ly0d2rbjr' to group 'vcap'

LogssenttoexternalloggingtoolssuchasPapertrailmaybepresentedinadifferentformat.

Thefollowingtabledescribestheloggingtagsusedinthistemplate:

Tag Description

PRI Thisisavaluewhichinfuturewillbeusedtodescribetheseverityofthelogmessageandwhichfacilityitcamefrom.

TIMESTAMPThisisthetimestampofwhenthelogisforwarded,forexample, 2016-08-24T05:14:15.000003Z .Thetimestampvalueistypicallyslightlyafterwhenthelogmessagewasgenerated.

IP_ADDRESS TheinternalIPaddressofserveronwhichthelogmessageoriginated

PROGRAM_NAMEProcessnameoftheprogramthegeneratedthemessage.Sameas app_name beforev1.9.0.Formoreinformationaboutprogramname,seeRabbitMQProgramNamesbelow.

NAME TheBOSHinstancegroupname(forexample, rabbitmq_server )

JOB_INDEX BOSHjobindex.Usedtodistinguishbetweenmultipleinstancesofthesamejob.

JOB_IDBOSHVMGUID.ThisisdistinctfromtheCIDdisplayedintheOpsManagerStatustab,whichcorrespondstotheVMIDassignedbytheinfrastructureprovider.

MESSAGE Thelogmessagethatappears

RabbitMQProgramNames

ProgramName Description

rabbitmq_server_cluster_check ChecksthattheRabbitMQclusterishealthy.Runsaftereverydeploy.

rabbitmq_server_node_check ChecksthattheRabbitMQnodeishealthy.Runsaftereverydeploy.

rabbitmq_route_registrar_stderr RegisterstherouteforthemanagementAPIwiththeGorouterinyourElasticRuntimedeployment.

rabbitmq_route_registrar_stdout RegisterstherouteforthemanagementAPIwiththeGorouterinyourElasticRuntimedeployment.

rabbitmq_server TheErlangVMandRabbitMQapps.Logsmayspanmultiplelines.

rabbitmq_server_drain ShutsdowntheErlangVMandRabbitMQapps.RunsaspartoftheBOSHlifecycle.

rabbitmq_server_http_api_access AccesstotheRabbitMQmanagementUI.

rabbitmq_server_init StartstheErlangVMandRabbitMQ.

rabbitmq_server_post_deploy_stderr Runsthenodecheckandclustercheck.Runsaftereverydeploy.

rabbitmq_server_post_deploy_stdout Runsthenodecheckandclustercheck.Runsaftereverydeploy.

rabbitmq_server_pre_start Runsbeforetherabbitmq-serverjobisstarted.

rabbitmq_server_sasl Supervisor,progress,andcrashreportingfortheErlangVMandRabbitMQapps.

rabbitmq_server_shutdown_stderr StopstheRabbitMQappandErlangVM.

rabbitmq_server_shutdown_stdout StopstheRabbitMQappandErlangVM.

rabbitmq_server_startup_stderr StartstheRabbitMQappandErlangVM,thenconfiguresusersandpermissions.

rabbitmq_server_startup_stdout StartstheRabbitMQappandErlangVM,thenconfiguresusersandpermissions.

rabbitmq_server_upgrade ShutsdownErlangVMandRabbitMQappifrequiredduringanupgrade.

WhatAreMetricsMetricsareregularly-generatedlogmessagesthatreportmeasuredcomponentstates.Themetricspollingintervaldefaultsto30seconds.ThemetricspollingintervalisaconfigurationoptionontheRabbitMQtile(Settings>RabbitMQ).Theintervalsettingappliestoallcomponentsdeployedbythetile.

Metricsarelong,singlelinesoftextthatfollowtheformat:

origin:"p.rabbitmq" eventType:ValueMetric timestamp:1441188462382091652 deployment:"cf-rabbitmq" job:"cf-rabbitmq-node" index:"0" ip:"10.244.3.46" valueMetric: < name:"/p.rabbitmq/rabbitmq/system/memory" value:1024 unit:"MB">

ThefollowingsectionsdescribethemetricsusedasKeyPerformanceIndicatorsandotherusefulmetricsformonitoringtheRabbitMQforPCFon-demandservice.

KeyPerformanceIndicatorsKeyPerformanceIndicators(KPIs)forRabbitMQforPCFaremetricsthatoperatorsfindmostusefulformonitoringtheirRabbitMQservicetoensuresmoothoperation.KPIsarehigh-signal-valuemetricsthatcanindicateemergingissues.KPIscanberawcomponentmetricsorderivedmetricsgeneratedbyapplyingformulastorawmetrics.

PivotalprovidesthefollowingKPIsasgeneralalertingandresponseguidancefortypicalRabbitMQforPCFinstallations.Pivotalrecommendsthatoperatorscontinuetofine-tunethealertmeasurestotheirinstallationbyobservinghistoricaltrends.Pivotalalsorecommendsthatoperatorsexpandbeyondthisguidanceandcreatenew,installation-specificmonitoringmetrics,thresholds,andalertsbasedonlearningfromtheirowninstallations.

ForalistofallRabbitMQforPCFrawcomponentmetrics,seeComponentMetricsReference.

ComponentHeartbeatsKeyRabbitMQforPCFcomponentsperiodicallyemitheartbeatmetrics:theRabbitMQservernodes,HAProxynodes,andtheServiceBroker.TheheartbeatsareBooleanmetrics,where 1 meansthesystemisavailable,and 0 ortheabsenceofaheartbeatmetricmeanstheserviceisnotrespondingandshouldbeinvestigated.

ServiceBrokerHeartbeat

p.rabbitmq/service_broker/heartbeat

Description

RabbitMQServiceBroker is alive poll,whichindicatesifthecomponentisavailableandabletorespondtorequests.

Use:IftheServiceBrokerdoesnotemitheartbeats,thisindicatesthatitisoffline.TheServiceBrokerisrequiredtocreate,update,anddeleteserviceinstances,whicharecriticalfordependenttilessuchasSpringCloudServicesandSpringCloudDataFlow.

Origin:Doppler/FirehoseType:boolean

Frequency:30s(default),10s(configurableminimum)

Recommendedmeasurement Averageoverlast5minutes

RecommendedalertthresholdsYellowwarning:N/ARedcritical:<1

Recommendedresponse

ChecktheRabbitMQServiceBrokerlogsforerrors.YoucanfindthisVMbytargetingyourRabbitMQdeploymentwithBOSHandrunningoneofthefollowingcommandsdependingonyourOpsManagerVersion:

ForOpsManagerv1.10orearlier:bosh vms service-instance_GUID

ForOpsManagerv1.11orlater:bosh2 -d service-instance_GUID vms

ServerHeartbeat

p.rabbitmq/rabbitmq/heartbeat

Description

RabbitMQServer is alive poll,whichindicatesifthecomponentisavailableandabletorespondtorequests.

Use:Iftheserverdoesnotemitheartbeats,thisindicatesthatitisoffline.Tobefunctional,serviceinstancesrequireRabbitMQServer.

Origin:Doppler/FirehoseType:booleanFrequency:30s(default),10s(configurableminimum)

Recommendedresponse

ChecktheRabbitMQServerlogsforerrors.YoucanfindtheVMbytargetingyourRabbitMQdeploymentwithBOSHandrunningoneofthefollowingcommands,whichlists rabbitmq :

RabbitMQServerKPIsThefollowingKPIsfromtheRabbitMQservercomponent:

FileDescriptors

p.rabbitmq/rabbitmq/system/file_descriptors

Description

Filedescriptorsconsumed.

Use:Ifthenumberoffiledescriptorsconsumedbecomestoolarge,theVMmightlosetheabilitytoperformdiskIO,whichcancausedataloss.

Origin:Doppler/FirehoseType:countFrequency:30s(default),10s(configurableminimum)

Note:Thisassumesnon-persistentmessagesarehandledbyretriesorsomeotherlogicbytheproducers.

RecommendedalertthresholdsYellowwarning:>50000Redcritical:>55000

Recommendedresponse

Thedefault ulimit forRabbitMQforPCFv1.6andlateris60000.Ifthismetricismetorexceededforanextendedperiodoftime,consideroneofthefollowingactions:

ScalingtherabbitnodesinthetileResourceConfigpane.

Increasingthe ulimit

ErlangProcesses

p.rabbitmq/rabbitmq/system/erlang_processes

Description

Erlang processesconsumedbyRabbitMQ,whichrunsonanErlangVM.

Use:Thisisthekeyindicatoroftheprocessingcapabilityofanode.

RecommendedresponseThedefaultErlangprocesslimitinRabbitMQforPCFv1.6andlateris1,048,816.Ifthismetricmeetsorexceedstherecommendedthresholdsforextendedperiodsoftime,considerscalingtheRabbitMQnodesinthetileResourceConfigpane.

BOSHSystemHealthMetricsTheBOSHlayerthatunderliesPCFgenerates healthmonitor metricsforallVMsinthedeployment.However,thesemetricsarenotincludedintheLoggregatorFirehosebydefault.Togetthesemetrics,doeitherofthefollowing:

TosendBOSHHMmetricsthroughtheFirehose,installtheopen-sourceHMForwarder .

ToretrieveBOSHhealthmetricsoutsideoftheFirehose,installtheJMXBridge forPCFtile.

AllBOSH-deployedcomponentsgeneratethefollowingsystemhealthmetrics.ComingfromRabbitMQforPCFcomponents,thesesystemmetricsserveasKPIsfortheRabbitMQforPCFservice.

system.mem.percent

Description

RAMbeingconsumedbythe p.rabbitmq VM.

Use:RabbitMQisconsideredtobeinagoodstatewhenithasfewornomessages.Inotherwords,“anemptyrabbitisahappyrabbit.”Alertingonthismetriccanindicatethattherearetoofewconsumersorappsthatreadmessagesfromthequeue.

HealthmonitorreportswhenRabbitMQusesmorethan40%ofitsRAMforthepasttenminutes.

Origin:JMXBridgeorBOSHHMType:percentFrequency:30s(default),10s(configurableminimum)

Recommendedresponse Addmoreconsumerstodrainthequeueasfastaspossible.

system.cpu.percent

Description

CPUbeingconsumedbythe p.rabbitmq VM.

Use:AnodethatexperiencescontextswitchingorhighCPUusagebecomesunresponsive.Thisalsoaffectstheabilityofthenodetoreportmetrics.

HealthmonitorreportswhenRabbitMQusesmorethan40%ofitsCPUforthepasttenminutes.

Recommendedresponse Rememberthat“anemptyrabbitisahappyrabbit”.Addmoreconsumerstodrainthequeueasfastaspossible.

EphemeralDisk

system.disk.percent

Description

EphemeralDiskbeingconsumedbythe p.rabbitmq VM.

Use:Ifsystemdiskfillsup,therearetoofewconsumers.

PersistentDisk

persistent.disk.percent

Description

PersistentDiskbeingconsumedbythe p.rabbitmq VM.

DetermineIfThereIsaNetworkPartitionYoucanusethe reachable_nodes metrictohelptoidentifynetworkpartitions.Thismetricshowshowmanynodesintheclustereachindividualnodeisawareof.Agoodindicationthatanodemightbeinapartitioniswhenitisawareofonlyitself.

Hereisanexampleofthismetrics:

origin:"p.rabbitmq" eventType:ValueMetric timestamp:1441188462382091652 deployment:"cf-rabbitmq" job:"cf-rabbitmq-node" index:"0" ip:"10.244.3.46" valueMetric: < name:"/p.rabbitmq/rabbitmq/erlang/reachable_nodes" value:3 unit:"count">

Youcancreatemonitorstoemitalertsincaseaclusterseemstobeinapartition.Inahealthyclusterthatisnotundergoingupgrades,eachnode’sreachable_nodes countisequaltothenumberofnodesinthecluster.

Tomonitorfornetworkpartition,Pivotalrecommendsalertingwhenoneofthenodesstartsreportinga reachable_nodes countthatislessthanthesizeofthecluster.

Duringrollingupgrades,nodeslosecontactwithothernodes.Therefore,onlyalertifalowered reachable_nodes countpersistslongerthantheexpectedupgradetime.

RecoverfromaNetworkPartitionForinformationabouthowtorecoverfromanetworkpartition,seetheRabbitMQdocumentation .

ComponentMetricReferenceRabbitMQforPCFcomponentVMsemitthefollowingrawmetrics.Thefullnameofthemetricfollowstheformat: /p.rabbitmq/COMPONENT/METRIC-NAME

RabbitMQServerMetricsRabbitMQforPCFmessageservercomponentsemitthefollowingmetrics.

FullName Unit Description

/p.rabbitmq/rabbitmq/heartbeat boolean IndicateswhethertheRabbitMQserverisavailableandabletorespondtorequests

/p.rabbitmq/rabbitmq/erlang/erlang_processes

count ThenumberofErlangprocesses

/p.rabbitmq/rabbitmq/erlang/reachable_nodes count Thenumberofnodesthecurrentnodecanreach

/p.rabbitmq/rabbitmq/system/memory MB ThememoryinMBusedbythenode

/p.rabbitmq/rabbitmq/system/mem_alarm boolean Indicatesifthememoryalarmhasgoneoff

/p.rabbitmq/rabbitmq/system/disk_free_alarm boolean Indicatesifthediskfreealarmhasgoneoff

/p.rabbitmq/rabbitmq/connections/count count Thetotalnumberofconnectionstothenode

/p.rabbitmq/rabbitmq/consumers/count count Thetotalnumberofconsumersregisteredinthenode

/p.rabbitmq/rabbitmq/messages/delivered count Thetotalnumberofmessageswiththestatus deliver_get onthenode

/p.rabbitmq/rabbitmq/messages/delivered_no_ack

count Thenumberofmessageswiththestatus deliver_no_ack onthenode

/p.rabbitmq/rabbitmq/messages/delivered_rate

rate Therateatwhichmessagesarebeingdeliveredtoconsumersorclientsonthenode

/p.rabbitmq/rabbitmq/messages/published count Thetotalnumberofmessageswiththestatus publish onthenode

/p.rabbitmq/rabbitmq/messages/published_rate

rate Therateatwhichmessagesarebeingpublishedbythenode

/p.rabbitmq/rabbitmq/messages/redelivered count Thetotalnumberofmessageswiththestatus redeliver onthenode

/p.rabbitmq/rabbitmq/messages/redelivered_rate

rate Therateatwhichmessagesaregettingthestatus redeliver onthenode

/p.rabbitmq/rabbitmq/messages/get_no_ack count Thenumberofmessageswiththestatus get_no_ack onthenode

/p.rabbitmq/rabbitmq/messages/get_no_ack_rate

rate Therateatwhichmessagesgetthestatus get_no_ack onthenode

/p.rabbitmq/rabbitmq/messages/pending count Thenumberofmessageswiththestatus messages_unacknowledged onthenode

/p.rabbitmq/rabbitmq/messages/depth countThenumberofmessageswiththestatus messages_unacknowledged ormessages_ready onthenode

/p.rabbitmq/rabbitmq/system/filedescriptors

count Thenumberofopenfiledescriptorsonthenode

/p.rabbitmq/rabbitmq/exchanges/count count Thetotalnumberofexchangesonthenode

/p.rabbitmq/rabbitmq/messages/available count Thetotalnumberofmessageswiththestatus messages_ready onthenode

/p.rabbitmq/rabbitmq/queues/count count Thenumberofqueuesonthenode

/p.rabbitmq/rabbitmq/channels/count count Thenumberofchannelsonthenode

/p.rabbitmq/rabbitmq/queues/VHOST-NAME/QUEUE-NAME/consumers

count Thenumberofconsumerspervirtualhostperqueue

/p.rabbitmq/rabbitmq/queues/VHOST-NAME/QUEUE-NAME/depth

countThenumberofmessageswiththestatus messages_unacknowledged ormessages_ready pervirtualhostperqueue

ClusteringandNetworkPartitions

ClusteringinRabbitMQforPCFInRabbitMQforPCF,theRabbitMQ®brokerisalwaysdeployedasaclusterofoneormorevirtualmachines(nodes).ARabbitMQbrokerisalogicalgroupingofoneorseveralErlangnodes,eachrunningtheRabbitMQapplicationandsharingusers,virtualhosts,queues,exchanges,bindings,andruntimeparameters.

WhatisReplicatedbetweennodesinaRabbitMQcluster?

Alldata/staterequiredfortheoperationofaRabbitMQbrokerisreplicatedacrossallnodes.Anexceptiontothisaremessagequeues,whichbydefaultresideononenode,thoughtheyarevisibleandreachablefromallnodes.ThismeansthattheRabbitMQclustermaybeavailableandservingrequests,whileanindividualqueueresidingonasinglenodeisoffline.

Replicatingmessagequeuesacrossnodesisanexpensiveoperationandshouldonlybedonetotheextentneededbytheapplication.Tounderstandmoreaboutreplicatingqueuesacrossnodesinacluster,seethedocumentation onhighavailability.

AutomaticNetworkPartitionBehaviorsinRabbitMQClustersTheRabbitMQ®tileusesthe pause_minority optionforhandlingclusterpartitionsbydefault.Thisensuresdataintegritybypausingthepartitionoftheclusterintheminority,andresumesitwiththedatafromthemajoritypartition.Youmustmaintainmorethantwonodes.Ifthereisapartitionwhenyouonlyhavetwonodes,bothnodesimmediatelypause.

Youcanalsochoosethe autoheal optionintheRabbitMQPolicytab.Inthismode,ifapartitionoccurs,RabbitMQautomaticallydecidesonawinningpartition,andrestartsallnodesthatarenotinthewinningpartition.Thisoptionallowsyoutocontinuetoreceiveconnectionstobothpartsofpartitions.

DetectingaNetworkPartitionWhenanetworkpartitionoccurs,alogmessageiswrittentotheRabbitMQnodelog:

=ERROR REPORT==== 15-Oct-2012::18:02:30 ===Mnesia(rabbit@da3be74c053640fe92c6a39e2d7a5e46): ** ERROR ** mnesia_event got {inconsistent_database, running_partitioned_network, rabbit@21b6557b73f343201277dbf290ae8b79}

Youcanalsorunthe rabbitmqctl cluster_status commandonanyoftheRabbitMQnodestoseethenetworkpartition.Torun rabbitmqctl cluster_status ,dothefollowing:

1. $ sudo su -

2. $ cd /var/vcap/packages

3. $ export ERL_DIR=$PWD/erlang/bin/

4. $ cd rabbitmq-server/bin/

5. $ ./rabbitmqctl cluster_status

[... {partitions, [{rabbit@da3be74c053640fe92c6a39e2d7a5e46, [rabbit@21b6557b73f343201277dbf290ae8b79]}]}]

RecoveringBecausetheRabbitMQtileusesthe pause_minority option,minoritynodesrecoverautomaticallyafterthepartitionisresolved.Afteranoderecovers,itresumesaccessingthequeuealongwithdatafromthequeuesontheothernodes.However,ifyourqueuesuse ha-mode: all ,theyonlysynchronizefullyafterconsumingallthemessagescreatedwhilethenodewasdown.Thisissimilartohowmessagessynchronizewhenyoucreateanewqueue.

ManuallySynchronizingafteraPartitionAfteranetworkpartition,aqueueonaminoritynodesynchronizesafterconsumingallthemessagescreatedwhileitwasdown.Youcanalsorunthe

sync_queue commandtosynchronizeaqueuemanually.Torun sync_queue ,dothefollowingoneachnode:

1. $ sudo su -

2. $ cd /var/vcap/packages

3. $ export ERL_DIR=$PWD/erlang/bin/

4. $ cd rabbitmq-server/bin/

5. $ ./rabbitmqctl list_queues

6. $ ./rabbitmqctl sync_queue name

TroubleshootingOn-DemandRabbitMQforPCFInthistopic:

AbouttheBOSHCLI

TroubleshootingErrors

FailedInstallCannotCreateorDeleteServiceInstancesBrokerRequestTimeoutsCannotBindtoorUnbindfromServiceInstancesCannotConnecttoaServiceInstanceUpgradeAllInstancesFailsMissingLogsandMetricsFailedDeploymentonUpgradeorafterApplyChanges

TroubleshootingComponents

BOSHproblemsConfigurationAuthenticationNetworkingQuotas

TechniquesforTroubleshooting

ParseaCloudFoundry(CF)ErrorMessageAccessBrokerandInstanceLogsandVMsRunServiceBrokerErrandstoManageBrokersandInstancesSelecttheBOSHDeploymentforaServiceInstanceGetAdminCredentialsforaServiceInstanceReinstallaTileViewResourceSaturationandScalingIdentifyServiceInstanceOwnerMonitorQuotaSaturationandServiceInstanceCount

FrequentlyAskedQuestions

WhatshouldIcheckbeforedeployinganewversionofthetile?WhatisthecorrectwaytostopandstartRabbitMQinPCF?WhathappenswhenIstoptheRabbitMQserver?WhathappenswhentheRabbitMQserverfailstostop?WhatdoIdowhentheRabbitMQserverfailstostop?HowcanImanuallybackupthestateoftheRabbitMQcluster?Whatpre-upgradechecksshouldIdo?

KnowledgeBase(Community)

FileaSupportTicket

Thistopicprovidesoperatorswithbasicinstructionsfortroubleshootingon-demandRabbitMQ®forPivotalCloudFoundry(PCF).

AbouttheBOSHCLITheBOSHCLIisavailableintwomajorversions,v1andv2.PivotalrecommendsthatyouusetheBOSHCLIv2whenpossible.

ThistopicprovidesexamplesofusingeachversionoftheBOSHCLI.ConsultthetablebelowtodeterminewhichversionoftheCLIissupportedforyourinstallation.

OpsManagerVersion BOSHCLIVersion

1.10 CLIv1

1.11 CLIv1orCLIv2(PivotalrecommendsCLIv2)

1.12andlater CLIv2

TroubleshootingErrorsStarthereifyou’rerespondingtoaspecificerrororerrormessages.

FailedInstall

1. Certificateissues:Theon-demandbroker(ODB)requiresvalidcertificates.Ensurethatyourcertificatesarevalidandgeneratenewones ifnecessary.

2. Deployfails:Deployscanfailforavarietyofreasons.ViewthelogsusingOpsManagertodeterminewhythedeployisfailing.

3. Networkingproblems:

CloudFoundrycannotreachtheRabbitMQforPCFservicebrokerCloudFoundrycannotreachtheserviceinstancesTheservicenetworkcannotaccesstheBOSHdirector

4. Registerbrokererrandfails.

5. Thesmoketesterrandfails.

6. Resourcesizingissues:TheseoccurwhentheresourcesizesselectedforagivenplanarelessthantheRabbitMQforPCFservicerequirestofunction.CheckyourresourceconfigurationinOpsManagerandensurethattheconfigurationmatchesthatrecommendedbytheservice.

7. Otherservice-specificissues.

CannotCreateorDeleteServiceInstances

Ifdevelopersreporterrorssuchasthefollowing:

Instance provisioning failed: There was a problem completing your request. Please contact your operations team providing the following information: service: redis-acceptance, service-instance-guid: ae9e232c-0bd5-4684-af27-1b08b0c70089, broker-request-id: 63da3a35-24aa-4183-aec6-db8294506bac, task-id: 442, operation: create

1. IftheBOSHerrorshowsaproblemwiththedeploymentmanifest:

a. Downloadthemanifestfortheon-demandserviceinstancebyrunning:bosh download manifest service-instance_SERVICE-INSTANCE-GUID MY-SERVICE.yml.

b. Checkthemanifestforconfigurationerrors.

2. Tocontinuetroubleshooting,LogintoBOSH andtargettheRabbitMQforPCFserviceinstanceusingtheinstructionsonparsingaCloudFoundryerrormessage.

3. RetrievetheBOSHtaskIDfromtheerrormessageandrunoneofthefollowingcommandsdependingonyourOpsManagerversion:

OpsManagerVersion BOSHCommand

1.10andearlier bosh task TASK-ID

1.11 bosh2 task TASK-ID

1.12andlater bosh task TASK-ID

4. Ifyouneedmoreinformation,accessthebrokerlogsandusethe broker-request-id fromtheerrormessageabovetosearchthelogsformoreinformation.Checkfor:

Note:ThiserrordoesnotapplyifyouareusingBOSHCLIv2.Inthatcase,totroubleshootpossibleproblemswiththemanifest,openitinatexteditorandinspectthemanifestthere.

AuthenticationerrorsNetworkerrorsQuotaerrors

BrokerRequestTimeoutsIfdevelopersreporterrorssuchas:

Server error, status code: 504, error code: 10001, message: The request to the service broker timed out: https://BROKER-URL/v2/service_instances/e34046d3-2379-40d0-a318-d54fc7a5b13f/service_bindings/aa635a3b-ef6d-41c3-a23f-55752f3f651b

1. ConfirmthatCloudFoundry(CF)isconnectedtotheservicebroker.

2. ChecktheBOSHqueuesize:

a. LogintoBOSHasanadmin.b. RunoneofthesecommandsdependingonyourOpsManagerversion:

1.10andearlier: bosh tasks1.11: bosh2 tasks1.12andlater: bosh tasks

3. Iftherearealargenumberofqueuedtasks,thesystemmaybeundertoomuchload.BOSHisconfiguredwithtwoworkersandonestatusworker,whichmaynotbesufficientresourcesforthelevelofload.Adviseappdeveloperstotryagainoncethesystemisunderlessload.

CannotBindtoorUnbindfromServiceInstances

InstanceDoesNotExist

Ifdevelopersreporterrorssuchas:

Server error, status code: 502, error code: 10001, message: Service broker error: instance does not exist`

Followthesesteps:

1. Type cf service MY-INSTANCE --guid .ThisconfirmsthatthetheRabbitMQforPCFserviceinstanceexistsinBOSHandCF,andreturnsaGUID.

2. UsingtheGUIDobtainedabove,runoneofthefollowingBOSHCLIcommandsdependingonyourOpsManagerversion:

1.10andearlier bosh vms service-instance_GUID

1.11 bosh2 -d service-instance_GUID vms

1.12andlater bosh -d service-instance_GUID vms

IftheBOSHdeploymentisnotfound,ithasbeendeletedfromBOSH.ContactPivotalsupportforfurtherassistance.

OtherErrors

Server error, status code: 502, error code: 10001, message: Service broker error: There was a problem completing your request. Please contact your operations team providing the following information: service: example-service, service-instance-guid: 8d69de6c-88c6-4283-b8bc-1c46103714e2, broker-request-id: 15f4f87e-200a-4b1a-b76c-1c4b6597c2e1, operation: bind

Tofindouttheexactissuewiththebindingprocess:

1. Accesstheservicebrokerlogs.

2. Searchthelogsforthe broker-request-id stringlistedintheerrormessageabove.

3. ContactPivotalsupportforfurtherassistanceifyouareunabletoresolvetheproblem.

4. Checkfor:

AuthenticationerrorsNetworkerrors

CannotConnecttoaServiceInstanceIfdevelopersreportthattheirappcannotuseserviceinstancesthattheyhavesuccessfullycreatedandbound:

Asktheusertosendapplicationlogsthatshowtheconnectionerror.Iftheerrorisoriginatingfromtheservice,thenfollowRabbitMQforPCF-specificinstructions.Iftheissueappearstobenetwork-related,then:

1. Checkthatapplicationsecuritygroups areconfiguredcorrectly.Accessshouldbeconfiguredfortheservicenetworkthatthetileisdeployedto.

2. EnsurethatthenetworkthePCFElasticRuntimetileisdeployedtohasnetworkaccesstotheservicenetwork.YoucanfindthenetworkdefinitionforthisservicenetworkintheOpsManagerDirectortile.

3. InOpsManagergointotheservicetileandseetheservicenetworkthatisconfiguredinthenetworkstab.

4. InOpsManagergointotheERTtileandseethenetworkitisassignedto.Makesurethatthesenetworkscanaccesseachother.

UpgradeAllInstancesFailsIfthe upgrade-all-service-instances errandfails,lookattheerrandoutputintheOpsManagerlog.

Ifaninstancefailstoupgrade,debugandfixitbeforerunningtheerrandagaintopreventanyfailureissuesfromspreadingtootheron-demandinstances.

OncetheOpsManagerlognolongerliststhedeploymentas failing ,re-runtheerrandtoupgradetherestoftheinstances.

MissingLogsandMetricsIfnologsarebeingemittedbytheon-demandbroker,checkthatyoursyslogforwardingaddressiscorrectinOpsManager.

1. Ensureyouhaveconfiguredsyslogforthetile.

2. Ensurethatyouhavenetworkconnectivitybetweenthenetworksthatthetileisusingandthesyslogdestination.Ifthedestinationisexternal,youneedtousethepublicip VMextensionfeatureavailableinyourOpsManagertileconfigurationsettings.

3. VerifythattheFirehoseisemittingmetrics:

a. Installthe cf nozzle plugin

b. Run cf nozzle -f ValueMetric | grep --line-buffered "on-demand-broker/MY-SERVICE" tofindlogsfromyourserviceinthecf nozzle output.

Ifnometricsappearwithinfiveminutes,verifythatthebrokernetworkhasaccesstotheLoggregatorsystemonallrequiredports.

ContactPivotalsupportifyouareunabletoresolvetheissue.

FailedDeploymentonUpgradeorafterApplyChangesIfthedeploymentfailsaftereditingtheAssignAZsandNetworkspaneoftheRabbitMQforPCFtile,itmightbeduetoachangetotheIPaddressesassignedtothe RabbitMQ Server job.RabbitMQforPCFrequiresthattheseIPaddressesdonotchangeonceassigned.Ifyouchangethem,thedeploymentfails.Thisincludeschangesmadetoyourcurrentinstallationorduringanupgrade.Todiagnoseandsolvethisissue,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .

TroubleshootingComponentsGuidanceoncheckingforandfixingissuesinon-demandservicecomponents.

BOSHproblems

MissingBOSHDirectorUUID

IfusingtheBOSHCLIv1,re-addthe director_uuid tothemanifest:

1. Run bosh status --uuid andrecordthe director_uuid valuefromtheoutput.

2. Editthemanifestandaddthe director_uuid: DIRECTOR-UUID fromthelaststepatthetopofthemanifest.

Formore,seeDeploymentIdentification intheBOSHdocs.

LargeBOSHQueue

On-demandservicebrokersaddtaskstotheBOSHrequestqueue,whichcanbackupandcausedelayunderheavyloads.AnappdeveloperwhorequestsanewRabbitMQforPCFinstancesees create in progress intheCloudFoundryCommandLineInterface(cfCLI)untilBOSHprocessesthequeuedrequest.

OpsManagercurrentlydeploystwoBOSHworkerstoprocessitsqueue.FutureversionsofOpsManagerwillletusersconfigurethenumberofBOSHworkers.

Configuration

Serviceinstancesinfailingstate

YoumayhaveconfiguredaVM/DisktypeintileplanpageinOpsManagerthatisinsufficientlylargefortheRabbitMQforPCFserviceinstancetostart.Seetile-specificguidanceonresourcerequirements.

Note:ThiserrordoesnotoccurifyouareusingBOSHCLIv2

Authentication

UAAChanges

IfyouhaverotatedanyUAAusercredentialsthenyoumayseeauthenticationissuesintheservicebrokerlogs.

Toresolvethis,redeploytheRabbitMQforPCFtileinOpsManager.Thisprovidesthebrokerwiththelatestconfiguration.

NetworkingCommonissuesinclude:

1. NetworklatencywhenconnectingtotheRabbitMQforPCFserviceinstancetocreateordeleteabinding.

Solution:Tryagainorimprovenetworkperformance

2. NetworkfirewallrulesareblockingconnectionsfromtheRabbitMQforPCFservicebrokertotheserviceinstance.

Solution:OpentheRabbitMQforPCFtileinOpsManagerandcheckthetwonetworksconfiguredintheNetworkspane.Ensurethatthesenetworksallowaccesstoeachother.

3. NetworkfirewallrulesareblockingconnectionsfromtheservicenetworktotheBOSHdirectornetwork.

Solution:EnsurethatserviceinstancescanaccesstheDirectorsothattheBOSHagentscanreportin.

4. Appscannotaccesstheservicenetwork.

Solution:ConfigureCloudFoundryapplicationsecuritygroupstoallowruntimeaccesstotheservicenetwork.

5. ProblemsaccessingBOSH’sUAAortheBOSHdirector.

Solution:FollownetworktroubleshootingandcheckthattheBOSHdirectorisonline.

ValidateServiceBrokerConnectivitytoServiceInstances1. Tovalidateyoucan bosh2 ssh ontotheRabbitMQforPCFservicebroker:

WithBOSHCLIv2:Targetthedeployment,andreachtheserviceinstance.WithBOSHCLIv1:Downloadthebrokermanifestandtargetthedeployment,thentrytoreachtheserviceinstance.

2. IfnoBOSH task-id appearsintheerrormessage,lookinthebrokerlogusingthe broker-request-id fromthetask.

ValidateAppAccesstoServiceInstance

Use cf ssh toaccesstotheappcontainer,thentryconnectingtotheRabbitMQforPCFserviceinstanceusingthebindingincludedintheVCAP_SERVICES environmentvariable.

Quotas

Note:YoumustensurethatanychangestoUAAcredentialsarereflectedintheOpsManager credentials taboftheElasticRuntimetile.

PlanQuotaissues

Message: Service broker error: The quota for this service plan has been exceeded. Please contact your Operator for help.

1. Checkyourcurrentplanquota.

2. Increasetheplanquota.

3. LogintoOpsManager.

4. Reconfigurethequotaontheplanpage.

5. Deploythetile.

6. Findwhoisusingtheplanquotaandtaketheappropriateaction.

GlobalQuotaIssues

Message: Service broker error: The quota for this service has been exceeded. Please contact your Operator for help.

1. Checkyourcurrentglobalquota.

2. Increasetheglobalquota.

3. LogintoOpsManager.

4. Reconfigurethequotaontheon-demandsettingspage.

5. Deploythetile.

6. Findoutwhoisusingthequotaandtaketheappropriateaction.

Failingjobsandunhealthyinstances

TodeterminewhetherthereisanissuewiththeRabbitMQforPCFservicedeployment,inspecttheVMs.Todoso,runoneofthefollowingcommands:

1.10orearlier bosh vms --vitals service-instance_GUID

1.11 bosh2 -d service-instance_GUID vms --vitals

1.12andlater bosh -d service-instance_GUID vms --vitals

Foradditionalinformation,runoneofthefollowingcommands:

1.10andearlier bosh instances --ps --vitals

1.11 bosh2 instances --ps --vitals

1.12andlater bosh instances --ps --vitals

IftheVMisfailing,followtheservice-specificinformation.Anyunadvisedcorrectiveactions(suchasrunningBOSH restart onaVM)cancauseissuesintheserviceinstance.

TechniquesforTroubleshootingThissectioncontainsinstructionsoninteractingwiththeon-demandservicebrokerandon-demandserviceinstanceBOSHdeployments,andonperforminggeneralmaintenanceandhousekeepingtasks.

ParseaCloudFoundry(CF)ErrorMessageFailedoperations(create,update,bind,unbind,delete)resultinanerrormessage.YoucanretrievetheerrormessagelaterbyrunningthecfCLIcommand cf service INSTANCE-NAME .

$ cf service myservice

Service instance: myserviceService: super-dbBound apps:Tags:Plan: dedicated-vmDescription: Dedicated InstanceDocumentation url:Dashboard:

Last OperationStatus: create failedMessage: Instance provisioning failed: There was a problem completing your request. Please contact your operations team providing the following information: service: redis-acceptance, service-instance-guid: ae9e232c-0bd5-4684-af27-1b08b0c70089, broker-request-id: 63da3a35-24aa-4183-aec6-db8294506bac, task-id: 442, operation: createStarted: 2017-03-13T10:16:55ZUpdated: 2017-03-13T10:17:58Z

Usetheinformationinthe Message fieldtodebugfurther.ProvidethisinformationtoPivotalSupportwhenfilingaticket.

The task-id fieldmapstotheBOSHtaskID.FormoreinformationonafailedBOSHtask,usethe bosh task TASK-ID .

The broker-request-guid mapstotheportionoftheOn-DemandBrokerlogcontainingthefailedstep.Accessthebrokerlogthroughyoursyslogaggregator,oraccessBOSHlogsforthebrokerbytyping bosh logs broker 0 .Ifyouhavemorethanonebrokerinstance,repeatthisprocessforeachinstance.

AccessBrokerandInstanceLogsandVMsBeforefollowingtheproceduresbelow,logintothecfCLI andtheBOSHCLI .

AccessBrokerLogsandVM(s)

YoucanaccesslogsusingOpsManager byclickingontheLogstabinthetileanddownloadingthebrokerlogs.

ToaccesslogsusingtheBOSHCLI,dothefollowing:

1. Identifytheon-demandbroker(ODB)deploymentbyrunningoneofthefollowingcommands,dependingonyourOpsManagerversion:

1.10andearlier bosh deployments

1.11 bosh2 deployments

1.12andlater bosh deploymentsOpsManagerVersion BOSHCommand

2. ForBOSHCLIv1only:

a. Run bosh download manifest ODB-DEPLOYMENT-NAME odb.yml todownloadtheODBmanifest.b. SelecttheODBdeploymentusing bosh deployment odb.yml .

3. ViewVMsinthedeploymentusingoneofthefollowingcommands:

1.10andearlier bosh instances

1.11 bosh2 -d DEPLOYMENT-NAME instances

1.12andlater bosh -d DEPLOYMENT-NAME instances

4. SSHontotheVMbyrunningoneofthefollowingcommands:

1.10andearlier bosh ssh service-instance_GUID

1.11 bosh2 -d service-instance_GUID ssh

1.12andlater bosh -d service-instance_GUID ssh

5. Downloadthebrokerlogsbyrunningoneofthefollowingcommands:

1.10andearlier bosh logs service-instance_GUID

1.11 bosh2 -d service-instance_GUID logs

1.12andlater bosh -d service-instance_GUID logs

ThearchivegeneratedbyBOSHorOpsManagerincludesthefollowinglogs:

LogName Description

broker.logRequeststotheon-demandbrokerandtheactionsthebrokerperformswhileorchestratingtherequest(e.g.generatingamanifestandcallingBOSH).Startherewhentroubleshooting.

broker_ctl.log Controlscriptlogsforstartingandstoppingtheon-demandbroker.

post-start.stderr.log

Errorsthatoccurduringpost-startverification.

post-start.stdout.log

Post-startverification.

drain.stderr.log Errorsthatoccurwhilerunningthedrainscript.

AccessServiceInstanceLogsandVMs1. Totargetanindividualserviceinstancedeployment,retrievetheGUIDofyourserviceinstancewiththecfCLIcommand cf service MY-SERVICE --guid .

2. ForBOSHCLIv1only:

a. Run bosh status --uuid toretrievetheBOSHDirectorGUID.

b. TodownloadyourBOSHmanifestfortheservice,run bosh download manifest service-instance_BOSH-DIRECTOR-GUID MANIFEST.ymlusingtheGUIDyoujustobtainedandafilenameyouwanttosavethemanifestas.

c. Editthefollowinglineintheserviceinstancemanifestthatyoujustsaved,toincludethecurrentBOSHDirectorGUID:

director_uuid: BOSH-DIRECTOR-GUID

d. Run bosh deployment MANIFEST.yml toselectthedeploymentusingtheDirectorUUID.

Note:“GUID”and“UUID”meanthesamething.

3. ViewVMsinthedeploymentusingoneofthefollowingcommands:

1.10andearlier bosh instances

1.11 bosh2 -d DEPLOYMENT-NAME instances

1.12andlater bosh -d DEPLOYMENT-NAME instances

4. SSHontoaVMbyrunningoneofthefollowingcommands:

1.10andearlier bosh ssh service-instance_GUID

1.11 bosh2 -d service-instance_GUID ssh

1.12andlater bosh -d service-instance_GUID ssh

5. Downloadtheinstancelogsbyrunningoneofthefollowingcommands:

1.10andearlier bosh logs service-instance_GUID

1.11 bosh2 -d service-instance_GUID logs

1.12andlater bosh -d service-instance_GUID logs

RunServiceBrokerErrandstoManageBrokersandInstancesFromtheBOSHCLI,youcanrunservicebrokererrandsthatmanagetheservicebrokersandperformmassoperationsontheserviceinstancesthatthebrokerscreated.Theseservicebrokererrandsinclude:

register-broker registersabrokerwiththeCloudControllerandlistsitintheMarketplace

deregister-broker deregistersabrokerwiththeCloudControllerandremovesitfromtheMarketplace

upgrade-all-service-instances upgradesexistinginstancesofaservicetoitslatestinstalledversion

delete-all-service-instances deletesallinstancesofservice

orphan-deployments detects“orphan”instancesthatarerunningonBOSHbutnotregisteredwiththeCloudController

Torunerrands:

1. ForBOSHCLIv1only:Selectthebrokerdeploymentbyrunningthiscommand:bosh deployment BOSH_MANIFEST.yml

2. RunoneofthefollowingcommandsdependingonyourOpsManagerversion:

1.10andearlier bosh run errand ERRAND_NAME

1.11 bosh2 -d DEPLOYMENT_NAME run-errand ERRAND_NAME

1.12andlater bosh -d DEPLOYMENT_NAME run-errand ERRAND_NAME

Examples:bosh run errand deregister-broker bosh2 -d DEPLOYMENT-NAME run-errand deregister-broker

RegisterBroker

The register-broker errandregistersthebrokerwithCloudFoundryandenablesaccesstoplansintheservicecatalog.Runthiserrandwheneverthebrokerisre-deployedwithnewcatalogmetadatatoupdatetheCloudFoundrycatalog.

Planswithdisabledserviceaccessarenotvisibletonon-adminCloudFoundryusers(includingOrgManagersandSpaceManagers).AdminCloudFoundry

userscanseeallplansincludingthosewithdisabledserviceaccess.

Theerranddoesthefollowing:

RegisterstheservicebrokerwithCloudController.

Enablesserviceaccessforanyplansthathavetheradiobuttonsetto enabled inthetileplanpage.

Disablesserviceaccessforanyplansthathavetheradiobuttonsetto disabled inthetileplanpage.

Doesnothingforanyforanyplansthathavetheradiobuttonsetto manual .

Toruntheerrand,dothefollowing:

1.10andearlier bosh run errand register-broker

1.11 bosh2 -d DEPLOYMENT-NAME run-errand register-broker

1.12andlater bosh -d DEPLOYMENT-NAME run-errand register-broker

DeregisterBroker

ThiserrandderegistersabrokerfromCloudFoundry.

DeletestheservicebrokerfromCloudController

Failsifthereareanyserviceinstances,withorwithoutbindings

UsetheDeleteAllServiceInstanceserrandtodeleteanyexistingserviceinstances.

1. ForBOSHCLIv1only:Selectthebrokerdeploymentbyrunningthecommand:bosh deployment BROKER_MANIFEST.yml .

1.10andearlier bosh run errand deregister-broker

1.11 bosh2 -d DEPLOYMENT-NAME run-errand deregister-broker

1.12andlater bosh -d DEPLOYMENT-NAME run-errand deregister-broker

IfyouhavemadechangestotheplandefinitionoruploadedanewtileintoOpsManager,youmaywanttoupgradealltheRabbitMQforPCFserviceinstancestothelatestsoftware/plandefinition.

The upgrade-all-service-instances erranddoesthefollowing:

Collectsalloftheserviceinstancestheon-demandbrokerhasregistered.

Foreachinstancetheerrandserially:

Issuesanupgradecommandtotheon-demandbroker.Re-generatestheserviceinstancemanifestbasedonitslatestconfigurationfromthetile.Deploysthenewmanifestfortheserviceinstance.

Waitsforthisoperationtocomplete,thenproceedstothenextinstance.

AddstoaretrylistanyinstancesthathaveongoingBOSHtasksatthetimeofupgrade.

Retriesanyinstancesintheretrylistuntilallareupgraded.

Ifanyinstancefailstoupgrade,theerrandfailsimmediately.Thispreventssystemicproblemsfromspreadingtotherestofyourserviceinstances.Runtheerrandbyfollowingeitheroftheproceduresbelow.

Toruntheerrand,youcaneitherselecttheerrandthroughtheOpsManagerUIandhaveitrunwhenyouclick Apply Changes ,ordothefollowing:

1.10andearlier bosh run errand upgrade-all-service-instances

1.11 bosh2 -d DEPLOYMENT-NAME run-errand upgrade-all-service-instances

1.12andlater bosh -d DEPLOYMENT-NAME run-errand upgrade-all-service-instances

DeleteAllServiceInstances

Thiserranddeletesallserviceinstancesofyourbroker’sserviceofferingineveryorgandspaceofCloudFoundry.ItusestheCloudControllerAPItodothis,andthereforeonlydeletesinstancestheCloudControllerknowsabout.ItwillnotdeleteorphanBOSHdeployments.

OrphanBOSHdeploymentsdon’tcorrespondtoaknownserviceinstance.Whilerare,orphandeploymentscanoccur.Usethe orphan-deployments errandtoidentifythem.

Unbindsallapplicationsfromtheserviceinstances.

Deletesallserviceinstancessequentially.

Checksifanyinstanceshavebeencreatedwhiletheerrandwasrunning.

Ifnewly-createdinstancesaredetected,theerrandfails.

1. ForBOSHCLIv1only:Selectthebrokerdeploymentbyrunningthecommand:bosh deployment BROKER_MANIFEST.yml .

1.10andearlier bosh run errand delete-all-service-instances

1.11 bosh2 -d service-instance_GUID delete-deployment

1.12andlater bosh -d service-instance_GUID delete-deployment

DetectOrphanedInstancesServiceInstances

Aserviceinstanceisdefinedas‘orphaned’whentheBOSHdeploymentfortheinstanceisstillrunning,buttheserviceisnolongerregisteredinCloudFoundry.

The orphan-deployments errandcollatesalistofservicedeploymentsthathavenomatchingserviceinstancesinCloudFoundryandreturnthelisttotheoperator.ItisthenuptotheoperatortoremovetheorphanedBOSHdeployments.

WARNING:Useextremecautionwhenrunningthiserrand.Youshouldonlyuseitwhenyouwanttototallydestroyalloftheon-demandserviceinstancesinanenvironment.

1. ForBOSHCLIv1only:Selectthebrokerdeploymentbyrunningthecommand:bosh deployment BROKER_MANIFEST.yml

2. RuntheerrandusingoneofthefollowingcommandsdependingonyourOpsManagerversion:

1.10andearlier bosh run errand orphan-deployments

1.11 bosh2 -d DEPLOYMENT-NAME run-errand orphan-deployments

1.12andlater bosh -d DEPLOYMENT-NAME run-errand orphan-deployments

Iforphandeploymentsexist,theerrandscriptwill:

Exitwithexitcode10

Outputalistofdeploymentnamesundera [stdout] header

Provideadetailederrormessageundera [stderr] header

Forexample:

[stdout][{"deployment_name":"service-instance_80e3c5a7-80be-49f0-8512-44840f3c4d1b"}]

[stderr]Orphan BOSH deployments detected with no corresponding service instance in Cloud Foundry. Before deleting any deployment it is recommended to verify the service instance no longer exists in Cloud Foundry and any data is safe to delete.

Errand 'orphan-deployments' completed with error (exit code 10)

ThesedetailswillalsobeavailablethroughtheBOSH /tasks/ APIendpointforuseinscripting:

$ curl 'https://bosh-user:bosh-password@bosh-url:25555/tasks/task-id/output?type=result' | jq .{ "exit_code": 10, "stdout": "[{"deployment_name":"service-instance_80e3c5a7-80be-49f0-8512-44840f3c4d1b"}]\n", "stderr": "Orphan BOSH deployments detected with no corresponding service instance in Cloud Foundry. Before deleting any deployment it is recommended to verify the service instance no longer exists in Cloud Foundry and any data is safe to delete.\n", "logs": { "blobstore_id": "d830c4bf-8086-4bc2-8c1d-54d3a3c6d88d" }}

Ifnoorphandeploymentsexist,theerrandscriptwill:

Exitwithexitcode0

Stdoutwillbeanemptylistofdeployments

Stderrwillbe None

[stdout][]

[stderr]None

Errand 'orphan-deployments' completed successfully (exit code 0)

Iftheerrandencountersanerrorduringrunningitwill:

Exitwithexit1

Stdoutwillbeempty

Anyerrormessageswillbeunderstderr

Tocleanuporphanedinstances,runthefollowingcommandoneachinstance:

1.10andearlier bosh delete deployment service-instance_SERVICE-INSTANCE-GUID

1.11 bosh2 delete-deployment service-instance_SERVICE-INSTANCE-GUID

1.12andlater bosh delete-deployment service-instance_SERVICE-INSTANCE-GUID

SelecttheBOSHDeploymentforaServiceInstance

ThisisanadditionaltroubleshootingoptionforBOSHCLIv1only.ItdoesnotapplytotheBOSHCLIv2.

1. RetrievetheGUIDofyourserviceinstancewiththecommand cf service YOUR-SERVICE-INSTANCE --guid .

2. TodownloadyourBOSHmanifestfortheservice,run bosh download manifest service-instance_SERVICE-INSTANCE-GUID myservice.yml usingtheGUIDyoujustobtainedandafilenameyouwanttousewhensavingthemanifest.

3. Run bosh deployment MY-SERVICE.yml toselectthedeployment.

GetAdminCredentialsforaServiceInstance

Toretrievetheadminandread-onlyadmincredentialsforaserviceinstance,performthefollowingsteps:

1. IdentifytheservicedeploymentbyGUID.

2. LogintoBOSH .

3. DownloadthemanifestfortheserviceinstanceandaddtheGUIDifusingtheBOSHCLIv1.

4. Lookinthemanifestforthe admin and roadmin credentials.

ReinstallaTile

Toreinstallatileinthesameenvironmentwhereitwaspreviouslyuninstalled:

1. Ensurethattheprevioustilewascorrectlyuninstalledasfollows:

a. Loginasanadminwith cf login .b. Use cf m toconfirmthattheMarketplacedoesnotlistRabbitMQforPCF.c. DependingonwhichversionoftheBOSHCLIyouareusing,followoneofthestepsbelowtologintoBOSHasanadmin:

i. ForBOSHCLIv2:Use bosh2 log-in .ii. ForBOSHCLIv1:Use bosh login .

d. DependingonwhichversionoftheBOSHCLIyouareusing,followoneofthestepsbelowtodisplayyourBOSHdeploymentstoconfirmthattheoutputdoesnotshowaRabbitMQforPCFdeployment:

i. ForBOSHCLIv2:Use bosh2 deployments .ii. ForBOSHCLIv1:Use bosh deployments .

e. Runthe“delete-all-service-instances”errandtodeleteeveryinstanceoftheservice.f. Runthe“deregister-broker”errandtodeletetheservicebroker.g. DependingonwhichversionoftheBOSHCLIyouareusing,followoneofthestepsbelow:

WARNING:RunningthiscommandmayleaveIaaSresourcesinanunusablestate.

SkipthisstepifyouareusingtheBOSHCLIv2.YoucannotdownloadthemanifestwiththeBOSHCLIv2.Openitinatexteditorinstead.

i. ForBOSHCLIv2:Use bosh2 delete-deployment BROKER-DEPLOYMENT-NAME todeletetheservicebrokerBOSHdeployment.ii. ForBOSHCLIv1:Use bosh delete deployment BROKER-DEPLOYMENT-NAME todeletetheservicebrokerBOSHdeployment.

h. Reinstallthetile.

ViewResourceSaturationandScaling

BOSHCLIv2:Viewingstatistics

Toviewusagestatisticsforanyservicedothefollowing:

2. RunthefollowingcommandsdependingonyourOpsManagerversion:

OpsManagerVersion BOSHCommands

v1.10andearlier

RuntheBOSHCLIv1command bosh vms --vitals .Toviewprocess-levelinformation,run bosh instances --ps .

v1.11RuntheBOSHCLIv2command bosh2 -d DEPLOYMENT-NAME vms --vitals .Toviewprocess-levelinformation,runbosh2 -d DEPLOYMENT-NAME instances --ps

v1.12andlaterRuntheBOSHCLIv2command bosh -d DEPLOYMENT-NAME vms --vitals .Toviewprocess-levelinformation,runbosh2 -d DEPLOYMENT-NAME instances --ps

IdentifyServiceInstanceOwner

IfyouwanttoidentifywhichappsareusingaspecificserviceinstancefromtheBOSHdeploymentsname,youcanrunthefollowingsteps:

1. Takethedeploymentnameandstripthe service-instance_ leavingyouwiththeGUID.

2. LogintoCFasanadmin.

3. Obtainalistofallservicebindingsbyrunningthefollowing: cf curl /v2/service_instances/GUID/service_bindings

4. Theoutputfromtheabovecurlgivesyoualistof resources ,witheachitemreferencingaservicebinding,whichcontainsthe APP-URL .Tofindthename,org,andspacefortheapp,runthefollowing:

a. cf curl APP-URL andrecordtheappnameunder entity.nameb. cf curl SPACE-URL toobtainthespace,usingthe entity.space_url fromtheabovecurl.Recordthespacenameunder entity.namec. cf curl ORGANIZATION-URL toobtaintheorg,usingthe entity.organization_url fromtheabovecurl.Recordtheorganizationname

under entity.name

MonitorQuotaSaturationandServiceInstanceCount

QuotasaturationandtotalnumberofserviceinstancesareavailablethroughODBmetricsemittedtoLoggregator.Themetricnamesareshownbelow:

on-demand-broker/SERVICE-NAME-MARKETPLACE/quota_remaining globalquotaremainingforallinstancesacrossallplans

Note:Whenrunning cf curl ensurethatyouqueryallpages,becausetheresponsesarelimitedtoacertainnumberofbindingsperpage.Thedefaultis50.Tofindthenextpagecurlthevalueunder next_url

on-demand-broker/SERVICE-NAME-MARKETPLACE/PLAN-NAME/quota_remaining quotaremainingforaparticularplan

on-demand-broker/SERVICE-NAME-MARKETPLACE/total_instances totalinstancescreatedacrossallplans

on-demand-broker/SERVICE-NAME-MARKETPLACE/PLAN-NAME/total_instances totalinstancescreatedforagivenplan

FrequentlyAskedQuestions

WhatshouldIcheckbeforedeployinganewversionofthetile?

EnsurethatallnodesintheclusterarehealthyviatheRabbitMQManagementUI,orhealthmetricsexposedviathefirehose.

DonotrelyonBOSH instances ,toexecutethistaskcorrectly.ThatoutputreflectsthestateoftheErlangVMusedbyRabbitMQ,nottheRabbitMQapplication.

WhatisthecorrectwaytostopandstartRabbitMQinPCF?

OnlyBOSHcommandsshouldbeusedbytheoperatortointeractwiththeRabbitMQapplication.ExamplesforbothversionsoftheCLIaregivenbelow.

ForOpsManagerv1.10orearlier:bosh stop rabbitmq-server bosh start rabbitmq-server

ForOpsManagerv1.11orlater:bosh2 stop rabbitmq-server bosh2 start rabbitmq-server

ThereareBOSHjoblifecyclehookswhichareonlyfiredwhenrabbitmq-serverisstoppedthroughBOSH.Youcanalsostopindividualinstancesbyrunningoneofthefollowingcommands:

ForOpsManagerv1.10orearlier: bosh stop JOB [index]

ForOpsManagerv1.11orlater: bosh2 stop JOB [index]

WhathappenswhenIstoptheRabbitMQserver?

YoucanstoptheRabbitMQserverwithBOSH stop .

BOSHstartstheshutdownsequencefromthebootstrapinstance.

ThiscommandtellstheRabbitMQapplicationtoshutdownandthenshutdowntheErlangVMinwhichitisrunning.Ifthissucceeds,runthefollowingcheckstoensurethattheRabbitMQapplicationandErlangVMhavestopped:

1. If /var/vcap/sys/run/rabbitmq-server/pid exists,checkthatthePIDinsidethisfiledoesnotpointtoarunningErlangVMprocess.ThisprocesscheckstheErlangPIDandnottheRabbitMQPID.

2. Checkthat rabbitmqctl doesnotreturnanErlangVMPID.

Oncethiscompletesonthebootstrapinstance,BOSHwillcontinuethesamesequenceonthenextinstance.AllremainingRabbitMQserversstopsequentiallyafterthat.

Note:Quotametricsarenotemittedifnoquotahasbeenset.

Note:Donotuse monit stop rabbitmq-server

.Thiscommanddoesnotcallthedrainscripts.

WhathappenswhentheRabbitMQserverfailstostop?

IfBOSH stop fails,youwilllikelygetanerrorsayingthatthedrainscriptfailedwith:

result: 1 of 1 drain scripts failed. Failed Jobs: rabbitmq-server.

WhatdoIdowhentheRabbitMQserverfailstostop?

Thedrainscriptlogsto /var/vcap/sys/log/rabbitmq-server/drain.log .Ifyouhavearemotesyslogconfigured,thiswillappearasthe rmq_server_drain program.

1. BOSH ssh intothefailingrabbitmq-serverinstanceandstarttherabbitmq-serverjobbyrunning monit start rabbitmq-server ).

2. Whentherabbitmq-serverjobisrunning(confirmthisvia monit status ),run DEBUG=1 /var/vcap/jobs/rabbitmq-server/bin/drain .Thiswilltellyouexactlywhyit’sfailing.

HowcanImanuallybackupthestateoftheRabbitMQcluster?

YoucanbackupthestateofaRabbitMQclusterforboththeon-demandandpre-provisionedservicesusingtheRabbitMQManagementAPI.Backupsincludevhosts,exchanges,queuesandusers.

BackupManually1. LogintotheRabbitMQManagementUIastheadminuseryoucreated.

2. Selectexportdefinitionsfromthemainpage.

BackupandRestorewithaScript

UsetheAPItorunscriptswithcodesimilartothefollowing:

1. Forthebackup:

curl -u "$USERNAME:$PASSWORD" "http://$RABBIT_ADDRESS:15672/api/definitions"-o "$BACKUP_FOLDER/rabbit-backup.json"

2. Fortherestore:

curl -u "$USERNAME:$PASSWORD" "http://$RABBIT_ADDRESS:15672/api/definitions"-X POST -H "Content-Type: application/json" -d"@$BACKUP_FOLDER/rabbit-backup.json"

Whatpre-upgradechecksshouldIdo?

BeforedoinganyupgradeofRabbitMQ,Pivotalrecommendscheckingthefollowing:

Note:YouwillnotbeabletostartthejobwithBOSH start asthiscommandalwaysrunsthedrainscriptfirst.Itwillfailsincethedrainscriptisfailing.

1. InOperationsManagercheckthatthestatusofalloftheinstancesishealthy.

2. LogintotheRabbitMQManagementUIandcheckthatnoalarmshavebeentriggeredandthatallnodesarehealthy,thatis,theyshoulddisplayasgreen.

3. Checkthatthesystemisnotclosetohittingeitherthememoryordiskalarm.DothisbylookingatwhathasbeenconsumedbyeachnodeintheRabbitMQManagmentUI.

KnowledgeBase(Community)FindtheanswertoyourquestionandbrowseproductdiscussionsandsolutionsbysearchingthePivotalKnowledgeBase .

FileaSupportTicketYoucanfileasupporttickethere .Besuretoprovidetheerrormessagefrom cf service YOUR-SERVICE-INSTANCE .

Tohelpexpeditetroubleshooting,alsoprovideyourservicebrokerlogs,yourserviceinstancelogsandBOSHtaskoutput,ifyourcf service YOUR-SERVICE-INSTANCE outputincludesa task-id .

UsingOn-DemandRabbitMQforPCFThistopicprovidesinstructionsfordevelopersusingtheon-demandRabbitMQservicefortheirPivotalCloudFoundry(PCF)apps.RabbitMQenablesmessagingbetweencloud-basedservers,appsanddevices.

TheseproceduresusetheCloudFoundryCommand-LineInterface(cfCLI).YoucanalsouseAppsManager toperformthesametasksusingagraphicalUI.

Forgeneralinformation,seeManagingServiceInstanceswiththecfCLI .

PrerequisitesTouseon-demandRabbitMQforPCFwithyourPCFapps,youneed:

APCFinstallationwithRabbitMQforPCF installedandlistedintheMarketplace

ASpaceDeveloper orAdminaccountonthePCFinstallation

Alocalmachinewiththefollowinginstalled:

abrowserashelltheCloudFoundryCommand-LineInterface (cfCLI)theLinuxwatch command

Tologinto theorgandspacecontainingyourapp

DeveloperGuide

EntriesintheVCAP_SERVICESEnvironmentVariableAppsrunninginCloudFoundrygainaccesstotheboundserviceinstancesviaanenvironmentvariablecredentialshashcalledVCAP_SERVICES .Anexamplehashisshownbelow:

{ "p-rabbitmq": [{ "credentials": { "dashboard_url": "http://pivotal-rabbitmq.your.pcf.example.com/#/login/b5d0ad14-4352-48e8-8982-d5b1d257029f/tavk86pnnns1ddiqpsdtbchurn", "username": "b5d0ad14-4352-48e8-8982-d5b1d257029f", "password": "tavk86pnnns1ddiqpsdtbchurn", "protocols": { "amqp": { "password": "tavk86pnnns1ddiqpsdtbchurn", "username": "b5d0ad14-4352-48e8-8982-d5b1d257029f", "uris": [ "amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:tavk86pnnns1ddiqpsdtbchurn@10.0.0.41:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7", "amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:tavk86pnnns1ddiqpsdtbchurn@10.0.0.51:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7" ] } } } }]}

Youcanreadmoredetailsabouttheenvironmentvariable VCAP_SERVICES here.

TheCreate-BindProcessBecauseeveryappandserviceinPCFisscopedtoaspace ,anappcanonlyuseaserviceifaninstanceoftheserviceexistsinthesamespace.

TouseRabbitMQinaPCFapp:

1. UsethecfCLI orAppsManager tologintotheorgandspacethatcontainstheapp.

2. MakesureaninstanceoftheRabbitMQforPCFserviceexistsinthesamespaceastheapp.

IfthespacedoesnotalreadyhaveaRabbitMQforPCFinstance,createone.IfthespacealreadyhasaRabbitforPCFinstance,youcanbindyourapptotheexistinginstanceorcreateanewinstancetobindtoyourapp.

3. BindtheapptotheRabbitMQforPCFserviceinstance,toenabletheapptouseRabbitMQ.

ConfirmServiceAvailabilityForanapptouseaservice,1)theservicemustbeavailableintheMarketplaceforitsspaceand2)aninstanceoftheservicemustexistinitsspace.

YoucanconfirmbothoftheseusingthecfCLIasfollows.

1. TofindoutifOn-DemandRabbitMQforPCFserviceisavailableintheMarketplace:

a. Enter cf marketplaceb. Iftheoutputlists ondemand-rabbitmq inthe service column,on-demandRabbitMQforPCFisavailable.Ifitisnotavailable,askyour

operatortoinstallit.

$ cf marketplaceGetting services from marketplace in org my-org / space my-space as user@example.com...OKservice plans description[...]ondemand-rabbitmq Solo RabbitMQ Service[...]

2. ToconfirmthatanOn-DemandRabbitMQforPCFinstanceisrunninginthespace

a. Enter cf servicesb. Any ondemand-rabbitmq listingsinthe service columnareserviceinstancesofon-demandRabbitMQinthespace.

$ cf servicesGetting services in org my-org / space my-space as user@example.com...OKname service plan bound apps last operationmy-instance ondemand-rabbitmq Solo create succeeded

Youcanbindyourapptoanexistinginstanceorcreateanewinstancetobindtoyourapp.

CreateaServiceInstanceUnlikepre-provisionedservices,on-demandservicesarecreatedasynchronously,notimmediately.The watch commandshowsyouwhenyourserviceinstanceisreadytobindanduse.

Tocreateaninstanceoftheon-demandRabbitMQforPCFservice,run cf create-service

1. Enter cf create-service ondemand-rabbitmq Solo SERVICE_INSTANCE

Where SERVICE_INSTANCE isanameyouchoosetoidentifytheserviceinstance.Thisnamewillappearunder service [sic]inoutputfrom cf

services .

2. Enter watch cf services andwaitforthe last operation foryourinstancetoshowas create succeeded .

$ cf create-service ondemand-rabbitmq Solo my-instance

Creating service my-instance in org my-org / space my-space as user@example.com...OK

$ watch cf services

Getting services in org my-org / space my-space as user@example.com...OKname service plan bound apps last operationmy-instance ondemand-rabbitmq Solo create succeeded

Ifyougetanerror,seeTroubleshootingInstances.

BindaServiceInstancetoYourAppForanapptouseaservice,youmustbindittoaserviceinstance.Dothisafteryoupushorre-pushtheappusing cf push .

TobindanapptoaRabbitMQinstancerun $ cf bind-service

1. Enter cf bind-service APP SERVICE_INSTANCE

Where APP istheappyouwanttousetheRabbitMQserviceinstanceand SERVICE_INSTANCE isthenameyousuppliedwhenyouran cf create-

service .

$ cf bind-service my-app my-instance

Binding service mydb to my-app in org my-org / space test as user@example.com...OKTIP: Use 'cf push' to ensure your env variable changes take effect

UsetheRabbitMQServiceinYourAppToaccesstheRabbitMQservicefromyourapp:

1. Run cf env APP_NAME withthenameoftheappboundtotheRabbitMQforPCFinstance.

2. Intheoutput,notetheconnectionstringslistedinthe VCAP_SERVICES > credentials objectfortheapp.

3. Inyourappcode,calltheRabbitMQserviceusingtheconnectionstrings.

ForhowtocodeyourapptouseRabbitMQmessaging,seeAboutUsingPivotalRabbitMQ>ClientDocumentationintheRabbitMQdocumentation .

RestageanAppwithaNewServiceInstanceIfaserviceinstancehasbeenupdatedbasedonanewversionoftheservice,youneedtorun cf restage torestageyourapptousethenewinstance.

1. Enter cf restage-app APP

Where APP istheappyouwanttousetheupdatedserviceinstance.

$ cf restage-app my-app

Whenanewversionofaserviceobsoletesanolderversion,theplatformoperatormayaskyoutoupdateyourinstancesoftheserviceandrestageanyappsboundtotheserviceinstances.

Pushingnewversionofanappautomaticallyrestagestheapponanyserviceinstancesitisboundto.

UnbindaServiceInstancetoYourAppTostopanappfromusingaserviceitnolongerneeds,unbinditfromtheserviceinstanceusing cf unbind-

service.

1. Enter cf unbind-service APP SERVICE_INSTANCE

Where APP istheappyouwanttostopusingtheRabbitMQserviceinstanceand SERVICE_INSTANCE isthenameyousuppliedwhenyouran cf

create-service .

$ cf unbind-service my-app my-instance

Unbinding app my-app from service my-instance in org my-org / space my-space as user@example.com...OK

DeleteaServiceInstance

Todeleteaserviceinstance,run cf delete-service

1. Enter cf delete-service SERVICE_INSTANCE

Where SERVICE_INSTANCE isthenameoftheservicetodelete.

$ cf delete-service my-instance

Are you sure you want to delete the service my-instance ? yDeleting service my-service in org my-org / space my-space as user@example.com...OK

2. Enter watch cf service SERVICE_INSTANCE andwaitfora Service instance not found errorindicatingthattheinstancenolongerexists.

Youcannotdeleteaserviceinstancethatanappisboundto.

CreateanAdminUserforaServiceInstanceIfyouwanttogetadminprivilegestotheRabbitMQManagementUI,youcancreateanadminuserforaserviceinstance,andobtainusercredentialsthatyoucansharewithotherappdevelopers.

Bothoperatorsandappdeveloperscanusethisprocedure.Forinstructions,seeCreateanAdminUserforaServiceInstance.

FederateExchangesandQueuesYoucanfederateexchangesandqueuesinRabbitMQforPCF,asyouwouldinanyRabbitMQdeployment.

Tofederateexchangesandqueues,dothefollowing:

1. CreateaservicekeybyfollowingtheinstructionsinCreateanAdminUserforaServiceInstance.

Theoutputoftheaboveprocedurereturnsadminusercredentials,alongwithotherdata.

2. Intheoutputfromtheabovestep,lookforthe uris array.Itwillhavethispattern:

{..."uri": "amqp://USERNAME:PASSWORD@IP_ADDRESS/VHOST","uris": ["amqp://USERNAME:PASSWORD@IP_ADDRESS/VHOST"],...}

Forexample:

{..."uri": "amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f: passwordexample123456789@10.0.0.41:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7","uris": ["amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f: passwordexample123456789@10.0.0.41:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7","amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f: passwordexample123456789@10.0.0.51:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7"]},...}

3. Setupfederationasyounormallywould,usingtheRabbitMQManagementUIorAPI,withtheURIsfoundinthe uris arrayyougotfromthestepabove.

Forinstructionsonfederation,seetheRabbitMQdocumentation .

ShovelExchangesandQueues

YoucanshovelexchangesandqueuesinRabbitMQforPCF,asyouwouldinanyRabbitMQdeployment.

Toshovelexchangesandqueues,dothefollowing:

1. CreateaservicekeybyfollowingtheinstructionsinCreateanAdminUserforaServiceInstance.

Theoutputoftheaboveprocedurereturnsadminusercredentials,alongwithotherdata.

2. Intheoutputfromtheabovestep,lookforthe uris array.Itwillhavethispattern:

{..."uri": "amqp://USERNAME:PASSWORD@IP_ADDRESS/VHOST","uris": ["amqp://USERNAME:PASSWORD@IP_ADDRESS/VHOST"],...}

Forexample:

{..."uri": "amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f: passwordexample123456789@10.0.0.41:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7","uris": ["amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f: passwordexample123456789@10.0.0.41:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7","amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f: passwordexample123456789@10.0.0.51:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7"]},...}

3. Setupshovelasyounormallywould,usingtheRabbitMQManagementUIorAPI,withtheURIsfoundinthe uris arrayyougotfromthestepabove.

Forshovelinstructions,seetheRabbitMQdocumentation .

RabbitMQ®EnvironmentVariablesThistopicprovidesareferencefortheenvironmentvariablesthatCloudFoundrystoresforRabbitMQforPCFserviceinstances.Thesevariablesincludethecredentialsthatappsusetoaccesstheserviceinstances.

VCAP_SERVICESApplicationsrunninginCloudFoundrygainaccesstotheboundserviceinstancesviaanenvironmentvariablecredentialshashcalledVCAP_SERVICES .Anexamplehashisshowbelow:

{ "p-rabbitmq": [{ "label": "p-rabbitmq", "name": "my-rabbit-service-instance", "plan": "standard", "tags": ["rabbitmq", "messaging", "message-queue", "amqp", "pivotal"], "credentials": { "dashboard_url": "http://pivotal-rabbitmq.your.pcf.example.com/#/login/b5d0ad14-4352-48e8-8982-d5b1d257029f/tavk86pnnns1ddiqpsdtbchurn", "username": "b5d0ad14-4352-48e8-8982-d5b1d257029f", "vhost": "62e5ab21-7b38-44ac-b139-6aa97af01cd7", "password": "#passwordexample123456789", "ssl": false, "hostname": "10.0.0.41", "hostnames": [ "10.0.0.41", "10.0.0.51"], "uri": "amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:tavk86pnnns1ddiqpsdtbchurn@10.0.0.41/62e5ab21-7b38-44ac-b139-6aa97af01cd7", "uris": [ "amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:tavk86pnnns1ddiqpsdtbchurn@10.0.0.41/62e5ab21-7b38-44ac-b139-6aa97af01cd7", "amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:tavk86pnnns1ddiqpsdtbchurn@10.0.0.51/62e5ab21-7b38-44ac-b139-6aa97af01cd7"], "http_api_uri": "http://b5d0ad14-4352-48e8-8982-d5b1d257029f:tavk86pnnns1ddiqpsdtbchurn@10.0.0.41:15672/api", "http_api_uris": [ "http://b5d0ad14-4352-48e8-8982-d5b1d257029f:tavk86pnnns1ddiqpsdtbchurn@10.0.0.41:15672/api", "http://b5d0ad14-4352-48e8-8982-d5b1d257029f:tavk86pnnns1ddiqpsdtbchurn@10.0.0.51:15672/api"], "protocols": { "amqp": { "password": "passwordexample123456789", "port": 5672, "ssl": false, "username": "b5d0ad14-4352-48e8-8982-d5b1d257029f", "vhost": "62e5ab21-7b38-44ac-b139-6aa97af01cd7", "host": "10.0.0.41", "hosts": [ "10.0.0.41", "10.0.0.51"], "uri": "amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:passwordexample123456789@10.0.0.41:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7", "uris": [ "amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:passwordexample123456789@10.0.0.41:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7", "amqp://b5d0ad14-4352-48e8-8982-d5b1d257029f:passwordexample123456789@10.0.0.51:5672/62e5ab21-7b38-44ac-b139-6aa97af01cd7"]}, "management": { "username": "b5d0ad14-4352-48e8-8982-d5b1d257029f", "password": "passwordexample123456789", "path": "/api", "port": 15672, "ssl": false, "host": "10.0.0.41", "hosts": [ "10.0.0.41", "10.0.0.51"], "uri": "http://b5d0ad14-4352-48e8-8982-d5b1d257029f:passwordexample123456789@10.0.0.41:15672/api", "uris": [ "http://b5d0ad14-4352-48e8-8982-d5b1d257029f:passwordexample123456789@10.0.0.41:15672/api", "http://b5d0ad14-4352-48e8-8982-d5b1d257029f:passwordexample123456789@10.0.0.51:15672/api"]}}}}]}

Youcansearchforyourservicebyits name ,givenwhencreatingtheserviceinstance,ordynamicallyviathe tags or label properties.The credentials

propertycanbeusedasfollows:

Thetoplevelproperties uri , uris , vhost , username , password , hostname and hostnames provideaccesstotheAMQP0.9.1protocol.

Amoreflexibleapproachisprovidedbythe credentials.protocols property,whichhasakeyperenabledprotocol.Thepossiblekeysare amqp ,management , mqtt ,and stomp .IfSSLisenabled,thenthekeyswillbe amqp+ssl , management+ssl , mqtt+ssl ,and stomp+ssl respectively.

Thevaluesassociatedwitheachofthesekeysgivesaccesscredentialsspecifictoeachprotocol.Inallcases,URIsareprovided,alongwiththeindividualcomponents.

ChangingEnabledPluginsandProtocols

IfyouadjustthepluginsandprotocolsenabledforRabbitMQ,youmayneedtoforceallapp’s VCAP_SERVICES environmentvariabletoberegenerated.Addingandremovingthefollowingpluginsrequireboundapplicationstoberestaged:

rabbitmq_management

rabbitmq_stomp

rabbitmq_mqtt

rabbitmq_amqp1_0

IncommonwithallservicesinPivotalCloudFoundry (PCF),the VCAP_SERVICES environmentvariableforanapplicationisonlymodifiedwhentheapplicationisboundtoaserviceinstance.Userswillneedto cf unbind-

service, cf bind-

serviceand cf restage theirappinthisscenario.

Note:Removingoraddingplugins/protocolsmaycauseappsboundwithRabbitMQtobreak.

TroubleshootingInstancesThistopicprovidesbasicinstructionsforappdeveloperstroubleshootingOn-DemandRabbitMQ®forPCF.

ErrorsYoumayseeanerrorwhenusingtheCloudFoundryCommand-LineInterface(cfCLI)toperformbasicoperationsonaRabbitMQforPCFserviceinstance:

cf create

cf update

cf bind

cf unbind

cf delete

ParseaCloudFoundry(CF)ErrorMessageFailedoperations(create,update,bind,unbind,delete)resultinanerrormessage.YoucanretrievetheerrormessagelaterbyrunningthecfCLIcommand cf service INSTANCE-NAME .

$ cf service myservice

Service instance: myserviceService: super-dbBound apps:Tags:Plan: dedicated-vmDescription: Dedicated InstanceDocumentation url:Dashboard:

Last OperationStatus: create failedMessage: Instance provisioning failed: There was a problem completing your request. Please contact your operations team providing the following information: service: redis-acceptance, service-instance-guid: ae9e232c-0bd5-4684-af27-1b08b0c70089, broker-request-id: 63da3a35-24aa-4183-aec6-db8294506bac, task-id: 442, operation: createStarted: 2017-03-13T10:16:55ZUpdated: 2017-03-13T10:17:58Z

Usetheinformationinthe Message fieldtodebugfurther.ProvidethisinformationtoPivotalSupportwhenfilingaticket.

The task-id fieldmapstotheBOSHtaskID.FormoreinformationonafailedBOSHtask,usethe bosh task TASK-ID .

The broker-request-guid mapstotheportionoftheOn-DemandBrokerlogcontainingthefailedstep.Accessthebrokerlogthroughyoursyslogaggregator,oraccessBOSHlogsforthebrokerbytyping bosh logs broker 0 .Ifyouhavemorethanonebrokerinstance,repeatthisprocessforeachinstance.

RetrieveServiceInstanceInformation1. Logintothespacecontainingtheinstanceorfailedinstance.

$ cf login

2. Ifyoudonotknowthenameoftheserviceinstance,run cf services toseealistingofallserviceinstancesinthespace.Theserviceinstancesarelistedinthe name column.

$ cf servicesGetting services in org my-org / space my-space as user@example.com...OKname service plan bound apps last operationmy-instance ondemand-rabbitmq Solo create succeeded

3. Run cf service SERVICE-INSTANCE-NAME toretrievemoreinformationaboutaspecificinstance.

4. Run cf service SERVICE-INSTANCE-NAME --guid toretrievetheGUIDoftheinstance,whichisusefulfordebugging.

RetrieveRabbitMQInstanceCredentialsIfyouwanttoaccesstheManagementDashboardortheRabbitMQserverfortroubleshooting,youcancreateanewservice-keytoretrieveRabbitMQinstancecredentials.Pivotalrecommendsthatyouusethiskeyfortroubleshootingonly,andthatyoudeletethekeyaftertroubleshooting.Toretrievethecredentials,dothefollowing:

1. Createaservice-keyforyourRabbitMQinstanceusingthecommand cf create-service-key INSTANCE-NAME SERVICE-KEY-NAME .

2. Retrievethecredentialsusingthecommand cf service-key INSTANCE-NAME SERVICE-KEY-NAME .

Forexample:

$ cf create-service-key my-rmq-instance my-keyCreating service key my-key for service instance my-rmq-instance as admin...OK$ cf service-key my-rmq-instance my-keyGetting key my-key for service instance my-rmq-instance as admin...{ "host": "10.0.8.4", "password": "", "port": 6379}

SelecttheBOSHDeploymentforaServiceInstanceThisisanadditionaltroubleshootingoptionforBOSHCLIv1only.ItdoesnotapplytotheBOSHCLIv2.

1. RetrievetheGUIDofyourserviceinstancewiththecommand cf service YOUR-SERVICE-INSTANCE --guid .

2. TodownloadyourBOSHmanifestfortheservice,run bosh download manifest service-instance_SERVICE-INSTANCE-GUID myservice.yml usingtheGUIDyoujustobtainedandafilenameyouwanttousewhensavingthemanifest.

3. Run bosh deployment MY-SERVICE.yml toselectthedeployment.

KnowledgeBase(Community)FindtheanswertoyourquestionandbrowseproductdiscussionsandsolutionsbysearchingthePivotalKnowledgeBase .

FileaSupportTicketYoucanfileasupporttickethere .Besuretoprovidetheerrormessagefrom cf service YOUR-SERVICE-INSTANCE .

Tohelpexpeditetroubleshooting,ifpossiblealsoprovideyourservicebrokerlogs,serviceinstancelogs,andBOSHtaskoutput.Yourcloudoperatorshouldbeabletoobtainthesefromyourerrormessage.

DeleteRabbitMQInstanceOn-DemandBrokerprovidesaBOSHcommandtodeletealltheOn-DemandBrokerdeployedinstances.Todeletetheinstances,dothefollowingprocedure:

1. ForBOSHCLIv1only:SetthedeploymenttotheOn-DemandBrokerinBOSH.

2. RunoneofthefollowingcommandstodeleteallinstancesoftheOn-DemandBroker,dependingonyourOpsManagerversion:

Forv1.10orearlier: bosh run errand delete-sub-deploymentsForv1.11orlater: bosh2 run-errand delete-sub-deployments

WARNING:Thiscommanddeletesdeploymentinstancesserially.Itisverydestructiveandtheoperationcannotbeundone.

InstallingandConfiguringthePre-ProvisionedServiceThistopicprovidesinstructionstoPivotalCloudFoundry(PCF)operatorsabouthowtoinstall,configure,anddeploytheRabbitMQforPCFtiletoprovideapre-provisionedservice.

TheRabbitMQopensourceproductprovidesadditionaldocumentation.FormoreinformationaboutgettingstartedwithRabbitMQandensuringproductionreadiness,seetheProductionChecklistintheRabbitMQDocumentation .

DownloadandInstalltheTile1. DownloadtheproductfilefromPivotalNetwork .

2. NavigatetotheOpsManagerInstallationDashboardandclickImportaProducttouploadtheproductfile.

3. UndertheImportaProductbutton,click+nexttotheversionnumberofRabbitMQforPCF.Thisaddsthetiletoyourstagingarea.

4. ClickthenewlyaddedRabbitMQforPCFtile.Thisletsyoubeginconfiguringthetile.Theinstallationiscompletewhenyouapplythechangesfromtheconfiguration.

ConfigurePre-ProvisionedRabbitMQforPCFTheconfigurationscreenbelowappearswhenyouclicktheRabbitMQforPCFtileinOpsManager.Anorangecirclebesideatabindicatesthatyoumustcompleteaconfigurationinthetab.Agreencheckmarkindicatesthatthetabispreconfiguredandyoumayoptionallychangeitssettings.

Note:Forinstructionsabouthowtoinstall,configure,anddeploytheRabbitMQforPCFtileasanon-demandservice,seeInstallingandConfiguringRabbitMQforPCFasanOn-DemandService.

WhichSettingsTabstoConfigureforthePre-ProvisionedServiceConfigurethefollowingtabsforthepre-provisionedservice:

RabbitMQSettingsTab Instructions

AssignAZsandNetworks AssignAZsandNetworks

RabbitMQ RabbitMQ

Syslog Syslog

DedicatedInstance:SingleNodePlan DedicatedInstance:SingleNodePlan

Errands Errands

Stemcell Stemcell

AssignAZsandNetworksFollowthestepsbelowtoconfiguretheAZsandnetworks.

1. IntheSettingsscreen,clickAssignAZsandNetworks.

2. ConfigurethefieldsontheAssignAZsandNetworksasfollows.Allfieldsarerequired,thoughsomedonotapplytothepre-provisionedservice.

RequiredFields Instructions

Placesingletonjobsin

Selectaregion.Thisselectiononlyaffectstheon-demandservice.

Balanceotherjobsin

Selectadditionalregion(s).Thisselectiononlyaffectsthepre-provisionedservice.

Network

SelectanetworkfortheRabbitMQBroker.

ThisshouldbeaseparatenetworkfromtheoneyouselectforServiceNetwork.ThisnetworkisrepresentedbytheDefaultNetwork,describedinDefaultNetworkandServiceNetwork.Typically,youselectthenetworkusedforthePivotalElasticRuntimecomponents.

ServiceNetwork

Selectanetwork.Thisselectiononlyaffectstheon-demandservice.

3. ClickSave.

RabbitMQToconfigurethefollowingsectionsintheRabbitMQtab,intheSettingsscreen,clickRabbitMQ.

RabbitMQAdminUserCredentialsEnteranadminusernameandpasswordforRabbitMQ.ThisgrantsyoufulladminaccesstotheRabbitMQManagementUI.

EnterthedesiredcredentialsinthissectionoftheRabbitMQtab:

Important:YoucannotchangetheregionsornetworksafteryouhaveclickedApplyChangesinthefinalstepbelow.

WARNING:ChangingtheNetworkafteryouhaveconfiguredit,orchangingtheIPconfiguration,resultsinafaileddeployment.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .

PluginsChoosewhichpluginsyouwanttoenableinthissectionoftheRabbitMQtab.

Youmustleavetherabbitmq_managementpluginenabledforthisproducttowork.

FormoreinformationaboutRabbitMQplugins,seetheRabbitMQdocumentation .

HAProxyPortsEntertheportsHAProxyshouldloadbalancetotheRabbitMQnodesinthissectionoftheRabbitMQtab:

Allthedefaultportsofalltheavailablepluginsareload-balancedbydefault.However,ifyouinstallextraprotocolplugins,orprovideacustomconfigurationthatchangestheportsRabbitMQlistenson,thenyoumustupdatethelistofload-balancedports.

Youmustleavethemanagementpluginlisteningonport 15672 andloadbalancethatport.

IfyouchangethetopologyofyourRabbitMQcluster,theHAProxyisautomaticallyreconfiguredduringthedeployment.

SSL(Optional)ProvideSSLcertificatesandkeysforusebytheRabbitMQclusterinthissectionoftheRabbitMQtab:

Note:Torotateyouradministratorcredentials,enteranewusernameandpassword,saveyouroptions,andredeploybyreturningtotheOpsManagerInstallationDashboardandclickingApplyChanges.

SSLissimultaneouslyprovidedontheAMQPSport(5671)andthemanagementport(15672).NootherpluginsareautomaticallyconfiguredforusewithSSL.

IfyouprovideSSLkeysandcertificatesnon-SSLsupportisdisabled.

SSLsettingsareappliedequallyacrossallVMsinthecluster.

FormoreinformationaboutSSLsupport,seetheRabbitMQdocumentation .

ErlangCookie(Optional)ProvideanErlangcookietobeusedbytheclusterinthissectionoftheRabbitMQtab.ThisisusefulifyouwanttoconnectdirectlytotheRabbitMQcluster,forexamplewith rabbitmqctl ,ortoconnectotherVMsrunningErlang.

KnownIssueswithErlangCookie

TherearetwoknownissuesassociatedwiththeErlangcookiewhenyoudothefollowing:

ClusterScaling

ChangingtheErlangCookieValue

ClusterScalingKnownIssue

IfyouhavenotsettheErlangcookieandyouwanttoscaleoutyourclustersizewithoutdowntime,followthesesteps:

1. Followthestepsinoneofthelinksbelow,uptoandincludingthesectionLogintotheBOSHDirector:

ForOpsManagerv1.10orearlier:AdvancedTroubleshootingwiththeBOSHCLI

ForOpsManagerv1.11orlater:PreparetoUsetheBOSHCLI

ForOpsManagerv1.10orearlier: bosh ssh rabbitmq-server/0ForOpsManagerv1.11orlater: bosh2 ssh rabbitmq-server/0

3. Runthefollowingcommands:

sudo -iecho $(cat /var/vcap/store/rabbitmq/.erlang.cookie)

4. PastethevaluereturnedfromthelastcommandintotheErlangcookiefieldintheRabbitMQtab.Thisfieldisshownabove.

5. Toincreasethesizeofyourcluster,navigatetotheResourceConfigtaband,inthefirstrow,raisethevalueforthenumberofInstancesoftheRabbitMQnode.

6. ReturntotheOpsManagerInstallationDashboard,andclickApplyChanges.

ChangingtheErlangCookieValueKnownIssue

ChangingtheErlangcookievaluerequiresclusterdowntime.Pivotalstronglyrecommendsthatyoudonotchangeanythingelseduringthistime,becauseitispossiblefortheconfigurationtobeinconsistentlyappliedduringthisprocess.

Thedeploymentmayfailafterthisprocess.Ifso,redeployingfixestheissue.

RabbitMQConfiguration(Optional)Provideafull rabbitmq.config filebypastingitscontentsintheRabbitMQconfigurationfieldintheRabbitMQtab.This rabbitmq.config fileisthenprovidedtoallthenodesinthecluster.

TheinputinthisfieldmustbeBase64encoded.

Forexample,supposeyouwanttoconfigurethe rates_mode ofthe rabbitmq_management statsbelow:

[ {rabbitmq_management, [ {rates_mode, detailed} ]}].

1. EncodethefileintoBase64:

WwogIHtyYWJiaXRtcV9tYW5hZ2VtZW50LCBbCiAgICB7cmF0ZXNfbW9kZSwgZGV0YWlsZWR9CiAgXX0KXS4K

2. PastetheaboveintotheRabbitMQconfigurationfield:

Note:BOSHtellsyouthatthecookiehaschanged—thisisbecausethedefaultvalueinthemanifestisempty,whichresultsinanauto-generatedcookie.However,thevalueofthecookieontheserverremainsthesame,sotheknownissuebelowdoesnotapply.

Youcanseeanexample rabbitmq.config filehere .FormoreinformationabouttheRabbitMQconfiguration,seetheRabbitMQdocumentation .

TLSSupportConfigureTLSinthissectionoftheRabbitMQtab:

TLSv1.0isdisabledbydefault,duetosecurityissues.

TLSv1.1andv1.2areenabledbydefaultandcanbeturnedonandoff.

ExternalLoadBalancer(Optional)EnteraDNSnameorIPaddressofanexternalloadbalancertobereturnedinthebindingcredentials( VCAP_SERVICES )toapplicationdevelopers.EnterthisinthissectionoftheRabbitMQtab:

Ifyouconfigureanexternalloadbalancer,toavoidanunnecessaryVMdeployment,intheResourceConfigtabsettheHAProxyforRabbitMQinstancecountto0.

MetricsPollingIntervalThemetricspollingintervalissetinthissectionoftheRabbitMQtab:

Thedefaultsettingis30secondsforalldeployedcomponents.Pivotalrecommendsthatyoudonotchangethisinterval.Inordertoavoidoverwhelmingcomponents,donotsetthisbelow10seconds.

Changingthissettingaffectsalldeployedinstances.

DiskFreeAlarmLimitChoosehowmuchdiskspaceRabbitMQattemptstokeepfreeatanygiventimeinthissectionoftheRabbitMQtab:

RabbitMQperiodicallychecksifthereissufficientfreespaceondisk.Ifthereisnot,RabbitMQtemporarilystopsacceptingnewmessages.Thisgivesyourappstimetoconsumeexistingmessages,andthusfreeupsomediskspace.TheRabbitMQtileprovidesfouroptionsforthisvalue:

50MBistheminimumvalue.(NotRecommended)Selecting50MBisnotrecommendedandcancausedataloss.Formoreinformation,seeDangersofSettingThisValueTooLowandWhentoUsethe50MBValuebelow.

100%MemoryensuresthatatthetimewhenRabbitMQcheckstheavailabledisk,theremustbeenoughspaceforRabbitMQtopageallmemory-basedmessagesouttodisk.

150%Memoryisrecommended.Thisisbecauseitispossiblethatinbetweendisk-spacechecks,RabbitMQmay:

Writepersistentmessagestodisk(usingupsomediskspace).Acceptmorememory-basedmessagesintovariousqueues.Pageallmemory-basedmessagestodisk.

Intheabovesituations,RabbitMQmightrequiremorefreediskthanithasmemory.

200%MemoryisaconservativevalueusedwhentheoperatorwantshigherconfidencethatRabbitMQneverrunsoutofdiskspace.

Formoreinformationaboutdiskalarms,seetheRabbitMQdocumentation .

DangersofSettingThisValueTooLow

IfthediskofagivenRabbitMQnodecompletelyfillswhileRabbitMQisrunning,thatnodecrashes.Thiscanleadtodataloss,andlossofavailability.

RabbitMQreservestherighttopageanyandallmessagesinmemory(eventransientmessages)todiskatanytime.YoumustsetyourDiskfreealarmlimithighenoughtoensurethatRabbitMQalwayshasatleastenoughspacetodothis.

DisadvantagesofSettingThisValueTooHigh

IfyousetyourDiskfreealarmlimittoavaluelargerthanthesizeofyourpersistentdisk,thenRabbitMQisnotabletofreeupenoughdiskspacetoacceptnewmessages.EnsurethatyouhavealargeenoughdisktopersistallthemessagesyouintendtopersistwhilealsoleavingenoughspacefreetosatisfytheDiskfreealarmlimitthatyouchoose.

WhentoUsethe50MBValue

Pivotaldoesnotrecommendusingthisvalueinproduction.However,ifyouareexperimentingwithadevelopmentenvironmentyoumaywanttouseasmalldisktokeepdowncosts,thoughthisincreasesthepossibilitythatRabbitMQmaycrashandlosedata.

SyslogSpecifyasyslogdestinationintheSyslogtab.

ToenablemonitoringforRabbitMQforPCF,operatorsforwardthesyslogbydesignatinganexternalsyslogendpointforRabbitMQcomponentlogmessages.ThisendpointservesastheinputtoamonitoringplatformsuchasDatadog,Papertrail,orSumoLogic.

TospecifythedestinationforRabbitMQforPCFlogmessages,dothefollowing:

1. FromtheOpsManagerInstallationDashboard,clicktheRabbitMQtile.

2. IntheRabbitMQtile,clicktheSettingstab.

3. ClickSyslog.

4. ConfigurethefieldsontheSyslogpaneasfollows:

Option Description

Syslogaddress IPorDNSaddressofthesyslogserver

Syslogport Portofthesyslogserver

Transportprotocol

Transportprotocolofthesyslogserver.Oneof udp , tcp , relp .

Formatforlogs Formatforlogs.Pivotalrecommends RFC 5424 ,but Legacy Format canbeusedforcompatibilityreasons.

EnableTLS EnableTLStothesyslogserver.

Ifthereareseveralpeerserversthatcanrespondtoremotesyslogconnections,thenyoumayprovideawildcardinthe

PermittedPeer domain,suchas *.example.com .

CustomCACertificate

Iftheservercertificateisnotsignedbyaknownauthority,forexample,aninternalsyslogserver,providetheCAcertificateofthelogmanagementserviceendpoint.

5. ClickSave.

6. ReturntotheOpsManagerInstallationDashboardandclickApplyChangestoredeploywiththechanges.

DedicatedInstance:SingleNodePlanThistabonlyappliestotheon-demandservice.However,youmustcompletethefieldsonthistabevenifyouarenotusingtheon-demandservice.Therefore,ifyouarenotusingtheon-demandservice:

Selectorenteranyvaluesintherequiredfields,selecttheAcknowledgecheckbox,andclickSave.

Forinformationonconfiguringtheon-demandservice,seeInstallingandConfiguringtheOn-DemandService.

Errands(Optional)IntheErrandstab,choosethedefaultsforwhenerrandsrun.

Errandscanbethoughtofastasks.Forexample,whendeployingorupdatingRabbitMQforPCF,OpsManagercanoptionallyrunaseriesofpost-deployerrands.AnexampleistheSmokeTestserrand,whichchecksthehealthoftheRabbitMQclusterafteradeployorupgrade.

Youcandecidewhethertorunpost-deployerrandsbytogglingthemonoroffbeforeyouclickApplyChangestoupdateaconfigurationintheOpsManagerInstallationDashboard:

Thisisaone-timeactionbeforeanupdate.YoucanchangetheabovedefaultsintheErrandstab,aswellasthedefaultsforpre-deleteerrands.

Formoreinformationonerrandrunrules,seeErrandRunRules .

Post-DeployErrands

Errand Description

BrokerRegistrar

Makesthepre-provisionedRabbitMQserviceplansavailableintheMarketplace

SmokeTestsChecksthatapre-provisionedRabbitMQserviceinstancecanbeboundtoaCloudFoundryapp,andthattheappcanpublishandsubscribetoaRabbitMQcluster.SeePre-ProvisionedInstanceSmokeTestsbelow.

RegisterOn-Demand Makestheon-demandRabbitMQserviceplansavailableintheMarketplace.IfyouchangetheServicePlanConfiguration,youmustrun

Important:InRabbitMQforPCFv1.9.0andlater,allpost-deployerrandsareonbydefault.Pivotalrecommendskeepingthesedefaults,becausethesmoketestscanencounterunexpectedissues,andon-demandinstancesofRabbitMQforPCFmayfallbehindiftheUpgradeAllServiceInstanceserrandisnotonbydefault.

ServiceBroker

thiserrandinorderforthechangestobereflectedintheMarketplace.

DedicatedInstanceSmokeTests

Checksthaton-demandRabbitMQserviceinstancescanbeboundtoaCloudFoundryapp,andthattheappcanpublishandsubscribetoaRabbitMQcluster.SeeOn-DemandInstanceSmokeTests.

On-demandinstancesareupdatedandredeployediftherearechangestotheDedicatedInstancesettingsorthetileisupgraded.IfthiserrandissettoOfforWhenChanged,updatestoDedicatedInstancesettingswillnotbeappliedtoexistingserviceinstances.Pivotalstronglyrecommendsthatthiserrandisconfiguredtoalwaysrun.

Pre-ProvisionedInstanceSmokeTestsSmoketestsrunasapost-deploymenterrand.Formoreinformationaboutthesmoketestsprocess,seeSmokeTests.

Pre-DeleteErrandsPre-deleteerrandsrunafteranoperatorchoosestodeleteaproductintheOpsManagerInstallationDashboard,butbeforeOpsManagerfinishesdeletingtheproduct.

Errand Description

BrokerDeregistrarRemovesthepre-provisionedRabbitMQservicefromtheMarketplaceanddeletesallassociatedserviceinstancesandbindings

DeleteAllServiceInstancesUnbindsanddeletesexistingdedicatedserviceinstances.Thedurationofthiserranddependsonthenumberofdeployedon-demandinstances.

DeregisterOn-DemandServiceBroker

Removestheon-demandRabbitMQservicefromtheMarketplace

Stemcell1. ClickStemcell.

2. Ifprompted,importanewstemcellversion.Formoreinformation,seetheinformationaboutimportingthestemcellforyourIaaS:AWS ,Azure

,GCP ,orvSphere .

ApplyConfigurationandCompletetheInstallationReturntotheOpsManagerInstallationDashboardandclickApplyChangestocompletetheinstallationofRabbitMQforPCF.

OtherConfigurationTopics

ConnectingtoaHighlyAvailableRabbitMQClusterTheRabbitMQtile,allowsforahighlyavailableclusterthroughmultipleHAProxynodes.The hostnames , uris and hosts propertieshavebeenaddedandshouldbeusedinpreferenceovertheequivalentsingularproperties.Thesingularpropertiesaremaintainedforbackwardscompatibilityandalwayscontainthefirstvaluefromtheequivalentpluralproperty.Thesingularpropertieswilleventuallybedeprecated.

Forexample,withtwoHAProxyjobsdeployed,thefollowingpropertieswillbepresent:

"hostname": "10.0.0.41","hostnames": [ "10.0.0.41", "10.0.0.51"]

Porttoprotocolmappings15672=Managementdashboard

5672=RabbitMQ

5671=RabbitMQSSL

1883=MQTT

8883=MQTTSSL

61613=STOMP

61614=STOMPSSL

15674=WebSTOMP

4567=RabbitMQServiceBroker

3457-3459=CFLoggregator

SecurityGroupsToenableaccesstotheRabbitMQtileservice,youmustensureyoursecuritygroupallowsaccesstotheHAProxyandRabbitMQServiceBrokerVMsconfiguredinyourdeployment.YoucanobtaintheIPaddressesforthesefromtheOpsManagerStatuspagefortheRabbitMQtile.EnsurethefollowingportsareenabledforthoseVMs:

3457-3459

ThefollowingisatemplateforconfiguringyourCloudFoundrysecuritygroups:[ {"protocol":"tcp","destination":"<haproxy-node-IP-addresses>","ports":"5671,5672,1883,8883,61613,61614,15672,15674"}, {"protocol":"tcp","destination":"<service-broker-node-IP-addresses>","ports":"4567"} ]

ApplicationSecurityGroupsToallowthisservicetohavenetworkaccess,youmustcreateApplicationSecurityGroups (ASGs).

ApplicationContainerNetworkConnectionsApplicationcontainersthatuseinstancesoftheRabbitMQservicerequirethefollowingoutboundnetworkconnections:

Destination Ports Protocol Reason

HAProxy IPs 5672 tcp ApplicationcontainersusingAMQP

HAProxy IPs 5671 tcp ApplicationcontainersusingAMQPoverSSL

HAProxy IPs 1883 tcp ApplicationcontainersusingMQTT

HAProxy IPs 8883 tcp ApplicationcontainersusingMQTToverSSL

HAProxy IPs 61613 tcp ApplicationcontainersusingSTOMP

HAProxy IPs 61614 tcp ApplicationcontainersusingSTOMPoverSSL

Note:TheserviceisunusablewithoutApplicationSecurityGroups.

HAProxy IPs 61613 tcp ApplicationcontainersusingWebSTOMPDestination Ports Protocol Reason

CreateanASGnamed rabbitmq-app-containers withtheaboveconfigurationandbindittoeither:

Theappropriatespace

The default-running ASGsetifyouwanttoprovideaccesstoallstartedapps.Thenrestartyourapps.

Ifyouareusinganexternalloadbalancer,orhavemorethanoneIPaddressforHAProxy,youmustalsocreateegressrulesforthese.Forexample:

[ { "ports": "5671-5672", "protocol": "tcp", "destination": "10.10.10.10/32" }]

AssignedIPsRabbitMQforPCFdoesnotsupportchangingtheIPaddresseswhichhavebeenassignedtotheRabbitMQdeployments.Forexample,youcannotchangethesubnetintowhichtheRabbitMQclusterwasoriginallyprovisioned.Doingsocausesthedeploymenttofail.Formoreinformation,seeChangingNetworkorIPAddressesResultsinaFailedDeployment .

PreservingDynamicallyAssignedIPsYoucannotswitchfromdynamicallyassignedIPaddressestoadifferentsetofstaticIPaddresses.However,youcanconfigureOpsManagersothecurrentsetofdynamicallyassignedIPaddressesalwayscontinuetobeused.Thismaybeusefulwhenupgrading.

Todothis,followthesesteps:

1. GototheStatuspageintheRabbitMQtile.

2. TakenoteoftheIPaddressesfortheRabbitMQServerandHAProxyforRabbitMQjobs,intheordernodesappearintheUI.

3. GototheSettingspage,andclickNetworking.

4. EntertheIPaddressesyougotfromtheStatuspageasacomma-separatedlist.

5. ClickSave.

RabbitMQServerSettingsthatCannotbeOverwrittenInallcases:

rabbit halt_on_upgrade_failure false

rabbitmq_mqtt subscription_ttl 1800000

log_levels [{connection,info}]

halt_on_upgrade_failure false

{rabbit, [ {collect_statistics_interval, 60000}] }

{rabbitmq_management, [ {rates_mode, none}] }

WhenSSLisenabled:

rabbit tcp_listeners []

rabbit ssl_listeners [5671]

rabbitmq_management listener [{port,15672},{ssl,false}]

rabbitmq_mqtt ssl_listeners [8883]

rabbitmq_stomp ssl_listeners [61614]

CreatingIsolationwiththeRabbitMQforPCFReplicator

OverviewRabbitMQforPCFReplicatorisatoolthatallowsyoutoinstallmultipleRabbitMQforPivotalCloudFoundry(PCF)tilesinasingleOpsManagerenvironment.Thisletsyourunmultiplepre-provisionedRabbitMQclustersthatareisolatedfromeachother.

Forexample,youmaywanttoisolatetheclusterservingSpringCloudServices(SCS)fromtheclusterservingappsintheMarketplace.Oryoumaywanttogiveacertainteamtheirowndedicated,pre-provisionedclusterthatyoumanageforthem.Forinformationonhowtoaccomplishthesescenarios,seeCommonUseCases.

CommonUseCasesTheimagebelowillustrateshowtoisolateSCSonRabbitMQforPCFfromclusteredandsinglenodeserviceinstancesdedicatedtodifferentteams.Inthisusecase:

TheunreplicatedRabbitMQforPCFtiledeploystwotypesofservices:

Apre-provisionedserviceisusedasabackingserviceforSCSAnon-demandserviceisusedtocreatethreecompletelyisolatedsinglenodeserviceinstances

Tworeplicatilesareusedtocreatetwodedicatedpre-provisionedclusters,witheachonededicatedtoaspecificteam.

ThesescenariosareexplainedbelowinRunningSCSonaDedicatedRabbitMQClusterandProvidingaPre-ProvisionedDedicatedCluster.

RunningSCSonaDedicatedRabbitMQClusterReplicaRabbitMQtilescannotbeusedtoprovideabackingserviceforSpringCloudServices(SCS)becauseSCSexpectsthattheserviceiscalled

p-rabbitmq .Therefore,ifyouwanttoisolatetheRabbitMQclusterthatisusedbySCSfromothertenants,youcanreservetheunreplicatedRabbitMQforPCFtileforSCS,asshowninthediagrambelow.YoucanthenaddreplicaRabbitMQclustersforusebyappsintheMarketplace.

PivotalrecommendsthatyouusetheunreplicatedRabbitMQforPCFtilesolelyforSCStoavoidcontentionbetweenappsusingSCS,andappsusingRabbitMQforPCFintheMarketplace.

ToreservetheunreplicatedtileforSCS,turnofftheBrokerRegistrarerrandtopreventthebrokerfrombeingexposedintheMarketplace.Formoreinformation,seeErrands .

ToofferRabbitMQasacloudmessagingserviceintheMarketplace,createoneorseveralreplicas,installtheminOpsManager,andeitherallowthebrokerregistrarerrandtorun,orregistertheservicemanuallyusingCF .

ProvidingaPre-ProvisionedDedicatedClusterToreserveaRabbitMQclusterforusebyaspecificteam,allyouneedtodoisdisablethebrokerregistrarerrandinOpsManager .ThispreventsserviceregistrationintheMarketplace.Afteryoudeploythetile,manuallyexposetheservicebrokertoyourdesiredorgsandspaces.Forinstructions,seeRegisteraBroker .

UsingReplicasWhileOfferingtheOn-DemandRabbitMQServiceTheOn-Demandserviceisnotofferedinreplicatiles,sincethepurposeofthereplicatoristocreateadditionalpre-provisionedclusters.Ifyouwishtoofferon-demandserviceplans,usetheunreplicatedRabbitMQforPCFtileasshowninthediagramabove.

Blue-GreenUpgrades(Advanced)Inordertodoblue-greenstyleupgradestominimizedowntime,youcanstandupanewclusterandmigratedataandusersovertothenewclusteroveraperiodoftime.SpeakwithyourPlatformArchitectabouthowtoenablethisworkflow.

GeneratingReplicaTilesThistopicdescribeshowtoinstallthereplicatorandgeneratereplicatilesofRabbitMQforPCF.

PrerequisitesRabbitMQforPCFv1.8.xorv1.9.x

2.5GBoffreediskspace

DownloadtheReplicatorTheRabbitMQforPCFReplicatoriscurrentlyavailablefromPivotalNetwork .Searchforanddownloadthisarchive,andthenruntheenclosedbinary.

GenerateReplicaTilesThefollowingisthesyntaxforgeneratingareplicatile:

./rabbitmq-replicator-darwin\--name YOUR_DESIRED_TILE_NAME\--path PATH_TO_TILE\--output DESIRED_FILE_NAME.pivotal

Thefollowingaretheparametersexpectedintheabovesyntax:

Parameter Description

nameThedesireduniqueidentifierforthereplicatile,whichisusedtogeneratemanifests,deploymentnames,andMarketplacenamefortheservice.Onlyalphanumericcharactersand - areacceptedbythetool.

path Thelocationoftheoriginal,unreplicated,RabbitMQforPCFsourcetilethatyoudownloaded

output Thedesiredfilenameandpathforthereplicatile

NamingConventionsinOriginalandReplicaTilesThetablebelowshowsthenamingconventionsforvariouscomponentsrelatedtotheoriginalRabbitMQforPCFtileandtothereplicatile.

Forthepurposesofthisexample,assumethatwhenyougenerateareplicatileasshownabove,inthe name fieldyouprovidethestring finance\ .Thentheattributesfortheoriginalandreplicatilesareasfollows:

Component NamewithOriginalTile NamewithReplicaTile

Brokername p-rabbitmq p-rabbitmq-finance

BrokerURL pivotal-rabbitmq-broker.YOUR_CF_DOMAIN pivotal-rabbitmq-broker-finance.YOUR_CF_DOMAIN

Servicename p-rabbitmq p-rabbitmq-finance

URLfortheRabbitMQManagementUIDashboard pivotal-rabbitmq.YOUR_CF_DOMAIN pivotal-rabbitmq-finance.YOUR_CF_DOMAIN

TiledisplaynameinOpsManager RabbitMQ RabbitMQ(finance)

TilenameusedinternallybyOpsManager p-rabbitmq p-rabbitmq-finance

Metrics/LoggingOrigin p-rabbitmq p-rabbitmq-finance

InstallingReplicaTilesAfteryouhavegeneratedareplicatile,youcanuploadittoOpsManagerasyouwouldanyothertile.Afteryouhaveuploadedit,followtheinstructionsforInstallingandConfiguringRabbitMQforPCFasaPre-ProvisionedService .TheOn-Demandserviceisnotofferedonreplicatiles.

LimitingAccesstoReplicaTilestoSpecificOrgsWhenyoureplicateRabbitMQforPCF,thereplicatilehastheBrokerRegistrarerrandsettoOnbydefault.ThisfieldappearsintheErrandstabinthetile:

Withanytile,iftheBrokerRegistrarerrandissettoOn,itrunsautomaticallywhenyoufinishinstallingthetileandcausesthetiletobeavailabletoallCForgs.

Ifyouwanttolimitaccesstothetiletoaspecificorg,followthesesteps:

1. SetthebrokerregistrarerrandtoOff,andapplyyourchanges.

2. ManuallyregisterthetilewithaspecificCForgusingthefollowingcommand.Seetheabovetablefor BROKER_NAME , BROKER_URL ,andSERVICE_NAME :

cf create-service-broker BROKER_NAME BROKER_USERNAME BROKER_PASSWORD BROKER_URL

3. Togiveaccesstotheorg,usethefollowingcommandandrepeatforeachadditionalorg:

cf enable-service-access SERVICE_NAME -o ORG_NAME

UpgradingReplicaTilesYoucanupgradereplicatileslikeregulartileswithoneimportantdifference.YoumustgenerateareplicaofthenewerversionoftheRabbitMQforPCFtile,usingthereplicator,andgivethenewreplicathesame name astheexistingreplica.Thisisshownintheexampleworkflowbelow.

ExampleofanIn-PlaceUpgradeofaReplicaSupposeyouusedthereplicatortogenerateareplicaofv1oftheRabbitMQforPCFtile,withthe name trading-team,andyouinstalleditinOpsManager.Hereisthesamplereplicatorcommandyouusedfortheinitialinstallation:

./rabbitmq-replicator-darwin\--name trading-team\--path /download/p-rabbitmq-v1.pivotal\--output /output/p-rabbitmq-v1-trading-team.pivotal

Toupgradetov2,followthesesteps:

1. DownloadthenewRabbitMQforPCFv2.

2. Runthereplicatorcommand,usingthepathtothenewRabbitMQforPCFv2tile,andsupplythesame name ,trading-team,asshownbelow.

./rabbitmq-replicator-darwin\--name trading-team\--path /download/p-rabbitmq-v2.pivotal\--output /output/p-rabbitmq-v2-trading-team.pivotal

3. Afteryouhavethereplicatilep-rabbitmq-v2-trading-team.pivotal,uploadittoOpsManager.Thisupgradesthev1replicatileinplace.

Youcanthenproceedwithupgradingthecluster.

Ifyouwanttodoablue-greenstyleupgrade,seeBlue-GreenUpgrades.

LimitationsTheOn-Demandserviceisnotofferedonreplicatiles.

DeployingtheRabbitMQPre-ProvisionedService

DefaultDeploymentDeployingRabbitMQforPivotalCloudFoundry (PCF)throughOpsManagerwilldeployaRabbitMQclusterof3nodesbydefault.

Thedeploymentincludesasingleloadbalancer haproxy whichspreadsconnectionsonallofthedefaultports,foralloftheshippedpluginsacrossallofthemachineswithinthecluster.

Thedeploymentwilloccurinasingleavailabilityzone(AZ).

Thedefaultconfigurationisfortestingpurposesonlyanditisrecommendedthatcustomershaveaminimumof3RabbitMQnodesand2HAProxynodes

ConsiderationsforthisdeploymentProvidesHAfortheRabbitMQcluster

QueuesmustbejudiciouslyconfiguredtobeHAastheyareplacedononenodebydefault

Customersshoulddecideonwhichpartitionbehaviourisbestsuitedtotheirusecase.Fortwonodes‘automatic’ispreferred

HAProxyisasinglepointoffailure(SPOF)

TheentiredeploymentisinasingleAZ,whichdoesnotprotectagainstexternalfailuresfromfailuresinhardware,networking,etc.

RecommendedDeployment

WerecommendthatRabbitMQisdeployedacrossatleasttwoavailabilityzones.

RabbitMQservernodesshouldbescaledtoanoddnumberandshouldbegreaterthan3.

Replicationofqueuesshouldonlybeusedwhererequiredasitcanhaveabigimpactonsystemperformance.

TheHAProxyjobinstancecountshouldalsobeincreasedtomatchthenumberofAZstoensurethereisaHAProxylocatedineachAZ.ThisremovestheHAProxySPOFandprovidesfurtherredundancy.

Intheabovediagram,youcanseethatyoucannowsufferthefailureofasingleHAProxyandsingleRabbitMQnodeandstillkeepyourclusteronlineandapplicationsconnected.

ItisalsorecommendthatcustomerschoosesanoddnumberofRabbitMQservernodesofthreeormore.

UpgradingtothisdeploymentfromasingleAZdeploymentItisnotpossibletoupgradetothissetupfromthedefaultdeploymentacrossasingleAZ.

ThisisbecausetheAZsetupcannotbechangedoncethetilehasbeingdeployedforthefirsttime,thisistoprotectagainstdatalosswhenmovingjobsbetweenAZs.

UpgradingtothisdeploymentfromamultiAZdeploymentIfyouhavedeployedthetileacrosstwoAZs,butwithasingleHAProxyinstanceyoucanmigratetothissetupasfollows:

1. DeployanadditionalHAProxyinstancethroughOpsManager

2. Neworre-boundapplicationstotheRabbitMQservicewillseetheIPsofbothHAProxysimmediately

3. Existingboundapplicationswillcontinuetowork,butonlyusingthepreviouslydeployedHAProxyIPAddress.Theycanbere-boundasrequiredatyourdiscretion.

ConsiderationsforthisdeploymentRequiresIaaSconfigurationforavailabilityzonesaheadofdeployingtheRabbitMQtile

ApplicationdeveloperswillbehandedtheIPsofeachdeployedHAProxyintheirenvironmentvariables

QueuesmustbejudiciouslyconfiguredtobeHAastheyareplacedononenodebydefault

Customersshoulddecideonwhichpartitionbehaviourisbestsuitedtotheirusecase.For3ormorenodes'pause_minority’ispreferred

AdvancedDeploymentThisdeploymentbuildsupontheaboverecommendeddeployment,sofollowsthesameupgradepaths.

ThisallowsyoutoreplacetheuseofHAProxywithyourownexternalloadbalancer.

YoumaychoosetodothistoremoveanyknowledgeofthetopologyoftheRabbitMQsetupfromapplicationdevelopers.

Advantages

ApplicationdevelopersdonotneedtohandlemultipleIPsfortheHAProxyjobsintheirapplications

Disadvantages

TheloadbalancerneedstobeconfiguredwiththeIPsoftheRabbitMQNodes.Thesewillonlybeknownoncethedeploymenthasfinished.TheIPsshouldremainthesameduringsubsequentdeploymentsbutthereisarisktheycanchange.

UpgradingtothisdeploymentfromtherecommendeddeploymentItispossibletofirstdeploywithmultipleHAProxyjobs,aspertherecommendeddeploymentanddecidedtolateruseyourownexternalloadbalancer.

Thiscanbeachievedwithoutdowntimetoyourapplications.

Thiscanbeachievedasfollows:

1. ConfigureyourexternalloadbalancertopointtotheRabbitMQNodeIPs

2. ConfiguretheDNSnameorIPaddressfortheexternalloadbalancer(ELB)ontheRabbitMQtileinOpsManager

3. Deploythechanges

4. AnynewinstancesoftheRabbitMQserviceoranyre-boundconnectionswillusetheDNSnameorIPaddressoftheELBintheir VCAP_SERVICES

5. AnyexistinginstanceswillcontinuetousetheHAProxyIPaddressesintheir VCAP_SERVICES

6. Phasethere-bindingofexistingapplicationstohavetheirenvironmentvariablesupdated

7. Onceallapplicationsareupdated

8. Reducetheinstancecountofthe HAProxy jobinOpsManagerto1

9. Deploythechanges

Thisapproachworksasanyexistingboundapplicationshavetheir VCAP_SERVICES informationcachedinthecloudcontrollerandareonlyupdatedbyare-bindrequest.

DowngradingfromthisdeploymenttotherecommendeddeploymentIfyouarecurrentlyusinganexternalloadbalancer,thenyoucanmovebacktousingHAProxysinstead.

Youcanachievethisbyfollowingtheabovestepsinreverseorderandre-instatingtheHAProxyjobs.

ResourcerequirementsThefollowingtableshowsthedefaultresourceandIPrequirementsforinstallingthetile:

Product Resource Instances CPU Ram Ephemeral Persistent StaticIP DynamicIP

RabbitMQ RabbitMQnode 3 2 8192 16384 30720 1 0

RabbitMQ HAProxyforRabbitMQ 1 1 2048 4096 0 1 0

RabbitMQ RabbitMQservicebroker 1 1 2048 4096 0 1 0

RabbitMQ BrokerRegistrar 1 1 1024 2048 0 0 1

RabbitMQ BrokerDeregistrar 1 1 1024 2048 0 0 1

RabbitMQ SmokeTests 1 1 1024 2048 0 0 1

RabbitMQ RabbitMQon-demandbroker 1 1 1024 8192 1024 0 1

RabbitMQ RegisterOn-DemandServiceBroker 1 1 1024 2048 0 0 1

RabbitMQ DeregisterOn-DemandServiceBroker 1 1 1024 2048 0 0 1

RabbitMQ DeleteAllServiceInstances 1 1 1024 2048 0 0 1

RabbitMQ UpgradeAllServiceInstances 1 1 1024 2048 0 0 1

Notes:Thenumberof RabbitMQ Node canbeincreasedifrequired.

ChangingthenumberofRabbitMQnodeswhentheerlangcookieisnotdefinedwillrestartthecluster.Checkhereformoreinformation.

SettingDefaultPoliciesfortheRabbitMQService

UnderstandingaRabbitMQPolicyYoucansetadefaultqueueandanexchangepolicyintheRabbitMQforPivotalCloudFoundry(PCF)tile,tobeappliedtotheirRabbitMQcluster.Afteryoudeploythetile,PivotalrecommendsthatyouusetheRabbitMQManagementInterfacetomakeconfigurationchanges.

FormoreinformationaboutRabbitMQpolicies,seetheRabbitMQdocumentation .

RulesforPoliciesSetintheTileThefollowingrulesapplytopolicessetthroughtheRabbitMQforPCFtile:

Anewpolicy,oranupdatetoapolicy,onlyappliestonewinstances.Existinginstancesarenotaffectedbythepolicy.

ThepolicycanonlybedeletedmanuallyfromtheRabbitMQnodes.

PoliciescanbeaddeddynamicallyusingtheRabbitMQManagementInterface.Forgranularpolicysettings,PivotalrecommendsusingtheRabbitMQManagementUI.Seta priority number lowerthan 50 ,thedefault priority number appliedthroughtheOpsManagerconfiguration.

AnExamplePolicy:MirroronTwoNodesHereisanexamplepolicythatensuresmessagesaremirroredontwonodes:

{ "ha-mode": "exactly", "ha-params": 2, "ha-sync-mode": "automatic"}

Operatorsshouldconsidersomeoftheperformanceimplicationsofmakingqueuesandexchangeshighlyavailable.Formoreinformationabouthighlyavailablequeues,seetheRabbitMQdocumentation .

BestPracticeforSyncingQueuesWhenaqueuesyncsallitsmessages,theyareloadedintomemory.Whenqueuesaresyncing,theycanuseasmuchmemoryasthetotalsizeofallmessages.Thisappliestobothnodes—thenodewherethequeueleaderruns(fromnode)andthenodewherethequeuefollowerruns(tonode),butonlyappliestonewlycreatedqueuefollowers.

Thisbehaviorisespeciallyrelevantwhenanychangeaffectsthedeployment,forexample:stemcellupdates,deploymentconfigurationchanges,andnetworkchanges.Verifythatyouhaveenoughmemoryanddiskavailabletosupportallmessages.

Forexample:

Thereare5GBofmessagesinamirroredqueuethatissettoautomaticsync.Whenthisqueueneedstosync,thenodewherethequeueleaderrunscanuseupto5GBofextramemory.Thesameappliestothenodewherethenewqueuefolloweriscreated.

SettingorChangingthePolicyTosettheRabbitMQpolicy,dothefollowing:

1. FromtheOpsManagerInstallationDashboard,clicktheRabbitMQforPCFtile,thenclickRabbitMQPolicy.

2. SelectEnablecustompolicyonnewinstances.

3. InthePolicyfornewinstancesfield,pastethepolicy.ThepolicymustbevalidJSONandshouldmeetvalidRabbitMQpolicycriteria.

4. IntheSelectthenetworkpartitionbehavioroftheRabbitMQcluster,choosethedesiredbehavior:pause_minorityorautoheal.

Formoreinformationabouttheseoptions,andRabbitMQclustersandnetworkpartitions,seetheRabbitMQdocumentation .

Forproductionpurposes,PivotalrecommendsthatcustomershaveatleastthreeRabbitMQservernodesandtwoHAProxiesspreadacrosslowlatencyavailabilityzones.

ViewingPoliciesintheRabbitMQManagementDashboardYoucanviewRabbitMQpoliciesintheRabbitMQManagementDashboard,shownbelow.TheexamplepolicyweenteredintheRabbitMQforPCFtileaboveisappliedtoallqueuesandgivenaPriorityof50.Thisallowsyoutooverrideitbydefininganotherpolicywithahigherpriority.

IntheQueuessectionshownbelow,youcanseethatanynewqueuescreatedhavethepolicyautomaticallyapplied.

Note:Nopolicyvalidationoccursduringthedeployment,anderrorscancausethedeploymenttofailorpoliciestobeappliedincorrectly.

Note:DeveloperscanobtaintheURLofthepolicyfrom VCAP_SERVICES forapplicationdevelopers.

MonitoringandKPIsforPre-ProvisionedRabbitMQforPCFThistopicexplainshowtomonitorthehealthofthepre-provisionedversionoftheRabbitMQforPivotalCloudFoundry(PCF)serviceusingthelogs,metrics,andKeyPerformanceIndicators(KPIs)generatedbyRabbitMQforPCFcomponentVMs.

Pre-provisionedRabbitMQforPCFcomponentsgeneratemanyofthesamemetricsastheon-demandRabbitMQservicecomponents.

SeeLoggingandMetrics forgeneralinformationaboutloggingandmetricsinPCF.

SettingupSyslogForwardingOperatorscanenablelogforwardingbyconfiguringanexternalsyslogendpointforRabbitMQcomponentlogmessages.Forinstructionsonsettingupsyslogforwarding,seeSyslog.

Ifsyslogforwardingisenabled,logentrieswithtimestampsareavailablelocallyin /var/log/messages .Logsareavailableunder /var/vcap/sys/log/ whethersyslogforwardingisenabledornot.

LoggingFormatsWithpre-provisionedRabbitMQforPCFloggingconfigured,threetypesofcomponentgeneratelogs:theRabbitMQmessageservernodes,theservicebroker,andHAProxy.IfyouhavemultipleserverorHAProxynodes,youcanidentifylogsfromindividualnodesbytheirindex,whichcorrespondstotheindexoftheRabbitMQVMinstancesdisplayedinOpsManager:

ThelogsforRabbitMQservernodesfollowtheformat [job=rabbitmq-server-partition-GUID index=X]

ThelogsforHAProxynodesfollowtheformat [job=rabbitmq-haproxy-partition-GUID index=X]

ThelogsfortheRabbitMQservicebrokerfollowtheformat [job=rabbitmq-broker-partition-GUID index=X]

RabbitMQandHAProxyserverslogatthe info levelandcaptureerrors,warnings,andinformationalmessages.