Upload
others
View
59
Download
0
Embed Size (px)
Citation preview
• Cluster:Ø Aclusterisagroupofindependent,butinterconnectedcomputersthatactsasasinglesystem.Ø Deployedtoincreasetheavailability,performanceandtobalanceadynamicallychangedworkloadottobalancetheloadacross
thesystems.Ø Mostclusterarchitecturesuseadedicatedinterconnectnetworkforcommunica=onandcoordina=onbetweenclusternodes.It
alsousesshareddiskstoragearchitecutre.• Clusterware:
Ø Cluterwareisaso@warethatprovidesvariousinterfacesandservicesforacluster.Typically,thisincludescapabili=esthat:Ø Allowtheclustertobemanagedasasingleen=ty.Ø Protecttheintegrityoftheclustersothatdataisprotectedandtheclustercon=nuestofunc=onevenifcommunica=onwitha
clusternodeissevered.Ø Maintainaregistryofresourcessothattheirloca=onisknownacrosstheclusterandsothatdependenciesbetweenresourcesis
maintained.Ø Dealwithchangestotheclustersuchasnodeaddi=ons,removals,orfailures.Ø Providesacommonviewofresourcessuchasnetworkaddressandfilesinafilesystem
• OracleClusterware:Ø OracleClusterwareprovidesabovecapabili=esandit’sakeypartofaGridInfrastructureandintegratedwithOracleAutoma=c
StorageManagement(ASM)andASMClusterFilesystem(ACFS).Ø Comina=onofOracleClusterware,ASMandACFSprovidesfounda=onforOracleRealApplica=onClusters(RAC)database.Ø Servicesprovidedbyoracleclusterwareinclude:
§ ClusterManagement:Allowsclusterservicesandapplica=onresourcestobemonitoredandmanagedfromanynodeincluster.
§ NodeMonitoring:Providesreal-=meinforma=onregardingwhichnodesarecurrentlyavailableandtheresourcestheysupport.Clusterintegrityisalsoprotectedbyevic=nforfencingunresponsivenodes.
§ EventServices:Publishesclustereventssothatapplica=onsareawareofchangesinthecluster.§ TimeSynchroniza=on:Synchronizesthe=meonallnodesofthecluster.§ NetworkManagement:ProvisionsandmanagesVirtualIP(VIP)addressesthatareassociatedwithclusternodesto
provideaconsistentnetworkiden=tyregardlessofwhichnodesareavailable.Inaddi=on,GridNamingService(GNS)managesnetworknamingwithinthecluster
§ HighAvailability:Services,monitorsandrestartsallotherresourcesasrequired.
• OracleClusterwareStartup:Ø OracleclusterwareisstartedbytheOSinitdaemon.Ø Oracleclusterwareinstalla=onmodifiesthe/etc/iniSabfiletorestartohasdintheeventofacrash.Ø OracleHighAvailabilityServicesdaemon(ohasd)isresponsibleforstar=nginproperorder,monitoring,andrestar=ng
otherlocaloracledaemons,upthroughthecrsddaemon,whichmanagesclusterwideresources.
Ø OracleClusterwareProcess:Ø CRS(CRSdaemon):
§ Managesclusterresourcesbasedonconfigura=oninforma=onthatisstoredinOracleClusterRegistry(OCR)foreachresource.Thisincludesstart,stop,monitorandfailoveropera=ons.crsdgenerateseventswhenthestatusofaresourcechanges.
§ Whenaresourcefailesitwilltrytorestartandifconfiguredwillgailovertoanothernodea@erexhaus=ngrestartaSempts.
§ ItisresponsibletomaintaintheOCR.§ CRSDprocesswritestotracefilesin$GRID_HOME/log/<node-name>/crs/crsd.log
Ø Clustersynchroniza=onService:§ Managestheclusterconfigura=onbycontrollingwhichnodesaremembersoftheclusterandbyno=fyingmembers
whenanodejoinsorleavesthecluster.§ Itmasterthefirstac=venodeandothernodesintheclusteraresynchronizedbyregisteringwithmasternode.§ Itwritestracefiletoin$GRID_HOME/log/<node-name>/css/cssd.log§ CSShasthreeseparateprocess:theCSSdaemon(ocssd),theCSSAgent(cssdagent),andtheCSSMonitor(cssdmonitor).
1. cssdmonitor=>Monitorsnodehangs(viaoprocdfunc=onality)andmonitorsOCCSDprocesshangs(viaoclsomonfunc=onality)andmonitorsvendorclusterware(viavmonfunc=onality).Thisisthemul=threadedprocessthatrunswithelavatedpriority.
Startupsequence:INIT-->init.ohasd-->ohasd-->ohasd.bin-->cssdmonitor2. cssdagent=>SpawnedbyOHASDprocess.Previously(10g)oprocd,responsibleforI/Ofencing.Killingthisprocess
wouldcausenodereboot.Stops,startchecksthestatusofoccsd.bindaemonStartupsequence:INIT-->init.ohasd-->ohasd-->ohasd.bin-->cssdagent
3. occsd.bin=>Managesclusternodemembershiprunsasoragriduser.Failureofthisprocessresultsinnoderestart.
Startupsequence:INIT-->init.ohasd-->ohasd-->ohasd.bin-->cssdagent-->ocssd-->ocssd.binØ EventManager(EVM):
§ ProcessthatpublishesOracleClusterwareevents.Ithastwoprocessevmdandevmlogger.1. evmd.bin=>Distributesandcommunicatessomeclustereventstoalloftheclustermemberssothattheyare
awareoftheclusterchanges.2. evmlogger.bin=>StartedbyEVMD.binreadstheconfigura=onfilesanddetermineswhateventstosubscribeto
fromEVMDanditrunsuserdefinedac=onsforthoseevents.
Ø DiskMonitor(DiskMonitorDaemon):§ Monitorsandperformsi/ofencingforOracleExadatastorageserver.Diskmondaemonisalwaysstartedwhen
ocssdisstarted.Ø Mul=castDomainnameservice(mDNS):
§ UsedbyGridPlugandPlaytolocateprofilesinthecluster,aswellasbyGNStoperformnameresolu=on.ThemDNSprocessisabackgroundprocessonLinuxandUNIXandonWindows.
Ø GridNamingService(GridNamingServiceDaemon):§ IsagatewaybetweentheclustermDNSandexternalDNSservers.TheGNSprocessperformsnameresolu=on
withintheclusterØ OracleNo=fica=onService(ONS):
§ It’sapublishandsubscribeserviceforcommunica=onFANevents.Ø Oraagent:
§ ExtendsclusterwaretosupportOracle-specificrequirementsandcomplexresources.ThisprocessrunsservercalloutscriptswhenFANeventsoccur.ThisprocesswasknownasRACGinOracleClusterware11gRelease1(11.1).
§ ohasd'soraagent:o Performsstart/stop/check/cleanac=onsforora.asm,ora.evmd,ora.gipcd,ora.gpnpd,ora.mdnsd.
§ Crsd'soraagent:o Performsstart/stop/check/cleanac=onsforora.eons,ora.ons,SCANlistenersandlistener.o Performsstart/stop/check/cleanforservice,databaseanddiskgroupresources.o ReceiveseONSevents,andtranslatesandforwardsthemtointerestedclients.
Ø Clusterkilldaemon(oclskd):§ Handlesinstance/nodeevic=onsrequeststhathavebeenescalatedtoCSS.
Ø GridIPCdaemon(gipcd):§ Isahelperdaemonforthecommunica=oninfrastructure.
Ø Orarootagent:§ Aspecializedoraagentprocessthathelpscrsdmanagesresourcesownedbyrootsuchaasthenetwork,andthegrid
virtualipaddress.Ø ClusterTimeSynchroniza=onservice(CTSS):
§ ProvidesTimeManagementinaclusterforOracleClusterwareØ GridPlugandPlay(GPNPD):
§ ProvidesaccesstotheGridPlugandPlayprofile,andcoordinatesupdatestotheprofileamondthenodesoftheclustertoensurethatallofthenodeshavethemostrecentprofile.
Ø SystemMonitor(Osysmond):§ Themonitoringandopera=ngsystemmetriccollec=onservicethatsendsthedatatoclusterloogerservice.The
servicerunsoneverynodeinacluster.• ClusterwareComponents:
Ø PrivateInterconnect:§ usedforcommunica=onbetweentheinstances.(fortransferringofblocksacrosstheinstances)§ Itisrecommendedtoisolateprivateinterconnecttousedifferentswitchasitrequiresfastdatatransfer§ privateinterconnectmustbeondifferentsubnetthanpublicandvirualip.
Ø VirualIP:§ EachDatabasenodeinRACenvironmenthasonenodeipandonenodeVIPaddress.Themaindifferencebetween
thesetwoisnodevipscanmovetoanyothersystemincaseifcurrentworkingsystemiddownbutnodeIPcan'tdothat.Wheneverclientini=atesconnec=onusingSCANnamescanlistenerreplieswiththeleaseloadednode-VIP.
Ø Vo=ngdisk:§ Vo=ngdiskisusedforpollingmechanismwhereeachandeverynodeintheclusterreportsishealthbypollingthe
disk.§ Italsoarbitratesclusterownershipamongtheinstancesincaseofnetworkfailures.§ Togetvotediskloca=on:
$]crsctlquerycssvotedisk.
Ø OCR(OracleClusterRegistry):
§ OCRisusedtomaintainclusterresourcesasitstorestheinforma=onaboutclusterresourcesaswellastheresourcesdefinedinthecluster.
§ Registryisafilestoredinsharedstorage.§ EachnodemaintainsacopyofOCRinmemoryonlyoneprocessisdesignatedasmasterprocess§ ReadfromtheOCRisperformedbymasterOCRprocessandrefreshthelocalcopieswhereaswritetoOCRis
performedbythelocalOCRprocess.§ CRSDmasterautoma=callycreatesOCRbackupseveryfourhours,andCRSretainsthelastthreecopies.The
CRSDprocessalsocreatesanOCRbackupatthebeginningofeachdayandofeachweek,andretainsthelasttwocopies.
§ OCRCHECK:--DisplayshealthofOCR(OracleClusterRegistry).§ Togettheinforma=onaboutOCRdisks$]ocrcheck§ Togettheinforma=onaboutOLR(11gR2)
$]ocrcheck-local-config§ OLRresidesoneverynodeintheclusterandmanagesOracleClusterwareconfigura=oninforma=onforeach
par=cularnode§ Tolistocrbackup's$]ocrconfig-showbackup---listsallautoandmanualbackup§ Totakemanualbackup
$]ocrconfig-manualbackup---in11g§ Totakeanexportofocrfile
$]ocrconfig-export<dumpfile>§ TorestoreOCRfrombackup
$]ocrconfig-restore<backupfile>$]ocrconfig-import<dumpfile>
§ Tochangetheautobackuploca=on$]ocrconfig-backuploc<newloca=on>
Ø SCAN:• SingleClientAccessName(SCAN)issanewOracleRealApplica=onClusters(RAC)11gRelease2featurethat
provides• asinglenameforclientstoaccessanOracleDatabaserunninginacluster.• ThebenefitisclientsusingSCANdonotneedtochangeifyouaddorremovenodesinthecluster.Havingasingle
nametoaccesstheclusterallowsclientstousetheEZConnectclientandthesimpleJDBCthinURLtoaccessanydatabaserunningintheclustersindependentlyofwhichserver(s)intheclusterthedatabaseisac=ve.
• SCANprovidesloadbalancingandfailoverofclientconnec=onstothedatabase.• TheSCANworksasanIPaliasforthecluster.• SCANis=edtomaximumof3ipaddressanditcanbeconfiguredinDNSorGNS.Itworksinroundrobin,.• EachSCANIP(SCANVIP)hasaSCANLISTENERassociatedwithit.Ifit’sa4nodeRACcluster3SCANLISTENERS
runningon3SCANips'willrunon3nodes.• AlldatabasesareregisteredwitheachSCANLISTENERintheClusterandPMONupdatesit'sloadtoeachSCAN
LISTENER.EachrequestgothroughusingSCAN_NAME,resolvestoSCANVIPi.e.SCANLISTENER.Now,SCANLISTENERredirectsittoVIPbydecidingusingLoadBalance.
Databaseclient
DNSGNS
Scanlistener
Scanlistener
Scanlistener
Listener
Listener
Rac-scan
192.168.56.211
DatabaselistenerrequestVIPofleast-loadednode
Ø SharedStorage:§ Datafiles,controlfiles,redologfiles,spfile,OCRandVo=nddiskmustbepresentinthesharedstorage.§ From11gASMisthegotofilesystemforsharedstorage.
Ø GPnPPofile:§ GPnPProfileisanXMLfilewhichcontainsinforma=onaboutclustername,Networkclassifica=on,storageto
beusedforASManddigitalsignatureofthecluster.• HowCSSD,CRSandASMcomesupifOCRandVo=ngdiskisstoredinASMdiskgroup:
• RACDatabase:Ø SharedResources:
§ Datafiles§ Controlfiles§ Temptablespaces§ Vo=ngdiskandOCR§ RacInterconnect§ VIP's
Ø Non-sharedresources:§ Onlineredologfiles§ UndoTablespace§ Archivelogfiles§ SGA§ Backgroundprocess§ Alertlog,listener.ora,tnsnames.oraandpfile§ Traceandauditfiles.
Ø Interconnect-dedicated:§ Internodecommunica=on§ Heartbeats§ Datablocks.
Ø CacheCoherency:§ Maintainingconsistencyofdatablocksinthebuffercacheofmul=pleinstancesiscalledcachecoherency.
Ø CacheFusion:§ Cachefusionisamechanismbywhichblocksaretrasferredfromoneinstancetoanotherinstanceviaprivate
interconnect.
• RACspecificSGAcomponent:Ø GRD(GlobalResourceDirectory)
§ Itispartofthesharedpool,MaintainedbyGES(GlobalEnqueservices)andGCS(GlocalCacheservice)§ wheneverablockistransferedoutoflocalcachetoanotherinstance'scacheGRDisupdated§ itholdsthemetadataaboutdatablocks,thatareavailbleinDBBC§ itholdsinforma=onlike
1. SCN(systemchangenumber)2. DBI(datablockiden=fier)3. loca=onoftheblock4. modeoftheblock
I. null(N)-Nullmodeistheleastrestric=vemode.Itindicatesnoaccessrights.actsasaplaceholder.II. shared(S)-Sharedmodeindicatethatdatabaseblockisbeingreadandnotmodified.However
anothersessioncanreadthedatablockIII. exclusive(E)-Exclusivemodeindicateexclusiveaccesstoblock.Otherresourcecannothavewrite
overthisdatablock.Howeveritcanhaveconsistentreadonthisdatablock.5. roleoftheblock
I. local-Whenadatablockisfirstreadintotheinstancefromthediskithasalocalrole.Meaningthatonly1copyofdatablockexistsinthecache.Nootherinstancecachehasacopyofthisblock.
II. Global-Globalroleindicatesthatmul=plecopyofdatablockexistsinclusteredinstance.Forexampleauserconnectedtooneoftheinstancerequestforadatablock.Iftheblockislocalinoneinstanceandrequestedbyanotherinstancethenblockistransferredtothereques=ngnodeandtherolewillbeupdatedasglobal.
6.typesofdatablockimagei. currentimage-updatedatablockvalueii. consistentimage-previousdatablockvalueiii. pastimage-grdupdatedimage
§ itconverttocurrentimagewheninstanceiscrash
• RACDatabasespecificservices:Ø GlobalCacheService:
§ GlobalCacheService(GCS)isthemaincomponentofOracleCacheFusiontechnology.ThisisrepresentedbybackgroundprocessLMSn.Therecanbemax10LMSprocessforaninstance.Themainfunc=onofGCSistotrackthestatusandloca=onofdatablocks.Statusofdatablockmeansthemodeandroleofdatablock(Iwillexplainmodeandrolefurther).GCSisthemainmechanismbywhichcachecoherencyamong“mul=plecache”ismaintained.GCSisalsoresponsibleforblocktransferbetweentheinstances.
Ø GlobalEnqueueService:§ TheGESisprimarilyresponsibleformaintainingcoherencyinthedic=onaryandlibrarycaches.Thedic=onary
cacheconsistsofthedatadic=onarymasterinforma=onforeachnodeinitsSGA(SystemGlobalArea)primarilyforquickerlookupandaccess.AnyDDLcommiSedfromareques=ngnodeneedstobesync’edandwriSenacrossalldatadic=onariesinallnodesoftheRACenvironment.TheGESmakessurethatthechangesremainconsistentacrossthenodesandthattherearenodiscrepancies.Moreover,withthesamedirec=ve,thelocksmustbecreatedandmaintainedacrossthenodesandGESmustensurethattherearenodeadlocksacrossreques=ngnodesoveraccesstothesameobjects.LMON,LCKandLMDprocessesworkintandemtomaketheGESoperateinasmoothandseamlessfashion.
• RACDatabaseBackgroundProcess:Ø LMS(GlobalCacheServiceProcess):
§ Maintainsrecordsofdatafilestatusesandeachcachedblockbyrecordinginforma=oninaGRD.§ Controlstheflowofmessagestoremoteinstanceandmanagedglobaldatablockaccessandtransmitblock
imagesbetweenbuffercachesofdifferentinstances.Ø LMON(GlobalEnqueueServiceMonitor:
§ Detectsinstancetransi=ons(joinsorleavesthecluster)andperformsreconfigura=onofGECandGCS.§ Monitorsglobalenqueuesandresourcesacrosstheclusterandperformglobalenqueuerecoveryopera=ons§ Responsibleforexecu=ngDynamiclocalremasteringevery10mins.
Ø LMD(GlobalEnqueueserviceDaemon):§ Processesincomingenquerequestmessagesandcontrolsaccesstoglobalresources.§ Performsdistributeddeadlocksdetec=ons.
Ø LCK0(InstanceEnqueuProcess):§ Managesnon-cachefusionresourcerequestssuchaslibraryandrowcacherequests.
Ø LMHB:§ MonitorsLMON,LMDandLMSnprocesstoensuretheyarerunningnormallywithoutblockingorspinning.
• DynamicResourcereconfigura=on:
Ø Whenoneinstancedepartsthecluster,theGRDpor=onofthatinstanceneedstoberedistributedtothesurvivingnodes.Similarly,whenanewinstanceentersthecluster,theGRDpor=onsoftheexis=nginstancesmustberedistributedtocreatetheGRDpor=onofthenewinstance.Thisiscalleddynamicresourcereconfigura=on.
• DynamicResourceremastering:
Ø InRACeverydatablockismasteredbyaninstance.Masteringablocksimplymeansthatmasterintstancekeepstrackofthestateoftheblockun=lthenextconfigura=oneventoccurs.GCSkeepstrackofthenumberofGCrequestsonaper-instanceandper-objectbasis.Thismeansthatifaninstance,comparedtoanotherisheaviliyaccessingblocksfromthesameobjects,theGCScantakethedecisiontodynamicallymigrateallofthatobject'sresourcestotheinstancethatisaccessingtheobjetmostandmakingitmasterofthoseblocks.LMON,LMDandLMSprocessesareresponsiblefordynamicremastering.
Ø Remasteringcanbetriggeredasaresultof:§ Manualremastering§ Resourceaffinity(objectaffinity,undoaffinityandfilesaffinity)§ Instancecrash.
Ø Tofindthecurrentmasterandpreviousmasterofaresource:SQL>selecto.object_name,m.CURRENT_MASTER,m.PREVIOUS_MASTER,m.REMASTER_CNTfromdba_objectso,v$gcspfmaster_infomwhereo.data_object_id=73181andm.data_object_id=73181;SQL>selectkj.kjblname,kj.kjblname2,kj.kjblowner,kj.kjblmasterfrom(selectkjblname,kjblname2,kjblowner,kjblmaster,kjbllockpfromx$kjblwherekjblname='[0x97][0x4],[BL]')kj,x$lelewherele.le_kjbl=kj.kjbllockporderbyle.le_addr;
• Workingofcachefusion:INSTANCE1 INSTANCE2 INSTANCE3
Lockheld:XGO-------------------ExclGlobal
Nopastimage
Lockheld:none
Lockheld:NG1-------------------NullGlobalWith1past
image
Lockheld:none
Lockheld:SL0-------------------SharedLocalNopastimage
Lockheld:none
Lockheld:none
Lockheld:none
Lockheld:XLO-------------------ExclLocal
Nopastimage
Lockheld:none
Lockheld:SL0-------------------SharedLocalNopastimage
Lockheld:SL0-------------------SharedLocalNopastimage
Instage1datablockisrequestedbyauserCwhoisconnectedtoinstance3.Soadatablockisreadintothebuffercacheofinstance3.SQL>selectsalfromempwhereempno=10;àreturns300
Instage2samedatablockisrequestedbyauserBwhoisconnectedtoinstance2.Astheblockisalreadypresentininstance3itwillbeshippedtoinstance2viaprivateinterconnectusingcachefusion.SQL>selectsalfromempwhereempno=10;àreturns300
Instage3userBwhoisconnectedtoinstance2triestoupdatethesalofemployeeforempno10.Atthisstage,instance2acquiresEXCLUSIVElockforupda=ngthedataatinstance2andSHAREDlockfrominstance3isdowngradedtoNULLlock.roleoftheblockiss=lllocalasitisdir=edforthefirst=meonlyoninstance2SQL>updateempsetsal=400whereempno=10;commit;
Instage4userAwhoisconnectedtoinstance1triestoupdatethesalofemployeeforempno10.Datablockisshippedfrominstance2toinstance1andthelockoninstance2isdowngradedtonullandroleisglobalandhas1pastimage.instance1willhavexclusivelocalwithnopastimageSQL>updateempsetsal=500whereempno=10;
Lockheld:none
Lockheld:XLO-------------------ExclLocal
Nopastimage
Lockheld:none
Lockheld:SG1-------------------SharedGlobalWith1past
image
Lockheld:SG0-------------------SharedGlobalNopastimage
Lockheld:NG1-------------------NullGlobalWith1pastimage
Lockheld:NG1-------------------NullGlobalWith1past
image
Lockheld:XGO-------------------ExclGlobal
Nopastimage
Lockheld:NG1-------------------NullGlobalWith1past
image
Lockheld:SG1-------------------SharedGlobalWith1past
image
Lockheld:SG0-------------------SharedGlobalNopastimage
Lockheld:SG1-------------------SharedLocalWith1past
image
Instage5usercconnectedtoinstance3triedtoselectthesalofempno10.Datablockwillbecopiedfrominstanc1toinstance3andthelockoninstance1andinstance3willbesharedasinstance3requestedsharedlockifinstance3requestedxclusivelockthenoninstance3lockwillbenullSQL>selectsalfromempwhereempno=10;àreturns500
Instage6UserBissuesthesameselectstatementagainsttheemptablewithempno10oninstance2.Instance2willrequestforaconsistentcopyofbufferfromanotherinstance,whichhappenstobethecurrentmaster.Thereforeinstance1willshiptheblocktoinstance2,whereitwillberequiredwithSG1(SHARED,GLOBALwith1PASTIMAGE).Soinstance2modebecomesSG1.
Instage7UserConinstance3updatesthesamerow.Thereforetheinstance3requiresanexclusivelockandinstance1andinstance2willbedowngradedtoNULLlockwithGLOBALroleand1PASTIMAGE.Instance3willhaveEXCLUSIVElock,GLOBALroleandwithnoPASTIMAGES(XG0).
Instage8Thecheckpointisini=atedanda“WritetoDisk”takesplaceatinstance3.Asaresultpreviouspastimageswillbediscarded(astheyarenotrequiredforrecovery)andinstance3willholdthatblockinEXCLUSIVElockLOCALrolewithnoPASTIMAGES(XL0).
• RACAdministra=on:Ø Totroubleshootclusterstartup/clusterhealthissues:
§ ReviewthealertlogofGridinfrastructureunder$GRID_HOME/log/<node-name>/alert<node>.log§ Reviewtheclusterbackgroundprocessspecificlogfilesunder$GRID_HOME/log/<node-name>/<process>/
<process-name>.log§ Reviewtheocrchecklogfileunder$GRID_HOME/log/<node-name>/client/ocrcheck_<=mestamp>.logfile§ Nodeevi=onsissuesmightincludesomeaddi=onalclusterwarelogfiles.
/etc/oracle/lastgasp/*or/var/opt/oracle/lastgasp/*§ Iftheabovelogfilesdoesn'tprovidesufficientinforma=ontotroubleshoottheissuewecanenablethetrace/
debugsezngstogetthedetailinforma=on.Usebelowcommandstosetthedebug/tracelevels:$]crsctlgetlogcss/crs/evm/all-àgetthecurrentlevelofthetrace$]crsctlsetlogcrscrsmain=4$]crsctlsetlogcrsall-=3-àyoucansetthelevelfrom1to5higherthevaluehigherthedetailsprovided.$]exportSRVM_TRACE=TRUE
§ CVU(ClusterVerifica=onu=lityisusedtocollectpreandpostclusterconfigura=ondetailsatvariouslevelsandvariouscomponents.Italsoprovidestheabilitytoverifytheclusterhealth.
$]cluvfycomphealthcheck–collectcluster|database$]cluvfycompnodecon–n<node1>,<node2>--toverifynetworkconnec=ons$]cluvfystage-precrsinst–n<node1>,<node2>---toverifyprechecks
§ ClusterDiagnos=cCollec=onTool(diagcollec=on.sh):Asclustermanagesseverallogfilesitmaybe=meconsumingtocollectthefilesmanually.Diagcollec=onu=lityrefersvariousclusterlogfilesandgathersrequiredinforma=ontodiagnosecri=calclusterproblems
$]diagcollec=on.sh–collect–crs$GRID_HOME§ Loga=cketinmyOraclesupporttoreviewthefilesandprovidethecausefortheissue.§ LookaterrorreportinAIX(errpt–a)or/var/log/messages(linux)or/var/adm/messages(Solaris)or/var/adm/
syslog/syslog.log(HP-UX)toseeanyerrorsreportedattheoslevel
• InterconnectTroubleshoo=ng:Ø LookforlostblocksinAWRreports.Ø Lookforerrorsino/pofifconfig–aØ Lookforpacketreassemblesinnetstat-s
• Nodeevic=ons:Ø CSSDmonitorsnodesusing2communica=onchannels:
§ PrivateinterconnectónetworkHeartbeat§ Vo=ngDiskbasedcommunica=onóDiskHeartbeat.
Ø ForN/Whearbeatnodesmustrespondincssd_misscount=me(defaultis30seconds)Ø Lookfor"heartbeatfatal"incssd.logfile.Ø N/Wheartbeatfailuresresultinnodeevic=ons.Ø Fordiskheartbeateachnodeintheclusterpings(r/w)thevo=ngdiskseverysecond.Ø Nodesmustreceiveresponsein(long/short)diskTimeout=me.Ø Diskheartbeatfailureswillleadtonodeevic=onssearchfor"DskHeartbeat"inocssd.logfile.
• CommandstomanageClusterresources:Ø Tofindthemasternodeinthecluster:
§ Method#1)o gotoGRID_HOME/log/<node-name>/cssdo catocssd.log|grep-i'masternode'|tail-1---thiswillshowus#ofnodesintheclusterlocalnode#and
masternode#§ Method#2)
o CRSDinMasternodeisresposibleforperformingautobackupofocr,solookingatOCRbackupwecaniden=fythemasternode
o ocrconfig-showbackupauto---ListthenodeonwhichOCRwasbackedup
Ø TochecktheCRSstatusonallthenodes:$]crsctlcheckcluster–all$]crsctlstatusserver–f
Ø Tostop/startCRSonallthenodes(stopseverythingexceptohasd)$]crsctlstop/startcluster–all
Ø Tostop/startcrsononenode(includingohasd)$crsctlstop/start/checkcrs
Ø Tocontrolscanlistener:$]srvctlstop/status/startscan_listener$]srvctlstartscan_listener–i<scan_listener#>-n<node-name>$]srvctlstopscan_listener-i<scan_listener#>
Ø Tostart/stopaspecificclusterresource$]crs_stop<resource-name>-àNeedtorunfromthesamenode.$]crs_start–c<node-name><resource-name>àif–cisusedwithstartwecanrunitfromothernode.
Ø ToManageASMdiskgroup(WecannotstopthediskgroupusedbyOCRandVo=ngdisk)$]srvctlstart/stop/statusdiskgroup–g<dg-name>-node<node1>,<node2>
Ø WecannotstoptheASMalone(weneedtobringdowncluster)Ø Tomanagethelocallistener
$]srvctlstatus/stop/startlistenr–l<LISTENER-NAME>-n<node-name>Ø Tomanagethedatabaseinstance:
$]srvctlstatus/stop/startinstance–i<instance-name>-d<db-name>Ø ToManagedatabase:
Ø Srvctlstatus/stop/startdatabase–d<db-name>-o<open/mount/'readonly'>Ø Srvctlconfigdatabase–d<db-name>Ø Srvctlmodifydatabase–d<db-name>-o<ORACLE_HOME>-p<SPFILE-LOCATION>
• Usefulclusterwarecommands:Ø Tocheckthenodesinthecluster:
$]olsnodesØ Tochecktheclustername
$cemutlo–nØ Toviewsubnetsoftheinterfacesusedandmodifythem
$oifcfgge=fàtogettheinterfacesubnet$oifcfgse=f–globaleth1/<new-subnet>:<interface-type>àToaddnewsubnet$]Ifconfigeth1<new-priv-ip>netmask<value>broadcast<maxvalueforthesubnet>$]oifcfgdelif–globaleth1<old-subnet>:<interface-type>àTooldnewsubnet
Ø Tocheckthevo=ngdisk$]crsctlquerycssvotedisk
Ø Tochecktheocrinterity$]ocrcheck
Ø Tomovevotedisktodifferentdiskgroup$]crsctlreplacevotedisk<new-dg-name>
Ø TomoveOCR$]ocrconfig–manualbackup$]ocrconfig–add<new-dg-name>$]ocrcheck$]ocrconfig–delete<old-dg-name>
Ø Tochangethescanlistenerport$]srvctlmodifylistener–p<port#>
Ø 2-wayBlockRequest:(Itwillbeon2nodeRAC)§ SGA1sendsadirectrequesttoSGA2.SoSGA1waitsonthegccurrentblockrequestevent.§ WhenSGA2receivestherequest,itslocalLGWRprocessmayneedtoflushsomerecoveryinforma=ontoitslocal
redologfiles.§ Then,SGA2sendstherequestedblocktoSGA1.WhentheblockarrivesinSGA1,thewaiteventiscomplete,andis
reflectedasgccurrentblock2-way.
Ø 3-wayblockrequest§ Thisisamodifiedscenarioforaclusterwithmorethantwonodes.However,themasterforthisblockisona
nodethatisdifferentfromthatoftherequestor,andwheretheblockiscached§ Thus,therequestmustbeforwarded.§ WhentheblockarrivesinSGA1,thewaiteventiscomplete,andisreflectedasgccurrentblock3-way.
Ø 2-wayGrant§ Inthisscenario,agrantmessageissentbythemasterbecausetherequestedblockisnotcachedinany
instance.§ Ifthelocalinstanceistheresourcemaster,thegranthappensimmediately.Ifnot,thegrantisalways2-way,
regardlessofthenumberofinstancesinthecluster.§ Theround-triplookssimilartoa2-wayblockround-trip,withthedifferencethatthewire=meisdetermined
byasmallmessage,andtheprocessingdoesnotinvolvethebuffercache
• WhyweneedVIPS?Ø WithoutVIPsifusertriestoconnecttoRACdatabasewithallthelocalip'softheRACinaddresslist.Ifoneofthe
nodeisdownandifuserconnec=onwillwaitun=lit=mesoutbeforemovingtoothernode.TimeoutdependsonOSanditmayvaryfromscondsto2minutes.
Ø WithVIPSasCRSfailovertheVIPtootherhealthynode.WhenusetriestoconnectwithVIPSitwillgetthenolistenerimmediatelyanditmovestoothernodeimmediately.
Ø In11gR2evenwithscanvip'sitprovidesthesamefunc=onality.
• WhySCAN?Ø WithSCANwheneverweaddordeleteanodeuserdoesn’tneedtomakeanychangestotheconnec=onstring.