Configuring Oracle Enterprise Manager Cloud Control 12c for

High AvailabilitySession ID# 326

Leighton L. Nelson

Mercy

About Me

•Oracle Certified DBA with 10+ years experience•Oracle RAC SIG US Events Chair/IOUG Liaison•Co-Author (Expert Oracle Enterprise Manager 12c- Apress)

Agenda•Architecture•HA Overview

•Level 1 HA – Single OMS & Repository•Level 2 HA – Active/Passive OMS with Local Data Guard•Level 3 HA – Multiple OMS behind SLB with RAC Data Guard

•General Recommendations for HA•Configuration Steps•Backup & Recovery

What is High Availability?

99.9999999999999999999999

“Seven Nines” = 3.15 seconds downtime per

year

What is High Availability?

Architecture

High Availability Levels

Level 1- Single OMS & Repository

Level 1- Single OMS & Repository

•Separate servers for OMS & Repository

•Keeps resource usage separate

•memory_target or sga_size on database repository

•Java VM Heap Size - Xmx

Level 1- Single OMS & Repository

Pros Cons

Simple Setup No redundancy

Low cost

Scalable

Level 2 – Active/Passive OMS w/ local Data Guard

Level 2 – Active/Passive OMS with local Data Guard

OMS binaries installed on shared filesystemNFS/OCFS2

Uses virtual hostnameManaged by Oracle Clusterware or 3rd party solution

Virtual hostname should resolve to unique IP addressConfigure repository database using local physical standby w/ Data Guard

Level 2 – Active/Passive OMS with local Data Guard

Pros Cons

Protects against repository and OMS host failure

More complex

Switchover time minimal No storage redundancy for OMS

Requires additional hardware

Level 3 – Active/Active OMS w/ SLB and RAC Data Guard Repository

Level 3 – Active/Active OMS w/ SLB and RAC Data Guard Repository

•Primary OMS with RAC Repository•SLB for multiple OMSs•All access to OMS done via SLB

•EM Console•EMCLI•Agents

•Standby Database (RAC or Single-Instance)

Level 3 – Active/Active OMS w/ SLB and RAC Data Guard Repository

Pros Cons

Protects against repository and OMS host failure

Very complex

Switchover time minimal More hardware

Protects against storage failure

Recommendations for Repository HA

•Use dedicated database•Use ASM for database storage•Configure RAC Repository•Configure Standby Database•Configure Fast Start Failover•Review MAA Advisor after OEM installation

Configure DB service for Repository

11gR2 methodsrvctl add service -d emrep -s emrepsrvc -l PRIMARY -q FALSE -e NONE -m NONE -w 0 -z 0

Pre-11gR2 method•Add service using DBMS_SERVICE.CREATE_SERVICE•Start Service using DBMS_SERVICE.START SERVICE•Create trigger to start service when in primary role

Recommendations for OMS HA

•Configure Multiple OMSs•Configure SLB in Active/Active configuration•Install OMS on separate hosts from repository•Use “Add Management Service” Deployment Procedure to Add Primary OMS•Use Modified “Add Management Service” Procedure to Add Standby OMSs

Configure OMS HA•Active/Passive configuration uses virtual hostname•Create Application VIP using Clusterware

•11gR2 uses appvipcfg command•Pre-11gR2 uses crs_profile command

•Clusterware manages failover of virtual hostname•OEM can also be configured as Clusterware resource

Configure OMS HA•Active/Active configuration requires SLB•SLB requirements

•Virtual Servers•Persistence (Stickiness)• Health Monitoring

•Register virtual hostname and IP in DNS

Configure OMS HA

Configure OMS HA1 Create Pools

• Group of servers receiving on same port

• One pool per service 2 Create Virtual Servers

• Redirects requests to servers in pool

• Clients access virtual hostname and IP

3 Create Health Monitors• Checks health of services at an

interval

Configure OMS HA•Secure OMS with SLB

$OMS_HOME/bin/emctl secure oms -sysman_pass \-reg_pwd regpass \-host slb.example.com \-secure_port 4904 \-slb_port 4904 \-slb_console_port 443 \-reset \-console \-lock_console

Configure OMS HA

•Re-secure Agents with SLB•emctl secure agent –emdWalletSrcUrl https://slb.example.com:4899/em

•Re-configure EMCLI•emcli setup –url=https://slb.example.com/em –username=sysman

Configure OMS HA

•Configure OMS to use SCAN name in 11gR2 or VIPs in pre-11gR2 for RAC

emctl config oms -store_repos_details -repos_conndesc " (DESCRIPTION= (ADDRESS=(PROTOCOL=TCP) (HOST=emrep-scan.example.com)(PORT=1521) ) (CONNECT_DATA=(SERVER=DEDICATED) (SERVICE_NAME = emrep)))" -repos_user sysman

Configure OMS HA

•Configure OMS to use service name when database role changes in 11gR2

emctl config oms -store_repos_details -repos_conndesc '(DESCRIPTION=(FAILOVER=ON)(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=oemhost1)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=oemhost2)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=emrepsrvc.smrcy.com))(FAILOVER_MODE=(TYPE=select)(METHOD=basic)))' -repos_user sysman

Configure OMS HAAdd Management Service

Configure OMS HA

Recommendations for Agent HA

•Configure to start automatically on boot•Install on redundant storage•Configure restart environment variables for watchdog daemon

•EM_MAX_RETRIES•EM_RETRY_WINDOW

Recommendations for Software Library

•Replicate to secondary system•Storage based replication•rsync

Backup & Recovery•Repository•OMS•Agent•Software Library

Backup & Recovery•Enable Archivelog mode•Backup database using RMAN•Enable Flashback database•Enable Flash/Fast Recovery Area

Backup & RecoveryRepository Recovery•Stop OMS•Restore and Recover database using RMAN•Reconfigure OMS if DB restored to diff host•Complete Repos Resync if Incomplete Recovery•Restart Admin Server and OMS

Backup & Recovery•Backup OMS filesystem and instance home:•Export static configuration

$OMS_HOME/bin/emctl exportconfig oms -sysman_pwd oracle12c -dir /mnt/backup

Backup & RecoveryOMS Recovery•Restore OMS HOME from backup•Software-Only install•Re-install plug-ins using PluginInstall.sh•$OMS_HOME/bin/omsca recover –as –ms -nostart –backup_file <exportconfig file>•Restart OMS

Backup & RecoveryAgent Backup

•No special considerations•Maintain reference image

Backup & RecoveryAgent Recovery

Reinstall Agent Software from Reference Image

SummaryHA Level should match business needs

Complexity increases costHA for Enterprise Manager consists of HA

for all components

Who are we?•Sixth largest Catholic health care system in the U.S.

•Serves more than 3 million people annually.

•Highly integrated organization that includes 32 hospitals, 300 outpatient locations, 39,000 co-workers and 1,700 integrated physicians in Arkansas, Kansas, Missouri and Oklahoma.

•Among 10 percent of health care organizations nationwide to have a comprehensive, integrated electric health record that provides real-time, paperless access to patient information

•MyMercy, a free service, allows patients to connect online with their doctors, plus see medical information, view test results, renew prescriptions an schedule appointments using a personal computer or smart phone

•Health Information and Management Systems Society, a global leader in health care technology, designated Mercy a stage six of a seven-stage process, joining the ranks of health care leaders such as Johns Hopkins and Mayo Clinic

•$60 million data center provides fail-safe protection and power. When an EF5 tornado struck Mercy's Joplin facility in 2011, patient records electronically backed up at offsite data center were immediately available

•Mercy SafeWatch, the largest electronic ICU in the nation, provides 24-hour vigilance to critically ill patients. Alongside on-site medical teams, high-definition cameras and sophisticated technology allow specialized ICU physicians and nurses in a central command center to expand each patient's care team

Questions? Comments?

blogs.griddba.com

linkedin.com/pub/leighton-nelson/4/6b8/a7a

@leight0nn

leighton.nelson@mercy.net

THANK YOU