Express Scripts Leverages Enterprise Manager 12c to Monitor

Confidential and Proprietary Information © 2013 Express Scripts Holding Company. All Rights Reserved. 1

Express Scripts Leverages

Enterprise Manager 12c

to Monitor Critical Databases

Twain Hoang, Sr. Oracle DBA

9/25/2013


Making the Use of Prescription Drugs Safer, More Affordable

National Leader in Pharmacy Benefits

1.4 BILLION Rx per year

100 MILLION members

30,000 employees

3,500 clients


ESI Oracle DBA Team Operations

Responsible for enterprise-wide Oracle Database

operations

Support core pharmacy functions (patient care, order fulfillment,

adjudication) for Home Delivery and Specialty Pharmacies

Manage Enterprise Manager deployment

Twain Hoang, Sr. Oracle DBA

Barbara Porter, Sr. Oracle DBA


Goals & Requirements for EM12c Deployment

Unify multiple Grid Control 10g sites into single monitoring tool for ~650 Oracle

Databases

Versions 9i to 11gR2, RAC and non-RAC

Platforms: Linux (x86_64, zLinux), AIX, HPUX, Windows

Carry over Grid Control10g metric settings

Meet ESI monitoring requirements

Scalable

Cutover custom scripts to standard metrics

Meet on-call notification requirements

Flexible to add new metrics


ESI EM12c Setup – Administration Groups

Level 1 Prod vs NonProd

Level 2 RAC vs NonRAC


Oracle Metrics and Metric Extensions

Used standard Oracle metrics for database, host, listener targets

Created Metric Extensions (ME) to monitor specific conditions

Locked accounts status

Failed login attempts

Broken and failed scheduler jobs (out-of-the-box job metric queries

DBA_JOBS)

Metrics and ME included in Monitoring Template/Template

Collections


Associate Template Collections with Administration

Groups



NonProd Template Collection

Prod Template Collection

NonProd RAC Template Collection

NonProd NonRAC Template Collection

Prod RAC Template Collection

Prod NonRAC Template Collection

Template Collections

Global Sync runs daily @6pm


ESI EM12c Setup – Incident Rule Set

Specify top level administration group as Rule Set target

Rules for different event types (Metric Alert and Target Availability): Send email to DBA team, CMEA; send SNMP traps


Monitoring Transition from Grid Control to Cloud Control

Install EM12c agent using different port from 10g agent (default port 3872).

Configure and promote discovered targets

Ensure targets status are green before assigning target properties

Assign Target Properties:

• For small numbers of targets, configure target properties on individual target pages:

Target Setup > Target Properties

• Batch update via emcli command: emcli set_target_property_value

Enterprise Manager automatically adds target to Admin Group and propagates

metric settings

In GC10g, apply full host blackout before shutting down GC10g & disabling

agent


Refer to Appendix Slides for Implementation Details

Administration Group structure

Metrics Used for Database, Host, Listener targets

Metric Extensions – deploying on RAC and non-RAC

databases

Monitoring Templates structure

Roles / Privileges for monitoring setup

Use of System Dashboard


Tips Strategy

To scale, take advantage of Administration Groups to

automatically propagate monitoring metrics

Before creating anything, think about what kind of metrics

suitable for your organization. The admin hierarchy may

have to be recreated as requirements are worked out

In ESI case, rules definition are broken down by metric

types


Benefits / Goals Met

Meets ESI Monitoring requirements

Single tool to monitor the overall health of Oracle Grid

Infrastructure and Database components

Centrally manage all Oracle Database performance

Information Publisher Reports provides the framework to

create custom reports, i.e. DBA Dashboards

Provides the framework to further monitor Database

Compliance and Security Policy


Thank You


Appendix: Details of Monitoring

Implementation



Utilize properties Life-cycle & Department

Level 1 - Life-cycle

• Prod (Production, Mission Critical) vs NonProd (Development, Staging, Test)

• Carry Prod & NonProd groups over from GC10g setup

Level 2 – Re-purposed Department

• RAC vs NonRAC

• Added RAC & NonRAC groups to avoid duplicate alerts for cluster

databases






Define Standard Metrics for Target Types Agent : Count of targets not uploading data

Host: CPU Utilization

Listener: Status Down

DB Instance:

Archiver Hung Alert Log Error, Data Block Corruption Alert Log Error,

Generic Alert Log Error, Media Failure Alert Log Error, Session Terminated

Alert Log Error, Broken & Failed Job Count, Process & Session Limit Usage

(%), Failed Login Count, Status (Down), Tablespace Free Space (MB),

Tablespace Space Used (%), Segment Approaching Maximum Extents

Count, Segment Not able to Extend Count, Blocking Session Count


Define Standard Metrics for Target Types

Cluster Database:

Database Job Broken & Failed Count, Failed Login Count, Tablespace

freespace (MB), Tablespace Used (%), Segment Approaching Maximum

Extents Count, Segment Not able to Extend Count; Blocking Session Count

Cluster ASM:

Disk Group Usage, Disk Status, Offline Disk Count

ASM:

Disk Group Used (%), Disk Mode Status (Offline), ASM Block Corruption,

Read Write Errors, Offline Disk Count, Generic Operational Error, Media

Failure, Status (Down)


ESI EM12c Setup – Define Metric Extensions Define Metric Extensions (ME) to extend monitoring capabilities beyond out-of-box metrics

ESI monitors specific conditions

Locked accounts status

• Security requires accounts to be locked after x number of attempts

Failed login attempts

• Tracks when failed attempt occurs and notifies

Broken and failed scheduler jobs

• Out-of-the-box broken & failed job count metrics queries DBA_JOBS

Define same ME for Cluster Database and Database Instance target types

Split Level 1 administration groups into RAC group and Non-RAC group

ME for Cluster Database included in Template for RAC group

ME for Database Instance included in Template for non-RAC group to be deployed to single instance databases

Split groups so ME for Database Instance not deployed on RAC instances which cause over-alerting


ESI EM12c Setup – Create Templates Template for each Target

Types:

Listener

Host

Database Instance

Cluster

Cluster Database

Cluster ASM

ASM

Agent


Associate Template Collections with Administration

Groups



NonProd Template Collection

Prod Template Collection

NonProd RAC Template Collection

NonProd NonRAC Template Collection

Prod RAC Template Collection

Prod NonRAC Template Collection


Define Synchronization Schedule

ESI Sync runs daily @6pm


Define Incident Rule Set Specifying Admin Group Configure rules for basic email notifications and advanced SNMP traps

notification methods

SNMP notification methods have to be defined before they can be added to

rules. (Setup => Notifications => Notification Methods)

• Leverages advanced notifications to send SNMP traps to SNMP servers which

generates Remedy incidents.

• Only the “ESI Std ME Rule - Email CMEA” rule emails CMEA. The rest of the rules

only email the DBA team.

• Central Monitoring Team operates 7x24 and reacts as tier1 support for specified

events defined in playbook.

• On-call DBA acts as tier1 / tier2 support for database-related critical incidents.


ESI EM12c Setup – Incident Rule Set

Specify Admin Group Target

Create Rule Set for different events and

metric alerts


ESI EM12c Setup – Multiple Rules in Incident Rule Set


ESI EM12c Setup – Roles and Responsibilities

Subset of the Oracle DBA Team manages Enterprise Manager and

has Super Admin privilege. Responsibilities include monitoring

configuration, target promotions, management of metrics, ME,

templates/template collections, incident rules, …

Defined super administrator EM account MONITOR_ADMIN which

is used for monitoring related tasks

Defined ESI_OPERATOR role with EM_USER role and necessary

privileges for operational DBAs (non-EM Admin)


ESI DBA Dashboard: At a glance assess if everything is ok


References

Oracle White Paper

Title: Strategies for Scalable, Smarter Monitoring using Oracle

Enterprise Manager 12c Cloud Control

EM 12c Online Documentation

http://download.oracle.com/docs/cd/E24628_01/index.htm

http://download.oracle.com/docs/cd/E24628_01/index.htm


Thank You

Documents

Express Scripts Leverages Enterprise Manager 12c to Monitor