Best Practices for Upgrading to Oracle Enterprise Manager ... › technetwork › oem › install... · Global Marketing Support Disaster Recovery Zero downtime for OEM rolling patching

Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |

Best Practices for Upgrading to Oracle Enterprise Manager 13c: Learn from CustomersCON6989

Sumesh BalakrishnanPrincipal Product Manager, Oracle

Yong Kwong LungSystem Integration Advisor, DELL

Presented withSeptember 22, 2016

Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |

Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.


Agenda

Why Upgrade to EM 13.2 ?

New! Features of EM Lifecycle Management

Best Practices for a Smart Upgrade

Let’s hear it from -- DELL

1

2

3

4

Copyright © 2016, Oracle and/or its affiliates. All rights reserved.

Why Upgrade to EM 13.2Building On Success

Enterprise Manager 12c

Enterprise Manager 13.1

Enterprise Manager 13.2

• Enhanced Hybrid Cloud Management

• Fleet Maintenance

• TLS1.2 & IPV6 Support

• Stability & Performance Improvements

• Latest FMW Stack

• Enhanced Always on Monitoring

• Unified Hardware Management

• Compliance Management

• DB: Fine Grained Access Control

• Engineered Systems Management

• Database Consolidation Workbench

• Auto Discovery, Agent Push

• Patch Advisor

• Patch Plans

• Patch Automation

• Database management

• My Oracle Support Integration

• RAC provisioning

• Application Performance Management

• Always On Monitoring

• Unified Hardware Management

• Drift, Consistency and Compliance Management

• Database Management: Fine Grained Access Control

• Engineered Systems Management

• Database Consolidation Workbench

• Auto Discovery, Agent Push

• Patch Automation

• Database Management


Upgrade Step by Step Planning

Pre-Flight– Plan your upgrade

– Upgrade paths

– Fallback strategy

– Talk to us (Oracle) to avoid pitfalls

Production– Stage the new Software

– All in one downtime window

– OMS Bundle Patches

Test System– Development system upgrade

– Test system upgrade

Post Upgrade– Performance Monitoring

– New features

1 3

24


Upgrade Paths to EM 13.2

• Supported Direct Upgrade Path

12.1.0.4/12.1.0.5/13.1 13.2

• Pre-12.1.0.x Upgrade Path

11g /10.2.0.5 12.1.0.5 13.2

13.2

12.1.0.4 12.1.0.5 13.1

11.1/10.2.0.5

12.1.0.3


EM 13.2 Lifecycle Management

• Install / Upgrade

– Support on IPV6 platform

– Support for TLSv1.2 for DBs

• EM Prereq kit utility in GUI mode

• Reduced downtime: Smart-Upgrade

• Transition to DR Readiness – Storage Based Replication

• JDK 1.7U111 uptake with SHA2 certificate

• 12c Agent [version 12.1.0.5] deployment from 13.2 EM


EM Repository Health Check

• Enhanced EM Prereq Kit execution in OUI mode

– Option to installer em_linux64.bin EMPREREQ_KIT=true

Upgrade / Fresh Install


12.1.0.5 Agent Support on OEL 5.x

• PS4 Agent deployment on OEL 5.x platform from 13.2 OMS

• Deployment method supported

– Only emcli submit_add_host

• 12c Agent Life cycle– Agent Core Bundle Patch

emcli submit_add_host -

host_names=example.com -platform=226 -

installation_base_directory=/opt/agent -

credential_name=oracle -

privilege_delegation_setting=sudo -

agent_version="12.1.0.5.0"

EM 13.2

12c Agent on OEL/RHEL 5.x Hosts


EM 13.2 – Upgrade Recommendations

• Upgrade options supported : 1-System

• OMS and Repository Upgrade

– Upgrade your existing repository database to 12.1.0.2

– Apply latest Bundle Patch on top of 12.1.0.2 Database

– Perform Software only install with Plug-ins (Less Downtime - Recommended)

• Agent Upgrade

– Upgrade Target Agents using Gold Agent Image


Upgrade — runInstaller (OUI)

I. Mandatory: Upgrade Repository DB to 12.1.0.2

II. OMS Downtime & execute em_linux64.bin (Use1-system option in Installer)

– Copying the bits and creating new homes– Start the configuration assistants

III. Use Gold Agent Image console to upgrade agents to 13.2

Reduce the downtime using – Smart Upgrade

Upgrade – Software + Plug-in only Install

I. Mandatory: Upgrade Repository DB to 12.1.0.2

II. Advanced Option - Copying the bits/plug-ins and creating new homes – OMS + Plug-ins

III. Apply 13.2 OMS and Plug-in bundle patch in ‘bit-only’ mode (optional )

IV. OMS Downtime Start - Configuration$<MW_HOME>/sysman/install/ConfigureGC.sh

V. Use Gold Agent Image console to upgrade agents to 13.2

DOWNTIME REDUCED


Upgrade Best Practices Repository DB should be 12.1.0.2 version with latest bundle patch

Run EM diag kit.

Before Upgrade

Remove obsolete plug-ins from OMS/Agent e.g. Exalogic Elastic Cloud, EMC Symmetrix …etc.

Download the latest version of All Oracle plug-ins from otn e.g. Ebiz, PSFT, GoldenGate …etc.

Upgrade on certified platforms. Refer to MOS certification matrix for details

Perform Software only Install with Plug-ins to reduce downtime

Weblogic customizations are not carried...

pre-upgrade remove them and post upgrade redo the customization

Upgrade ALL Agents to 12.1.0.4/5 before Upgrading the OMS


EM 13.2: Upgrade & Transition to DR ReadinessSingle Domain Setup with Storage Replication

Primary Site Standby Site

OMS 1 (Active)Physical: Host 1Alias: emoms1

OMS 2 (Active)Physical: Host 2

Alias: emoms2

OMS 1 (Passive)Physical: Host 3

Alias: emoms1

OMS 2 (Passive)

Alias: emoms2

Physical: Host 4

STORAGE REPLICATION

STORAGE REPLICATION

STORAGE REPLICATIONOMS1 SoftwareGCDomain

OMS2 SoftwareGCDomain

ReposRAC DB

ReposRAC DB

Shared or ReplicatedSWLIB and BIP Shared

Storage



Shared or ReplicatedSWLIB and BIP Shared

Storage

Alias Host Name configuration for OMS hosts

OMS and Central Agent binaries are on replicated storage

OMS and Central Agent inventory is on replicated storage


EM 13.2: Single Step Adoption of Level-4 HA

• EM 13.2 Installer has new mode Upgrade and Transition to DR Readiness

– Enabled by passing new parameter UPGRADE_TRANSITION=true

– Also requires additional new parameter OLD_INVENTORY_LOCATION

– Extends support for existing parameter ORACLE_HOSTNAME

– Uses existing parameter –invPtrLoc

– Only supported in OUI ( no support for silent or Software Only or ConfigureGC.sh )

• Installer performs the following in this mode– Relocation of OMS to replicated storage (software and inventory)

– Configure the OMS to use alias hostname


Update Agents with Gold Image and propagate with ease

Create Gold Image and versions for each platform series.Subscribe agents to the images

Select source agent from the system and apply patches and test the agent

Push ImageAnd Update

Agents

Create Gold Image &

SubscribeAgents

Agent Upgrade Using Gold Agent ImageSimplified Agent Upgrade Operations at Scale

Patch SourceAgent

Ver. 3

Ver. 2

Ver. 1


Agent Upgrade Using Gold Agent Image321

2. Create Gold Images and Subscribe

Agent Image – Versioning

Ver. 3*

AUG PSU + couple more patches*

AUG 2016

JULY PSU + Config Changes

JULY 2016

Ver. 2

Ver. 1

APR 2016

APR PSU + 2 patches

A. Define Agent software as Images and Versions.


Subscribes to

Subscribes to

Agent Upgrade Using Gold Agent Image2. Create Images and SubscribeB. Subscribe Agents to the images

12.1.0.4/5/13.1 Agent(s)

321

Image ALinux x86_64_AGT_IMG

Ver. 3*

Ver. 2

Ver. 1

Subscribes to

Image BSolaris SPARC_AGT_IMG

Ver. 3*

Ver. 2

Ver. 1

Image CWindows x64_AGT_IMG

Ver. 3*

Ver. 2

Ver. 1


Agent Upgrade Using Gold Agent Image3. Propagate Changes and Upgrade Agents with Ease

321

DBAs Application Owners

Ver. 2

Push Agent Image on Targets Ver. 3*

Ver. 1

Ver. 3*

Ver. 2

Push Agent Image on TargetsVer. 3*

Ver. 1

Ver. 3*

Ver. 2

Push Agent Image on TargetsVer. 3*

Ver. 1

Ver. 3*Ver. 3*

Ver. 3*Ver. 3* Ver. 3

Ver. 3

Ver. 3

Ver. 3

Ver. 3

Ver. 3

Ver. 3

Ver. 3

Ver. 3

Image ALinux x86_64_AGT_IMG

Image BSolaris SPARC_AGT_IMG

Image CWindows x64_AGT_IMG

Linux x86_64 BATCH 1 (1000 agents)

Solaris BATCH 2 (1000 agents)

Windows BATCH 3 (1000 agents)


Quick Patching – 13.2

No Auto Rollback of Patches during apply

Minor changes while packaging bundle patches

Time Reduction while applying bundle patches

Pre-Requisite:

Update the new OMSPatcher in OracleHome


Traditional vis-à-vis Quick patching

BP – Bundle Patch

M I

N U

T E

S

1.754.22

14.95

34.45

1.753.47

10.1

17.22

0

5

10

15

20

25

30

35

40

Feb BP1 Apr BP2 July BP3 Aug BP4

Regular Patching Smart Patching

50%Time Reduction

Global Marketing

Oracle Enterprise Manager Migration from 12c to 13.1

Yong Kwong LungSystems Integration Advisor

Dell | IT Engineering, Infrastructure & Operation

[email protected]

[email protected]

mailto:[email protected]


Global Marketing

Agenda Summary

• Overview Oracle Enterprise Manager at Dell

• Enterprise Manager integration in Dell (Upstream and Downstream Apps)

Oracle Enterprise Manager in Dell

• Overall planning of hardware, architecture, key focuses – at a glance

Architecture, Hardware Spec & key Focuses

• Best practices / Activities during OEM13c migration

• Resource allocation plan

• Mass agent deployment via gold image and application plugin such as E-Business Suite, SOA, and Siebel

• Gold Agent Image Result

• Targets Discovery via EMCLI

• Lesson Learned, Team Feedback

Dell Approach on OEM Migration

Global Marketing

Oracle Enterprise Manager in Dell

Global Marketing

Overview Oracle Enterprise Manager at Dell

7.4KURL Monitoring

29KDatabase Targets

57KApplication Targets

(Middleware/EBS/Siebel/PS)

App/DB Servers Monitored in Production Enterprise Manager

App/DB Servers Monitored in Non-Production Enterprise Manager

2.6KAgents

350Reports

5DownstreamApplication

1.4KUDM

DB/Host

15Jobs /

Automation

1.5KUser

3KGrouping

Global Marketing

Monitored Targets Dell In-house ApplicationsOracle Enterprise Manager

Baseline Metric

Admin Group

Incident Rule

User Access

Reports

UDM

Notify Method

Targets DiscoveryAgent Installation

Compliance Policies

Groups

Targets Properties

OMR

(Database)

NT Authentication

PAC Group

Email Notification

SMTP Server

Incident (Remedy)Ticketing

Incident (ServiceNow)

JsonIncident

Convert to ServiceNowDec/2016

RMAN BackupDeployed in 2007

DMMR

Database Metadata Repository

Deployed in 2010

DBCM

Database Compliance Mgmt

Deployed in 2010

KPI

Key Performance Indicator

Deployed in 2013

DMP

Database Maintenance Planner

Deployed in 2012

Deployed in 2007

Oracle DB Monitoring

10g, 11g, 12c

Pilot Test 2011

Deployed in 2012

Oracle SOA Monitoring

10g, 11g

Pilot Test 2013

Deployed in 2015

Pilot Test 2013

Deployed in 2016

Oracle EBS Monitoring

11g, 12c

Oracle SIEBEL Monitoring

10g, 11g, 12c, 13c

Pilot Test & Deployed in 2015

URL Monitoring

N/A

Enterprise Manager in action at Dell

Automation

EMCLI

Jobs

Automation Service Improvement

Since 2010-2016

Global Marketing

Architecture, Hardware Spec & Key Focuses

Global Marketing

Support Disaster Recovery

Zero downtime for OEM rolling patching

New Enterprise Manager ArchitectureFrom 12c Level 3…

DataGuard

Oracle

Management Service

(13c)

Oracle Management

Repository (13c)

Oracle Database 12c

Oracle

Management Agent

13c

Archive: 512GB, FRA: 512GB, Data: 3TB Archive: 512GB, FRA: 512GB, Data: 3TB

Production Application Server Cluster

(U.S., Austin, PC1) Dell 13g M630'sDR Application Server Cluster

(U.S., Austin, PS3) Dell 13g M630's

EM12P3.PrdRAM 128GB

CPU 36 Core/u01 100GB

EM12P1.PrdRAM 128GB


EM12P2.PrdRAM 128GB


EM12P4.PrdRAM 128GB


Primary Cluster DB

(U.S., Austin, PC1) Dell 13g M630's

DR Cluster DB

(U.S., Austin, PS3) Dell 13g M630's

EM12P1.DRRAM 128GB


EM12P2.DRRAM 128GB


EM12P3.DRRAM 128GB


EM12P4.DRRAM 128GB


400GB

Additional OMSRAM 128GB




Primary ServerRAM 128GB




Primary ServerRAM 128GB






Agent Agent Agent Agent Agent Agent

400GBRsync







…to 13c Level 4

Shared Storage

Production Application Server Cluster

(U.S., Austin, PC1)

EM12P3EM12P1 EM12P2 EM12P4

em12p.prd.amer.dell.com (U.S., Austin, PC1)

Additional OMSAdditional OMSPrimary Server Additional OMS

Agent Agent Agent

Global Marketing

Monitored Targets Dell In-house ApplicationsOracle Enterprise

Manager

Baseline Metric

Admin Group

Incident Rule

User Access

350 Reports

1.5K UDM

Notify Method

Targets Discovery2.6 Agent Installation

Compliance Policies

3K Groups

Target Properties In-house Custom App build on Oracle APEX

OMR

(Database)

Non-Standard DBSNMP Password


10g, 11g, 12c

Convert to ASMSNMP Account


10g, 11g

Linux (V5, V6 & V7)

Windows Servers

Existing 12cR2 Agent


11g, 12c


10g, 11g, 12c, 13c

Unhealthy / Pending Decommission Targets

URL Monitoring

N/A

Project Implementation Key Focuses

Automation

EMCLI

Jobs

15 Automation for Service Improvement

High – Resource & Time

Medium – Resource & Time

Low– Resource & Time

NT Authentication

PAC Group

Email Notification

SMTP Server

Incident (Remedy)Ticketing

Incident (ServiceNow)

JsonIncident

Convert to ServiceNowDec/2016

RMAN Backup

DMMR


DBCM


KPI


DMP


Global Marketing

Dell Approach on OEM Migration

Resource Plan

Migration Plan

Agents Installation

Targets Discovery

Global Marketing

Group 3:

OEM Administrator

Gro

up

2:

Do

wn

stre

am

Ap

ps In

teg

ratio

n

Gro

up

1:

OE

M1

3c A

ge

nt M

igra

tio

nOracle Enterprise Manager

Baseline Metric

Admin Group

Incident Rule

User Access

350 Reports

1.5 UDM

Notify Method

Targets DiscoveryAgent Installation

Compliance Policies

3K Groups

Target Properties

OMR

(Database)


10g, 11g, 12c


10g, 11g


11g, 12c


10g, 11g, 12c, 13c

URL Monitoring

N/A

Resource Allocation Plan

350 Reports1.5k UDM

ComplinacePolicies

3K GroupsGroup 4:

Overall OEM Functional Validation, Monitoring health check and performance test

Project Manager

Automation

RMAN Backup

DMMR


DBCM


KPI


DMP


Global Marketing

Migration Plan:Two System Approach

Business Decision

• Downtime > 2 hrs.

OMR Health Concern

• Potential upgrade failure

End to End Infra Changes

• SLB, OMS, OMR

End to End Integration Testing

• Upstream & Downstream App

UDM convert to Metric Ext

• Complexity of the conversion

Customer Sign-Off

• Go live sign-off


OMR

Current OMR Server New OMR Server

OMR

PrimaryServer

AdditionalOMS

Agent 12c

/u02

/u01/u01


PrimaryServer

AdditionalOMS/u02

/u01/u01

Agent 13c Agent 12c Agent 13c Agent 12c Agent 13c Agent 12c Agent 13c

Global Marketing

Create Gold Agent Images

{APP}_{OS}{PLATFORM}

E.g.: odb_linux64

Pre-requisite• On OMS Server

• 24308359-August Bundle Patch 13.1.0.0.160816

• 24364619-Plugins 13.1.1.0.160816

• 22741755-Support OEL5.x

• On Agent Target Server (one only)

• Deploy 13.1 Agent

• Plugin Deploy

• 24308442-August Bundle Patch 13.1.0.0.16081

{APP}_{OS}{PLATFORM}_{VER}

E.g.: odb_linux64_v1

{APP}_{OS}{PLATFORM}

E.g.: odb_linux64

Application OS Platform Gold Image Version

Oracle E-Business Suite Linux 64bit EBS_Linux64 ebs_linux64_v1

Oracle Database Linux 64bit ODB_Linux64 odb_linux64_v1

Oracle SOA Linux 64bit SOA_Linux64 soa_linux64_v1

Oracle SIEBEL Win 64bit SIEBEL_Win64 siebel_win64_v1

1

2

3

4

5

6

Global Marketing

Agent Installation using Gold Agent Images

1

2

3

Add New Server

Gold Agent Image

Installation Details

Start Agent Deployment

Fresh Agent Installation

Agent Upgrade 1

3

Subscribe Agent(s)

Start Agent Deployment

Select Agent(s)

2

Global Marketing

Gold Agent Images Result

Day 1

•333 DB hosts

Day 2

•4 SEIBEL hosts

Day 3

•26 EBS hosts

Day 4

•195 SOA hosts

Day5

•229 DB hosts

604 hosts success

Pilot Test Result183 of 787 host failed:

• SSH Daemon Error

• Agent File-system Full

• Lack of Account Permission

• Un-contactable Servers

•Agent Installation =

•Patches Apply =

•Plugin Deploy =

•Parameter Update =

With Gold Agent Images (5 Hosts)

30Minutes

•Agent Installation = 24 minutes

•Patches Apply = 20 Minutes

•Plugin Deploy = 20 Minutes

•Parameter Update = 10 Minutes

Without Gold Agent Images (5 Hosts)

74 Minutes

604 Agents

Agent yet to be updated

Agent without images subscribe

Image Name with Total Agents

Compliance Status

Global Marketing

• Retrieve Targets Configuration

• gc$target

• gc$target_properties

Preparation

• Generate EMCLI command

• emcli add_target

Converting• Register Targets to OEM13c via EMCLI

Execution

Enterprise

Manager 12c

Enterprise

Manager 13c

Targets Discovery

ClusterCluster DB

Weblogic Domain

Cluster ASMDB InstanceListener

User Registration

ASM Instance

GroupingTarget’s Properties UDM conversion

User Role Granting

Fusion Middleware

Global Marketing

select

t.target_name,

t.host_name,

'$ORACLE_HOME/bin/emcli add_target -name=""' ||t.target_name

|| '"" -type=""oracle_listener"" -host=""' || t.host_name

|| '"" -properties=""LsnrName:' || p.lsnrname

|| ';ListenerOraDir:' || p.listeneroradir

|| ';Port:' || p.port

|| ';OracleHome:' || p.oraclehome

|| ';Machine:' || p.machine || '""'

from gc$target t

inner join ( select

target_guid,

max(decode(property_name, 'LsnrName', property_value, null)) as lsnrname,

max(decode(property_name, 'ListenerOraDir', property_value, null)) as listeneroradir,

max(decode(property_name, 'Port', property_value, null)) as port,

max(decode(property_name, 'OracleHome', property_value, null)) as oraclehome,

max(decode(property_name, 'Machine', property_value, null)) as machine

from mgmt_target_properties

group by target_guid) p

on t.target_guid = p.target_guid

where t.target_type ='oracle_listener'

Listener Target Discovery Example:Preparing EMCIL command line

Target Created

Target’s status healty

Global Marketing

Feedback by Dell Project Team…

Gold Images were excellentGood so far

Less effort and time

Strategic guidance AND execution are clearOEM13c Better UI & Performance

“Improve my knowledge”

Save CostEasy to follow up

Customer Loves it!Never patch Agent againSpeed up targets migration via EMCLI

Less communication gap

Proud to be part of the team!!

Global MarketingDell - Restricted - Confidential

Thank You

Yong Kwong Lung

Systems Integration Advisor

Dell | IT Engineering, Infrastructure & Operation

[email protected]

[email protected]




Zero Downtime UPGRADE

Zero Downtime PATCHING

Parallel OMSUPGRADE

What’s Next..


Additional Resources

Join the Conversation

Twitter.com/oracle_em

Oracle.com/newsletters

Facebook.com/oracleenterprisemanager

Blogs.Oracle.com/OEM

Enterprise Manager Page on O.com

Install and Upgrade Page on OTN

Upgrade Guide (Documentation)

http://www.oracle.com/us/products/enterprise-manager/index.html

http://www.oracle.com/technetwork/oem/enterprise-manager/downloads/index.html

http://www.oracle.com/us/products/enterprise-manager/index.html

http://www.oracle.com/technetwork/oem/cloud-mgmt-496758.html

http://www.oracle.com/technetwork/oem/install-upgrade/index.html

http://www.oracle.com/technetwork/oem/cloud-mgmt-496758.html

https://docs.oracle.com/cd/E63000_01/EMUPG/toc.htm

Documents

Best Practices for Upgrading to Oracle Enterprise Manager ... › technetwork › oem › install... · Global Marketing Support Disaster Recovery Zero downtime for OEM rolling patching