Nationwide Database Operations Team

Database Software Currency

Using EM 12c Provisioning and PatchingGary Henderson

2September 29, 2014Nationwide Database Operations Team

About Me

• Nationwide employee for 22 years

• Application Programming background

• Joined DBA Team at the beginning of 2000

• EM Administrator since 2008 (10g)

• EM 12c Early Adopter

• NW Champion of EM Provisioning and Patching

Contents

June 12, 2012Nationwide Database Operations Team 3

Insurance and Financial Services Company

Fortune 100 Company

Founded in 1925

34.5 Billion in Revenues

171 Billion in Assets

33,000 Employees

26 core DBAs with combined 410 years DBA experience

Nationwide

4September 29, 2014Nationwide Database Operations Team

Agenda

• Why software currency is important

• Current Configuration State

• Patching – Traditional - OPatch– With EM – In Place– With EM – Out of Place

• Provisioning

• Maturity Stages

• Nationwide Experiences– Success – Challenges– Future

• Questions

5September 29, 2014Nationwide Database Operations Team

Database Software Currency

• Why software currency is important to us ..at Nationwide

– Availability

– Security

– Efficiency

– Stability

6September 29, 2014Nationwide Database Operations Team

Current Situation -- Versions

Oracle 9.2< 1%

Oracle 10.212%

Oracle 11.124%

Oracle 11.264%

Oracle 12.1< 1%

Database Versions

7September 29, 2014Nationwide Database Operations Team

Current Situation -- Configuration

RAC - MultiNode17%

RAC - 1 Node17%

Veritas32%

Non-Clustered34%

Clustering

8September 29, 2014Nationwide Database Operations Team

Enterprise Manager 12c Deployment Architecture

9September 29, 2014Nationwide Database Operations Team

EM12c Monitoring & Management – Current Target Availability

10September 29, 2014Nationwide Database Operations Team

Patching (Classic)

• Find the proper patch(s) • Download• Stage the patch(s) on target server• Unzip patch(s)• Upgrade Opatch• Analyze patch(s) for conflicts

– Schedule Downtime• Blackout targets (databases & listeners)• Bring down database(s) & listener(s)• Apply patch(s)• Process post upgrade steps (if any)• Start database(s) & listener(s)• End Blackout

11September 29, 2014Nationwide Database Operations Team

Patching with EM12c – My Oracle Support Integration

12September 29, 2014Nationwide Database Operations Team

Patching with EM12c – Patch Selection

13September 29, 2014Nationwide Database Operations Team

Patching with EM12c – Create New Plan & Pick Target

14September 29, 2014Nationwide Database Operations Team

Patching with EM12c – Plan Information

15September 29, 2014Nationwide Database Operations Team

Patching with EM12c – Specify Patches

16September 29, 2014Nationwide Database Operations Team

Patching with EM12c – Deployment Options

17September 29, 2014Nationwide Database Operations Team

Patching with EM12c – Specify Credentials

18September 29, 2014Nationwide Database Operations Team

Patching with EM12c - Analyze

19September 29, 2014Nationwide Database Operations Team

Patching with EM12c – Execution Status,Logs

20September 29, 2014Nationwide Database Operations Team

Complex Configuration – Real World example

21September 29, 2014Nationwide Database Operations Team

Lifecycle of an Oracle Home – Out of Place patching

Oracle Home

DB 1 DB 2

DB 3

Oracle Home + PSU

DB 1 DB 2

DB 3

22September 29, 2014Nationwide Database Operations Team

EM12c Provisioning

23September 29, 2014Nationwide Database Operations Team

Crawl … Walk … Run … Fly

In-place patching via EM12c

Some provisioning of Database homes

Some provisioning of Cluster Ready Services (CRS)

Out of place patching of database homes

Gold Copies for each unique configuration

Leveraging templates and profiles

All provisioning via EM12c

Patch frequency increased to semiannual or quarterly

Utilizing Change Activity Planner

Cloud Provisioning

24September 29, 2014Nationwide Database Operations Team

EM12c Provisioning – Lessons Learned - Success

• Test EM12c patches/upgrades

• Oracle Home and inventory verification (especially for Veritas)

• Sticking with Oracle (RAC) vs other Clustering (Veritas)

• Having a dedicated EM12c team (we have 2 people)

• Global Credentials

• Maintain currency with EM12c and database codebase

• Extensive testing of provisioning and patching

25September 29, 2014Nationwide Database Operations Team

EM12c Provisioning – Lessons Learned - Challenges

• Initial /tmp space too small

• Assign to single person

• Tried to implement prior to proper training

• Entrenched DBA team with decades of experience

• New approach to one-off patches

• “Configuration Pollution”

26September 29, 2014Nationwide Database Operations Team

EM12c Provisioning – Future

• Leverage Patch templates more• Leverage Provisioning Profiles more and better• Integrate

– Patch Levels– Gold Copies– Patch templates– Change Activity Planner

• Attempt the 100% provisioning goal• Accelerate the patching cycle• Develop Cloud Proof of Concept

27September 29, 2014Nationwide Database Operations Team

EM12c Lifecycle Management Pack

Configuration Management

Compliance

Change Management

Patching

Provisioning

Prerequisite to Cloud Provisioning Features used currently at Nationwide

Nationwide Database Operations Team

Summary:

Recognition:

Questions???

Nationwide Database Operations Team

Other Sessions:

HOL9532Achieving Standardization with Oracle Enterprise Manager Database Lifecycle Management