Upload
duongkien
View
224
Download
1
Embed Size (px)
Citation preview
1
<Insert Picture Here>
Exadata Planned Maintenance
Rene Kundersma
Oracle MAA Development June 2012
2
Agenda
• Overview
• Patch Planning and Testing
• Patch Installation
2
3
Terminology
• MAA – Oracle Maximum
Availability Architecture
• GI – Grid Infrastructure
• RAC – Oracle Real Application
Clusters
• ASM – Oracle Automatic
Storage Management
• PSU – Patch set update
• CPU – Critical patch update
• BP – Bundle Patch
• MOS – My Oracle Support
• ULN – Unbreakable Linux
Network
• OEL – Oracle Enterprise Linux
• IB – InfiniBand
• OFED – OpenFabrics
Enterprise Distribution
4
Overview
3
5
Exadata ArchitectureReview
Database Grid Storage Grid
InfiniBand Network
Switches
Exadata Storage ServersDatabase Servers
•Firmware / OS
•Oracle GI / RDBMS
Ethernet Switch
Power Distribution PDU
Keyboard, Video, Mouse KVM
Additional Components
6
Planned MaintenanceSoftware and Hardware
Component Software
Maintenance
Hardware
Maintenance
Storage servers Yes Yes
Database servers Yes Yes
InfiniBand switches Yes No
Additional components Yes No
4
7
Exadata Storage Server
• Single patch download
• Same patch applies to all hardware
• Exadata software, OS, ILOM, firmware
• README and Support Note
• May include updates for database servers
• Minimal pack (< 11.2.3.1)
• Delivered via ULN starting in 11.2.3.1
# imageinfo
Active image version: 11.2.2.4.2.111221
8
Exadata Database ServerOperating System and Firmware
• Linux or Solaris x86
• Exadata-supplied updates
• Unbreakable Linux Network
• Minimal pack (<11.2.3.1)
• Solaris Support Repository
• Any software, any source (but)
• Maintain IB compatibility
• Maintain GI and DB requirements
• Shipped images/versions tested
# imageinfo
Kernel version: 2.6.18-238.12.2.0.2.el5
Image version: 11.2.2.4.2.111221
5
9
Exadata Database ServerOracle Database Server (RDBMS) and Grid Infrastructure
• Major, Maintenance, Patch Set versions (e.g. 11.2.0.2)
• Same as with non-Exadata storage
• May require minimum Exadata storage version
• Database 11.2.0.3 requires Exadata 11.2.2.4 or later
• Database Patch for Exadata (e.g. 11.2.0.3.4, aka BP)
• Tied to patch set
• Installed using OPatch
• Cumulative, and includes recent PSU / CPU
• Other patches (overlays, one-offs)
$ opatch lspatches
Quarterly Diskmon Patch for Exadata (JAN 2012 - 11.2.0.3.2) : (13513982)
Quarterly CRS Patch for Exadata (JAN 2012 - 11.2.0.3.2) : (13540563)
Quarterly Database Patch for Exadata (JAN 2012 - 11.2.0.3.2) : (13513783)
10
InfiniBand Switch
• Only update via Exadata-branded patch
# version
SUN DCS 36p version: 1.3.3-2
6
11
Patch PlanningLatest Releases (June 2012)
• MOS 888828.1• “The Patching Note”
• Critical issues and additional required patches
• Guidelines for related products (OEM, DBFS, EBS, RM, GG,
SAP)
Component Latest Software Recommended Minimum
Exadata Storage
Server
11.2.3.1.0 11.2.2.4.2
Database Server
11.2.0.3: 11.2.0.3.6
11.2.0.2: BP16
11.2.0.1: DB_BP12,GI_BP6
11.2.0.3: 11.2.0.3.4
11.2.0.2: BP13
11.2.0.1: DB_BP12,GI_BP6
InfiniBand Switch 1.3.3-2 1.3.3-2
12
Patch Planning and Testing
7
13
Patching Improvements Timeline
20092009
Auto Cell firmware update
20102010
Exadata bundle patch (BP)
Out-of-place patching step-by-step
Minimal Pack firmware update
Cell rolling patch apply
BP EM install
DB+GI BP merge
BP OPatch Auto install
20112011
BP DG Standby-First install
Minimal Pack Linux kernel update
BP RAC rolling install
BP OPlan
IB Switch update w/ ILOM
Linux upgrade step-by-step
11.2.0.2 upgrade step-by-step
Oracle Advisor webcasts
IOUG Exadata SIG webcast
• Reduce Downtime
• Simplify Install
• Reduce Risk
• Transfer Knowledge
Patching Improvements
20122012
11.2.0.3 upgrade step-by-step
11.2.0.3 Exachk readiness
11.2.0.3 QDPE
11.2.0.3 QFSDP
EM 12c Out-of-place patching
Linux dbserver ULN updates
14
Patch PlanningRelease Frequency
Component Patch Release Frequency
Exadata Storage Server Patch set: 6-12 months
Patch set update: 1-6 months
Database Server Patch set: 12-18 months
Database Patch for Exadata
11.2.0.3: Quarterly (w/ CPU)
Monthly interim for reactive patching
11.2.0.2: Quarterly (w/ CPU)
InfiniBand Switch Semi-annually to annually
Subject to change without notice
8
15
Planned MaintenanceSuggested Schedule
Frequency Planned Maintenance Action
3-6 monthsUpdate Exadata Storage Server software
Update Database Quarterly Database Patch for Exadata (QDPE)
1-2 years
Update Database patch set
Update InfiniBand switch software
Update Additional components software (if necessary)
2 years Replace disk controller batteries in storage and database servers
2-4 yearsUpdate Database major or maintenance release (e.g. 11.1 to 11.2, or 11 to
12)
3-4 yearsReplace flash card Energy Storage Modules (ESMs) in V2/X2 storage
servers
16
Patch PlanningNew Release Adoption
• Planning
• README and referenced Support Notes
• Exachk before and after (MOS 1070954.1)
• Required overlays for critical issues and one-off conflicts
• Bug fix list
e.g. 11.2.0.3 - MOS 1393410.1 11.2.0.2 - MOS 1314319.1
Stage General Recommendation
Production and late pre-production Every patch not mandatory
Wait for field experience to grow
Early pre-production or evaluation Keep current with latest release
9
17
Patch Testing Reduces Risk
• Output of testing
• Application functionality
• Application performance
• Installation method
• Timings
• Rehearsal, documentation, automation
• HA stages during patching
• Failure contingency
Patching Guidelines MOS 1262380.1
18
Test Environment
Test DBM Env Comments
Identical Equivalent to production environmentPrimary database
Standby database
Middle tier
Full data set w/ identical schema stats
Workload framework to mimic production
Smaller or
Older
No production scale performance test
No firmware patching test
None DANGER
RISK
+
10
19
Exadata PatchingPlanning Recommendations
• Current knowledge, current software
• The Patching Note (888828.1)
• Don’t fall far behind
• Mitigate your risks
• Plan to test
• Test your plan
20
Patch Installation
11
21
Patch DeliveryQuarterly Full Stack Download Patch (QFSDP)
• Download convenience
• Contains latest releases
• Infrastructure
• Storage
• InfiniBand switch
• PDU
• Database
• Systems Management
• Release aligned on QDPE (and CPU) schedule
|- Infrastructure/
| --ExadataStorageServer/
| -11.2.2.4.2/
| --InfiniBandSwitchSoftware/
| -1.3.3-2/
| --SunRackIIPDUMeteringUnitFirmware/
| -1.04/
|- Database/
| --11.2.0.3.2/
| --OPatch/
| --OPlan/
|- SystemsManagement/
| --GridControl11/
| --Agent/
| --EMPlugins/
| --OMS/
22
Patch MethodsPatching Tools
• System readiness
• Exachk (MOS 1070954.1)
Component Software Patch Method
Exadata Storage Server All patchmgr
Database Server
OS and firmware Linux – YUM (11.2.3.1)
Solaris – PKG
GI/DB BP OPatch
GI/DB Patch set OUI and DBUA
InfiniBand Switch All ILOM
12
23
Patch MethodsRolling versus Non-Rolling
• Patch non-rolling if maintenance window allows
• Rolling patching considerations
• Patch during low workload
• Cell Rolling Patching – Reduce disk failure risk w/ high redundancy
• DB Rolling Patching - Use Automatic Workload Management
RollingNon-RollingPatch Install Time +-
Complexity +-
Database Downtime+ -
24
• Patch contains all cell software and firmware
• Install nothing but this patch on cells
• Unsupported configuration changes
• Will be lost
• May cause future patching FAILURE
• patchmgr controls install and rollback
• One invocation patches all cells
• Rollback to most recent successfully installed version
Exadata Storage Server Patching
Method Downtime Patch install time
Rolling None Up to 2 hours per cell
Non-Rolling Up to 2 hours total Up to 2 hours total
13
25
Exadata Storage Server PatchingNon-Rolling Patch Apply
• Benefits
• Lowest total patching time
• No risk to single disk failure
• Consider
• Full outage up to 2 hours total
• Failed patch install on multiple
cellspatchmgr
Patch cell1
Patch celln
Patch cell3
Patch cell2
# ./patchmgr -cells cell_group –patch
26
Exadata Storage Server PatchingRolling Patch Apply
• Benefits
• No database downtime
• Not exposed to multiple cell
patch failures
• Consider
• ASM high redundancy to
reduce disk failure exposure
• Up to 2 hours per cell
• BP requirements
patchmgr
ASM offline
Patch
ASM onlinecell1
ASM offline
Patch
ASM onlinecell2
ASM offline
Patch
ASM onlinecell3
ASM offline
Patch
ASM onlinecelln
Prepare cell*
# ./patchmgr -cells cell_group –patch -rolling
14
27
Exadata Storage Server PatchingPatch Rollback
• patchmgr rollback
• To most recent previous (inactive) version
• Timing similar to installation
# imageinfo
Active image version: 11.2.2.2.0.101206.2
…
Inactive image version: 11.2.2.1.0.100927
# ./patchmgr -cells cell_group –rollback [-rolling]
28
Database Server PatchingLinux Operating System
• Oracle Linux
• >= 11.2.3.1
• YUM and ULN
• New image equivalent
• <= 11.2.2.4.2
• Minimal Pack install.sh
• Kernel (Linux), OFED, Firmware, OSW, minor configuration
• Not new image equivalent
• 5.3 to 5.5 – MOS 1284070.1
StopInstances and CRS Patch
StartInstances and CRS
15
29
Database Server PatchingLinux Operating System
• Keep in step with Exadata release
• Oracle Linux
• User-managed backups necessary for rollback
• Avoid oracle-validated rpm
• <11.2.3.1 Use ULN with care
• ofa and megaraid_sas packages require matching kernel
• Responses to common Exadata security scan findings
• MOS 1405320.1
30
Database Server PatchingSolaris Operating System
• Solaris 11
• Support Repository Update (SRU)
• updates all installed packages
• Image Packaging System (IPS)
“pkg update”
• Firmware manual
• Solaris 11 Express upgrade to
Solaris 11 (11.2.3.1)
• Database 11.2.0.3 required
Patch new BE Reboot
OS SRU Index
Solaris 11 1372094.1
# pkg list entire
NAME (PUBLISHER) VERSION IFO
entire 0.5.11-0.151.0.1.10 i--
16
31
Database Server PatchingDatabase and Grid Infrastructure Patches
• Cumulative
• Bug fix list 11.2.0.3 - MOS 1393410.1 11.2.0.2 - MOS 1314319.1
• OPatch installed and rolled back
• Post install SQL required*
• Rollback to base release (same as PSUs)
$ opatch napply <PATCH_LOCATION>
SQL> @rdbms/admin/catbundle.sql exa apply
$ opatch rollback –id <PATCH>
SQL> @rdbms/admin/catbundle_EXA_<SID>_ROLLBACK.sql
32
Database Server PatchingDatabase and Grid Infrastructure Patches
• Reduce downtime and risk
• RAC Rolling Installable
• OPatch Automation
• Out-of-Place patching
• OPlan
• Enterprise Manager
• Data Guard Standby-First
17
33
RAC Rolling Installable
• Benefit
• No database-wide downtime
• Consider
• Use Automatic Workload
Management
• Configure services and
connection failover
• Available 11.2.0.1 BP9*
• Most one-offs
Stop instance
OPatch DB
Stop CRS
OPatch GI
Start CRS
Start instance
node1
nodenP
Stop instance
OPatch DB
Stop CRS
OPatch GI
Start CRS
Start instance
node2
34
OPatch Automation (opatch auto)
• Benefit
• No database-wide downtime
• Single command per node
• Consider
• Mixed 11.2.0.2 / 11.2.0.1
homes (README)
• Available 11.2.0.2 BP2
OPatch
Stop instance
OPatch DB
Stop CRS
OPatch GI
Start CRS
Start instance
node1
OPatch
Stop instance
OPatch DB
Stop CRS
OPatch GI
Start CRS
Start instance
node2
OPatch
nodenP
18
35
Out-of-Place (OOP) Patching
• Benefit
Patch inactive home to reduce risk
1. Clone home (GI or DB)
2. Patch cloned home
3. Switch current home to cloned home (rolling or non-rolling)
• MOS 1136544.1 and OPlan
• EM 12c supports OOP
Performed during normal business
hours, ahead of “patching window”
36
OPlan
• Benefit
• Step-by-step patching instructions for Exadata BPs specific to your
environment
• In-place Non-rolling
• In-place RAC rolling
• Out-of-place (Recommended)
• Available 11.2.0.2 BP2
• MOS 1306814.1 Patch 11846294
19
37
Enterprise Manager Installable
• Benefit
• No database-wide downtime
• EM managed using
Provisioning Pack
• Available 11.2.0.1 BP7
• MOS 1265998.1
• 12c supports OOP
Enterprise Manager
Stop CRS
OPatch GI
OPatch DB
Start CRS
node1
Stop CRS
OPatch GI
OPatch DB
Start CRS
node2
P noden
38
Database Server PatchingDatabase and Grid Infrastructure Bundle Patches
• Use OPlan to generate patching plan
• Patch using Out-of-Place plan
• Rolling or Non-rolling switch to patched home
• Do not hard code ORACLE_HOME in user scripts
20
39
Standby System
• Patch replicated copy and switch
• Reduce risk and downtime
• Patching on standby without impact to primary
• Exadata Storage Server
• Database Server
• Operating System and Firmware (Minimal pack)
• Grid Infrastructure home
• InfiniBand switch
• Data Guard Standby-First Installable qualified BPs only
• Database server - Database home (MOS 1265700.1)
Data Guard
40
Database Server PatchingDatabase and Grid Infrastructure Patch Set Upgrade
• Wait until Exadata validated
• 11.2.0.3
• MOS 1373255.1
• 11.2.0.2
• MOS 1315926.1
• Storage Server compatibility
Database Required Storage
11.2.0.3 11.2.2.4 or later
11.2.0.2 11.2.2 or later
11.2.0.1 11.2.1 or later
• GI upgrade is rolling
• Database upgrade requires
downtime
• Reduce downtime and risk
• (Transient) Logical standby
• Golden Gate
21
41
InfiniBand SwitchFirmware
• Only update via Exadata-branded patch
• 1.0.1 => 1.1.3 => 1.3.3
• Rolling update
• Spine => Leaf => Leaf
• Installed (and rolled back) via ILOM
# ibswitchcheck.sh pre
-> load -source /tmp/sundcs_36p_repository_1.3.3_2.pkg
# ibswitchcheck.sh post
42
Consolidation Environments
• GI version >= DB version
• Required to 4th digit, recommended to 5th digit
• ASM compatible.rdbms
• Once advanced cannot be reset
• Maintain as few shared database homes as needed
• Patch out of place
22
43
Exadata PatchingPatching Recommendations
• Only patch healthy system
• Run Exachk before (and after)
• Patch copy, switch to copy
• Out-of-place
• Standby-First
• Patch non-rolling (if possible)
• Test and rehearse lowers downtime
• Patch rolling (when necessary)
• During low workload
• Reduce disk failure risk w/ high redundancy
• Use Automatic Workload Management
44
Exadata Patching & Upgrades Resources and References
• Latest Exadata software (MOS 888828.1)
• Exadata Critical Issues (MOS 1270094.1)
• Exadata Testing and Patching Practices (MOS 1262380.1)
• OPlan (MOS 1306814.1)
• Data Guard Standby-First Patch Apply (MOS 1265700.1)
• Enterprise Manager Patch Apply (MOS 1265998.1)
• 11.2.0.3 Database Upgrade on Exadata (MOS 1373255.1)
• 11.2.0.2 Database Upgrade on Exadata (MOS 1315926.1)
• Exadata Healthcheck (MOS 1070954.1)
• Patch READMEs and associated MOS
• Database Machine Owner’s Guide (/opt/oracle/cell/doc)