Expert performance tuning tips for Oracle RAC

EXPERT TIPS AND TOOLS FOR MANAGING ORACLE RAC 12CLEIGHTON NELSONLEAD DBA, WORLDWIDE TECHNOLOGIES

2.25.2015

ABOUT ME

• Oracle DBA

• Oracle ACE

• Speaker

• Co-Author – Expert Oracle Enterprise Manager Cloud Control 12c

• RAC Attack Ninja

http://blogs.griddba.com

linkedin.com/pub/leighton-nelson/4/6b8/a7a

@leight0nn

AGENDA

• Oracle RAC 12c Features

• Management Tools

• SRVCTL

• CRSCTL

• Oracle Enterprise Manager

• Support Tools

• ORAchk

• TFA

• ADR

• PDBs and Services in RAC

• What’s New in 12.1.0.2

ORACLE RAC 12C OVERVIEW

• Flex ASM

• Flex Clusters

ORACLE RAC 12C OVERVIEW

Flex ASM

• Pre-12c ASM instances required 1-to-1 mapping to database

instances

• Flex ASM instance not needed on all nodes

ORACLE RAC 12C OVERVIEW

Flex Clusters

• Hub and Leaf Nodes

• Leaf nodes require shared storage

• Requires Flex ASM

• Only Hub node host databases

MANAGEMENT TOOLS

• SRVCTL enhanced to accept full-word options

• New SRVCTL commands have full-word options only

• Existing SRVCTL commands support single letters for backwards

compatibility

• More intuitive

• Single letter options confusing at times

• Inconsistent

• Allows for more options

MANAGEMENT TOOLS

srvctl modify service –s oltp –d orcl –r orcl1 –a orcl2 –P

BASIC –e SELECT –m BASIC –w 60 –j –SHORT –B SERVICE_TIME

MANAGEMENT TOOLS

srvctl modify service –s oltp –d orcl –r orcl1 –a orcl2 –P

BASIC –e SELECT –m BASIC –w 60 –j –SHORT –B SERVICE_TIME

srvctl modify service –service oltp –db orcl –preferred

orcl1 –available orcl2 –tafpolicy BASIC –failovermethod

NONE –timeout 60 –clbgoal SHORT –rlbgoal SERVICE_TIME

• Show single letter options using the –compatibile option after -help

MANAGEMENT TOOLS

• Specify input file for passing lengthy options using –file option

>cat services.in

-service radwwt_basic -db radwwt -serverpool wwt_sp -

cardinality uniform -clbgoal SHORT -rlbgoal SERVICE_TIME -

role PRIMARY -policy AUTOMATIC –verbose

>srvctl add service -file services.in

MANAGEMENT TOOLS

What-If Command Evaluation

• The –eval option allows commands to be simulated without

modifying the system

• Available for both SRVCTL and CRSCTL

• Supported SRVCTL Commands

srvctl add database

srvctl add service

srvctl add srvpool

srvctl modify database

srvctl start database

srvctl start service

srvctl stop database

srvctl stop service

MANAGEMENT TOOLS

What-If Command Evaluation

MANAGEMENT TOOLS

What-If Command Evaluation

MANAGEMENT TOOLS

• Using CRSCTL some attributes can be modified online without a

restart

• Modification of ora.* resources enforced

• crsctl modify ora.* resource fails with CRS-4995 in 12.1.0.2 and above

(Doc ID 1918102.1)

• Only user-defined resources can be modified

• There are exceptions

• CRS_STAT commands deprecated as of 11.2

• Replaced with CRSCTL status resource

MANAGEMENT TOOLS

What if?

MANAGEMENT TOOLS

Show what would happen if a resource failed and cannot be restarted

on the local node

• SRVCTL predict

• CRSCTL eval fail

MANAGEMENT TOOLS

Display status of all databases running running in a particular home

• srvctl status database–DB -thishome

• Display status of all databases running with same version

• srvctl status database–DB -thisversion

MANAGEMENT TOOLS

Oracle Enterprise Manager

• Cloud Control 12c

• EM Express 12c

MANAGEMENT TOOLS

MANAGEMENT TOOLS

Oracle Enterprise Manager

SUPPORT TOOLS

• ORAchk aka RACcheck

• Trace File Analyzer (TFA) Collector

• Automatic Diagnostics Repository (ADR)

SUPPORT TOOLS

• ORAchk provides health checks for Oracle stack

• Databases (Single Instance and RAC)

• Enterprise Manager Repository

• E-Business Suite

• ORAchk now included in 12c binaries

• $ORACLE_HOME/suptools

• Recommended to download latest version from MOS

• ORAchk - Health Checks for the Oracle Stack (Doc ID 1268927.2)

SUPPORT TOOLS

When to run ORAchk

• After Initial Oracle RAC Deployment

• Before maintenance

• After maintenance

• At least once every three months

SUPPORT TOOLS

• Setup ORAChk to check start automatically

>./orachk –auto_restart

• Configure expect utility or password-less sudo

• see http://expect.sourceforge.org

• Configure scheduled collection

>./orachk -id DBA -set

"NOTIFICATION_EMAIL=leighton.nelson@wwt.com;AUTORUN_SCHEDU

LE=4,8,12,16,20 * * *;AUTORUN_FLAGS=-profile

dba;COLLECTION_RETENTION=30"

SUPPORT TOOLS

• Use profiles to run specific checks

• >./orachk –profile <profilename>

SUPPORT TOOLS

Trace File Analyzer

• Installed in $GI_HOME/suptools

• Executed from $GI_HOME/tfa/bin

• Collects diagnostics, traces, log files for DB, Clusterware, OS across

the cluster

• Can be configured to trigger collection automatically

• Starts automatically on reboot

SUPPORT TOOLS

Trace File Analyzer

• Ability to analyze individual components

• $ tfactl analyze –node racdb01,racdb02 -comp crs -since 1h

• Integrates with oratop, oswatcher etc

• Some components require root or sudo access

• Repository stored in GI_BASE/tfa/repository

• Default size of 1GB

SUPPORT TOOLS

Trace File Analyzer

SUPPORT TOOLS

• Automatic Diagnostics Repository for G.I.

• Clusterware log files managed and treated like database related

logs, trace

• $GI_BASE/diag/

• adrci interface

SUPPORT TOOLS

• ADRCI for CRS logs

SUPPORT TOOLS

• ADRCI for CRS logs

SUPPORT TOOLS

Cluster Health Monitor (CHM/OS)

• Introduced since 11.2.0.1

• Collects OS level metrics using OS APIs

• Stores data in management repository (GIMR)

• Integrated with QoS and Oracle Enterprise Manager

SUPPORT TOOLS

Cluster Health Monitor (CHM/OS)

Processes

• osysmond

• ologgerd

• oclumon

Clusterware Resource

$ crsctl status resource ora.crf -init

NAME=ora.crf

TYPE=ora.crf.type

TARGET=ONLINE

STATE=ONLINE on ngdcracdb01

PDBS AND SERVICES IN RAC

• A default service is created with each PDB

• Default service is the same global name as the PDB

• Default service is NOT registered with Clusterware

• Create additional services for application access

• PDBs are mounted by default

• Starting a service opens the PDB

• SINGLETON service opens PDB on the node it runs on

• Use UNIFORM service to access PDB on more than one node

• Stopping a service doesn’t close the PDB

• PDB remains open but inaccessible

• Remember PDBs exist on all RAC nodes running CDB!

• Specify INSTANCES keyword when modifying PDB state

MANAGING PDBS

• SQL Plus

• SQL Developer

• Oracle Enterprise Manager Cloud Control 12c

• Oracle Enterprise Manager Express 12c

WHAT’S NEW IN 12.1.0.2

• Ping Targets

• Probe nodes in Virtual Cluster for network failure

• Detects reachability for targets on public network

• Rapid Home Cloning

• Store gold images of database software in RHP server

• Deploy images to dev/test environments

• ASM Filter Driver

• Consistent naming of ASM disks across cluster (replaces ASMLib)

• Filters I/O to ASM disks to prevent accidental overwriting

WHAT’S NEW IN 12.1.0.2

Ping Targets

• Configured during initial cluster install

• Set after installation using srvctl

[root@racdb01 ~]# srvctl modify network -k 1 -pingtarget "10.255.224.198"

[root@racdb01 ~]# srvctl config network

Network 1 exists

Subnet IPv4: 10.255.224.0/255.255.255.0/eth0, static

Subnet IPv6:

Ping Targets: 10.255.224.198

Network is enabled

Network is individually enabled on nodes:

Network is individually disabled on nodes:

WHAT’S NEW IN 12.1.0.2

Rapid Home Cloning

• Store gold images of database software in RHP server

• rhpctl import image -image DB12102_base -path

/u01/db/oracle/12.1.0.2/db_1/ -imagetype ORACLEDBSOFTWARE

• Deploy images to dev/test clusters

• rhpctl add workingcopy -workingcopy DB12102Home -image

DB12102_base

• Fast and standardized deployment of Oracle binaries

WHAT’S NEW IN 12.1.0.2

ASM Filter Driver

• The verdict is still out

• http://flashdba.com/2014/07/25/oracle-12-1-0-2-asm-filter-driver-first-

impressions/

REFERENCES

• Oracle RAC 12c

http://www.oracle.com/technetwork/database/options/clustering/rac-

wp-12c-1896129.pdf

• Oracle® Clusterware Administration and Deployment Guide 12c

Release 1 (12.1)

• Oracle RAC 12c Operational Best Practices

http://www.slideshare.net/MarkusMichalewicz

• Oracle RAC, Oracle Data Guard, and Pluggable Databases: When

MAA Meets Oracle Multitenant

http://www.ludovicocaldara.net/dba/maa-multitenant-presentation/

FREE TRIAL--RESOLVE PERFORMANCE

ISSUES QUICKLY

• Try Database Performance Analyzer FREE for 14 days

• Insight into overhead caused by delays in RAC coordination and

cache operations

• Improve root cause of slow performance

• Quickly identify root cause of issues that impact end-user response time

• See historical trends over days, months, and years

• Understand impact of VMware® performance

• Agentless architecture, installs in minutes

41© 2015 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.

www.solarwinds.com/dpa-download/

QUESTIONS?

42© 2015 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.

THANK YOU

© 2015 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.

The SOLARWINDS and SOLARWINDS & Design marks are the exclusive property of SolarWinds Worldwide, LLC, are registered with the U.S.

Patent and Trademark Office, and may be registered or pending registration in other countries. All other SolarWinds trademarks, service marks,

and logos may be common law marks, registered or pending registration in the United States or in other countries. All other trademarks

mentioned herein are used for identification purposes only and may be or are trademarks or registered trademarks of their respective companies.

43