A kind and gentle introducton to rac

©OraInternals Riyaj Shamsudeen

A kind and gentle intro to RAC

By Riyaj Shamsudeen

©OraInternals Riyaj Shamsudeen 2

Who am I?

  19 years using Oracle products/DBA

  OakTable member

  Oracle ACE Director

  Certified DBA versions 7.0,7.3,8,8i,9i &10g

  Specializes in RAC, performance tuning, Internals and E-business suite

  Chief DBA with OraInternals

  Co-author of “Expert Oracle Practices” ‘2009

  Co-author of “Pro Oracle SQL” ‘2010

  Email: rshamsud@orainternals.com

  Blog : www.orainternals.com/blog

  URL: www.orainternals.com

©OraInternals Riyaj Shamsudeen 3

Outline

  RAC architecture

  Good Reasons to RAC   Not-so-good reasons to RAC

  Srvctl commands and demo

  Cache fusion

©OraInternals Riyaj Shamsudeen 4

Shared Nothing vs Shared Everything

  In a shared nothing architecture, obviously, nothing is shared. In a shared everything architecture everything is shared.

  Oracle RAC uses Shared everything architecture, where as DB2 uses shared nothing architecture.

Node1 Node2 Node3 Node1 Node2 Node3

UDB2 RAC

  In RAC, loss of one node does not affect data availability.

©OraInternals Riyaj Shamsudeen 5

Good reasons

  Hardware fault tolerance

  Workload segregation   Application affinity

  To manage excessive redo generation

  To avoid SMP bottlenecks

©OraInternals Riyaj Shamsudeen 6

Availability

  RAC (non-Stretch ) provides hardware level fault tolerance.

  RAC + Data Guard is a good disaster recovery solution, but RAC alone is not a good DR solution.

  RAC as a DR solution does not protect from site disaster, human errors, or corruption (Both hardware and Software corruption).

©OraInternals Riyaj Shamsudeen 7

Not-So-Good reasons

  General Performance improvements

  To combat poor application design and coding practices

  RAC as a stand-alone Disaster Recovery solution   To maximize use of hardware

  Stretch cluster to enhance hardware usage

©OraInternals Riyaj Shamsudeen 8

Clusterware

  Clusterware is an integral component of RAC architecture.

  Clusterware provides node membership, monitoring, and node management.

  Few daemons run with root ownership.

  Databases, listeners, ASM instances are simply resources for clusterware.

  These resources are managed by clusterware for high availability reasons.

Demo: clusterware daemons, commands etc

©OraInternals Riyaj Shamsudeen 9

Instance vs Database

  Instance is defined as a set of memory structures (SGA) and processes.

  Database is defined as permanent structures such as data files.   Instance is transient and the database is permanent.

  In a single instance database, there is a one-to-one relationship between an instance and a database.

  In RAC database, multiple instances can mount one database concurrently.

  Resources are globalized in RAC database.

Demo: instance, database etc.

©OraInternals Riyaj Shamsudeen 10

Architecture

Inst 1 Inst 2 Inst 3

SAN switch

Private network switch

Non-routeable network For interconnect

©OraInternals Riyaj Shamsudeen 11

Redo, undo and temp

  Each instance has its own redo thread. You need to allocate at least a set of log file groups for a redo thread.

  Each instance uses its own undo tablespace. An undo tablespace must be assigned to each instance.

  TEMP tablespace is shared between the instances, but, we will see that sharing is at extent level later.

  Control files, data files are shared between the instances.

  Each instance has its own SGA.

  So, adding a new instance means that you need to add a redo thread, an undo tablespace, and possibly increase temporary tablespace etc.

©OraInternals Riyaj Shamsudeen 12

Private Interconnect

  Use teaming, bonding or aggregation to bond network cards in to one logical entity [ or 11gR2 haip feature].

  Loss of one network cards should not affect heart beats between the nodes.

  Use Jumbo frame only if complete path (routers and other network hardware) supports Jumbo frames.

  Use MTU of 9000 for jumbo frames and other MTU sizes are not really tested in most cases.

  Jumbo frames decrease CPU usage as the need for assembly and disassembly decreases. But, network hardware must be designed to support Jumbo frames.

  Keep the IP addresses private and non-routable.

©OraInternals Riyaj Shamsudeen 13

Initialization parameters

  There are two distinct class of parameters:

  Parameters which must be the same in all instances

  Parameters that can be different between the instances.

  V$parameter.isinstance_modifiable (11g) indicates whether the parameter must be alike or not.

  All parameters with isinstance_modifiable=FALSE must be alike in all instances. In 11gR2, 115 parameters must be alike and 230 parameters can be different.

©OraInternals Riyaj Shamsudeen 14

Parameter changes

  To change parameter specific to an instance:

Alter system set <parameter>=‘value’ scope=both sid=‘inst1’;

  To change parameters in all instances: Alter system set <parameter>=‘value’ scope=both sid=‘*’;

  If a parameter is specified with both * and instance_name, then the parameter with the instance_name has higher precedence.

  For example, with the parameter set below, inst1 will have 40G db_cache_size and other instances will have 38G db_cache_size:

inst1.db_cache_size = 40G;

*.db_cache_size = 38G;

©OraInternals Riyaj Shamsudeen 15

Command syntax

srvctl <action> <objects> <options>

  action is the action to be taken, such as start, stop, enable, disable, relocate etc.

  Object is the type of object such as database, listener, vip etc.

  Options specifies the details about object itself such as database_name, instance_name, listener_name etc.

  Options also allows you to specify more options such as startup _mode etc.

©OraInternals Riyaj Shamsudeen 16

Srvctl - database

srvctl start database –d solrac –o open,pfile=initsolrac.ora

  Above command starts all instances configured with that database.

  You can specify various modes such as restrict, mount etc as option.

  Open mode is optional and is default. Database will use configured spfile if no pfile specified in the command line.

©OraInternals Riyaj Shamsudeen 17

Cache coherency

  There are multiple buffer caches in an instance and Oracle RAC uses shared everything architecture.

  Cache coherency is the method by which consistency of the database is maintained.

  Only one instance can hold a block in current mode in exclusive mode and a block can be modified only if the block is held in exclusive current mode.

  There can be two pending transactions modifying the same block, but a block can only be held in exclusive mode in an instance.

©OraInternals Riyaj Shamsudeen 18

Cache fusion: Read-Read

Disk files

[0x1ad3][0x7],[BL]

PR mode lock

Master node for the resource

PR BL PR

FG LMS

PR

FG LMS

[0x1ad3][0x7],[BL]

PR mode lock

1. FG requests LMS process for a block.

2. LMS process finds a block in the buffer cache.

3. LMS process sends a block to the FG process through interconnect.

4. LMS also sets up GRD for the BL. 5. FG copies the buffer to buffer cache.

1

2

3

5

FG – Foreground Process LMS – GCS Server process

GRD – Global Resource Directory BL – Buffer Lock

PR – Protected Read

©OraInternals Riyaj Shamsudeen 19

Cache fusion: Write-Read

Disk files

[0x1ad3][0x7],[BL]

EX mode lock

Master node for the resource

FG – Foreground Process LMD – Lock Manager Daemon

GRD – Global Resource Directory

EX BL EX

FG LMS

FG LMS

CR copy

No local locks

In this case, FG asks for a Specific version of the block.

CR copy

1

2 3

©OraInternals Riyaj Shamsudeen 20

Row level locking

  A key advantage of RAC is that it doesn’t affect row level locking.

  We know that a row can not be modified by two sessions concurrently in a single instance database. This is true in RAC also.

  But, two sessions can modify rows in the same block, even from different instances.

  All Oracle features such as transaction integrity, transaction consistency, row level locking mechanism etc still maintained.

©OraInternals Riyaj Shamsudeen 21

Row level locking

Disk files

100

101

102

100

101

102

100

101

102

Session 1 Session 2 Session 3

Three sessions are modifying three rows in a block concurrently.

Inst1 Inst2 Inst3

Demo:demo_upd_rows_a.sql, demo_upd_rows_b.sql, demo_upd_rows_c.sql

©OraInternals Riyaj Shamsudeen 22

Affinity is still important!

  Workload segregation should be at the physical segment level, not just logical level.

  A batch process was designed to accept organization_id as an argument, and spawns multiple threads for each organization.

  This is multi-node RAC cluster and so

  Batch process was started in node 1 for org_id=1.

  Same batch process was started in node 2 for org_id=2.

  But batch code accesses just single set of tables and they are not partitioned.

©OraInternals Riyaj Shamsudeen 23

Affinity?

Inst 1 Inst 2 Inst 3

Batch processes for org_id=2

t1 t1 t1

CR mode CUR mode

Batch processes for org_id=1

But, same blocks are accessed And modified from both nodes, Increasing GCS waits and GCS Communication.

©OraInternals Riyaj Shamsudeen 24

What happened?

  Enormous increase in Global cache waits!

(5 minutes statspack) Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Ela Time

-------------------------------------------- ------------ ----------- -------- CPU time 3,901 46.17

PL/SQL lock timer 80 2,316 27.42 global cache cr request 123,725 670 7.93 global cache null to x 13,551 446 5.28

buffer busy global CR 12,383 215 2.54

  If this has been a non-RAC, access to buffers would be in terms of nano seconds instead of milli-seconds.

  Point is that since these batch processes are accessing same physical blocks, they need to be grouped at physical level and scheduled to run from one node.

  Proper partitioning will help. But pay attention to Global indexes.

©OraInternals Riyaj Shamsudeen 25

Load balancing & Failover

  Load balancing is a technique in which we try to utilize all nodes in a cluster equally.

  Failover is a technique in which session fails over to a different node at the onset of a node failure.

  Load balancing and Failover can be implemented together, or independent of each other.

  In most cases, both will be implemented together.

©OraInternals Riyaj Shamsudeen 26

Overview

  There are are many techniques available to implement failover and load balancing in various layers.

  Few techniques:

  Client Side load balancing

  Load balancing with SCAN IP   Server side load balancing

  VIPs with client side load balancing

  TAF – Transparent Application Failover

  Services based load balancing and failover.

  Combinations of above technique.

Review tnsnames.ora file

©OraInternals Riyaj Shamsudeen 27

SCAN addresses

SCAN-1 Listener

SCAN-2 Listener

SCAN-3 Listener

VIP-1 Listener

VIP-2 Listener

VIP-3 Listener

Server process

Server process

VIP-4 Listener

©OraInternals Riyaj Shamsudeen 28

Stretch RAC

  Stretch RAC means that nodes are physically separated by many miles.

  Both GES/GCS traffic latency can lead to application performance latency issues.

  Beware that as the distance between the nodes increases, latency increases and can lead to disastrous performance consequences.

  Dark fibers are must if you exceed few miles.

  Use third site for voting disks to detect split brain and communication failures between the sites.

©OraInternals Riyaj Shamsudeen 29

Architecture

Inst 1 Inst 2

SAN switch

Network switch

Network Switch

SAN switch

Voting disk

Site 1 Site 2 Site 3

FG1

Public Network or Dark Fibers

FG2

FG: Failover group

©OraInternals Riyaj Shamsudeen 30

Preferred mirror read

  Oracle 11g introduced preferred mirror read.

  ASM will try to read primary group extents.

  In extended RAC, since one of the failover group is local, it is preferable to read the extents from that local failover group.

  Parameter asm_preferred_read_failure_groups control the failover group to use for reading the extents.

  DB instance will read from local extents avoiding latency.

©OraInternals Riyaj Shamsudeen 31

Thank you for attending!

If you like this presentation, you will love my upcoming intensive RAC webinar

in 2012. Watch for updates in:

Orainternals.wordpress.com