Pro Oracle Database ^0g RAC on Linux Installation, Administration, and Performance

Julian Dyke and Steve Shaw

Apress*

Pro Oracle Database lOgRAC on Linux: Installation, Administration, and Performance

Copyright © 2006 by Julian Dyke and Steve Shaw

All rights reserved. No part of this work may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval system, without the prior written permission of the copyright owner and the publisher.

ISBN-13 (pbk): 978-1-59059-524-4

ISBN-10 (pbk): 1-59059-524-6

Printed and bound in the United States of America 9 8 7 6 5 4 3 2 1

Trademarked names may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, we use the names only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark.

Lead Editors: Matthew Moodie, Tony Davis Technical Reviewers: Kevin Closson, Wallis R. Pereira, Brett A. CoUingwood Editorial Board: Steve Anglin, Ewan Buckingham, Gary Cornell, Jason Gilmore, Jonathan Gennick,

Jonathan Hassell, James Huddleston, Chris Mills, Matthew Moodie, Dominic Shakeshaft, Jim Sumser, Keir Thomas, Matt Wade

Project Managers: Elizabeth Seymour, Beckie Brand Copy Edit Manager: Nicole LeClerc Copy Editors: Heather Lang, Nicole LeClerc, Jennifer Whipple Assistant Production Director: Kari Brooks-Copony Production Editor: Kelly Winquist Compositor: Kinetic Publishing Services, LLC Proofreaders: Lori Bring, Linda Seifert Indexer: Broccoli Information Management Artist: Kinetic Publishing Services, LLC Cover Designer: Kurt Krames Manufacturing Director: Tom Debolski

Distributed to the book trade worldwide by Springer-Verlag New York, Inc., 233 Spring Street, 6th Floor, New York, NY 10013. Phone 1-800-SPRINGER, fax 201-348-4505, e-mail orders-ny@springer-sbm.com, or visit http://www.springeronline.com.

For information on translations, please contact Apress directly at 2560 Ninth Street, Suite 219, Berkeley, CA 94710. Phone 510-549-5930, fax 510-549-5939, e-mail inf o@apress. com, or visit http: //www. apress. com.

The information in this book is distributed on an "as is" basis, without warranty. Although every precaution has been taken in the preparation of this work, neither the author (s) nor Apress shall have any liability to any person or entity with respect to any loss or damage caused or alleged to be caused directly or indirectly by the information contained in this work.

The source code for this book is available to readers at http: //www. apress. com in the Source Code section.

Contents at a Glance

About the Authors xxi

About the Technical Reviewers xxii

Acknowledgments xxiii

Introduction xxv

CHAPTER 1 Introduction to RAC 1

CHAPTER 2 Introduction to Linux 21

CHAPTER 3 RAC Concepts 31

CHAPTER 4 RAC Design 59

CHAPTER 5 Benchmarking Concepts 83

CHAPTER 6 Hardware 99

CHAPTER 7 Storage 125

CHAPTER 8 Oracle Cluster File System (OCFS) 171

CHAPTER 9 Automatic Storage Management (ASM) 20i

CHAPTER 10 Installing Linux 235

CHAPTER 11 Configuring and Verifying Linux 259

CHAPTER 12 Installing and Configuring Oracle Software 301

CHAPTER 13 Creating and Configuring Databases 327

CHAPTER 14 Installing and Configuring Enterprise Manager 363

CHAPTER 15 Linux Administration 385

CHAPTER 16 Oracle Clusterware 401

CHAPTER 17 Application Development 425

CHAPTER 18 Using Administrative Tools 465

CHAPTER 19 Workload Management 493

CHAPTER 20 Adding and Deleting Nodes and Instances 521

CHAPTER 21 Trace and Diagnostics 553

CHAPTER 22 RAC Internals 575

CHAPTER 23 Benchmarking Tools: Hammerora and Swingbench 607

CHAPTER 24 Performance Monitoring 649

CHAPTER 25 Backup and Recovery 683

CHAPTER 26 Disaster Recovery 717

APPENDIX Third-Party Clustering Solutions 743

• N D E X 753

Contents

About the Authors xxi

About the Technical Reviewers xxii

Acknowledgments xxiii

Introduction xxv

•CHAPTER 1 Introduction to RAC i

What Is Real Application Clusters? 1

Clustered Database Concepts 2

Cluster Terminology 2 High Availability 4 Scalability 5 Disaster Recovery 7

Why Deploy RAC? 7 Cost of Ownership Advantages and Benefits 8

High Availability Advantages and Disadvantages 10 Scalability Advantages and Disadvantages 12 Manageability Advantages and Disadvantages 14 Transparency Advantages and Disadvantages 15

Alternatives to RAC 15 Single Instance Databases 16 Data Guard 16 Third-Party Clustering Solutions 16

Differences Between Single-Instance and RAC Databases 16 Differences Between Oracle 9/and Oracle lOg 17 History of Real Application Clusters 19 Summary 20

•CHAPTER 2 Introduction to Linux 21

History of Linux 21 UNIX 21 Free Software 22 Linux 23 Open Source 24

^CONTENTS

Oracle, Open Source, and Linux 25 Unbreakable Linux 25

Red Hat Enterprise Linux 26 SUSE Linux Enterprise Server 27

Asianux 29 Summary 29

KHAPTER 3 RAC Concepts 31

Oracle Clusterware 31 Oracle Cluster Registry 32 Voting Disk 33

Node Applications 33 Listener 33

Oracle Notification Service (ONS) 33 Virtual IP (VIP) 34

Global Services Daemon (GSD) 34

Database Structure 34 Datafiles 34 Control Files 35 Online Redo Log Files 35 Undo Tablespaces 36 Archived Logs 36 Recovery Area 36

Instances 36

Global Cache Service (GCS) 37 Global Enqueue Service (GES) 37

Parameters 37 Initialization Parameter File (PFILE) 38 Server Parameter File (SPFILE) 39 Global Parameters 40 Instance-Specific Parameters 40 RAC-Specific Parameters 40

Background Processes 42 RDBMS Instance Background Processes 42 ASM Instance Background Processes 47

Workload Management 48 Services 48 Connection Load Balancing 49 Transparent Application Failover (TAF) 49

Administration 50 Enterprise Manager 50 Server Control Utility (SRVCTL) 51 SQLTIus 51 CRSCTL 51

CONTENTS vii

Monitoring 51

Operating System Utilities 51

Enterprise Manager 52

Automatic Worl

VIII CONTENTS

Application Design 70 Bind Variables 70

Sequences 70 Database Design 72

Cursor Sharing 72

Optimizer Statistics 72

Histograms 73

Dynamic Sampling 73 System Statistics 73 Locally Managed Tablespaces 74 Automatic Segment Space Management (ASSM) 74

Reverse Key Indexes 75 Partitioning 76

Summary 81

CHAPTER 5 Benchmarking Concepts 83

Load Testing and Benchmarks 83

TPC Benchmarks 86 TheTPC-C Benchmark 86 The TPC-H Benchmark 91

Summary 98

CHAPTER 6 Hardware 99

Oracle Availability 100 Server Architecture 101

Processors 101 Memory 112 Additional Platform Features 115

Network and Private Interconnect 117 Network I/O 118 Private Interconnect Selection 119 Fully Redundant Ethernet Interconnects 121

Summary 124

CHAPTER 7 Storage 125

RAG I/O Characteristics 125 Read Activity 125

Write Activity 126 Forced Reads and Writes 128 Asynchronous I/O 128 Direct I/O 129

CONTENTS

Storage Fundamentals 130 Disk Drive Performance 130 RAID 131 RAID 0 Striping 131 RAID 1 Mirroring 132

RAID 10 Striped Mirrors 133

RAID 0+1 Mirrored Stripes 133

RAIDS 134

Storage Cache 135 RAID Summary 136 Intelligent Storage 136

Storage Protocols for Linux 138

SCSI 139 Fibre Channel 140 iSCSI 141 S-A17\ 142

Using Block-Based Storage 143 Linux I/O Scheduling 143

NAS 144 SAN and NAS Compared 144

SAN Storage Example 146 PowerPath 147 HBA and Driver Configuration 149 Fibre Channel Switch Configuration 151 EMC CLARiiON Configuration 152

NFS and iSCSI Storage Example 158 NetApp Filer RAID Configurations 158 Aggregates and Volumes 159 NFS 160 Snapshot Backup and Recovery 164 iSCSI 166

Summary 170

CHAPTER 8 Oracle Cluster File System (OCFS) i7i

Cluster File Systems and OCFS 171 OCFS Version 1 172

OCFS Design Considerations 173 Installation and Configuration 175

Tools and Utilities 179

Debugging and Tracing OCFS 186 Operating System Utilities 188

•CONTENTS

OCFS Version 2 (0CFS2) 188 Installation and Configuration 188

Tools and Utilities 197 Summary 200

CHAPTER 9 Automatic storage Management (ASM) 201

Introducing ASM 201 ASM Installation 202

configure 203 start 204 stop 204

status 204

createdisk 205

scandisks 205

listdisks 205 querydisk 205

deletedisk 205

ASM Instance Configuration 206 Adding an ASM Instance Using DBCA 206 Adding an ASM Instance Manually 210 Deleting an ASM Instance Using DBCA 211

Deleting an ASM Instance Manually 211 Re-Creating the ASM Instance 212 Using Files Instead of Devices 212

ASM Components 212 ASM Instance 213 ASM Files 217

ASM Administration 220 ASM Administration Using DBCA 221 ASM Administration Using SQL*Plus 221 ASM Administration Using SRVCTL 221 ASM Administration Using Enterprise Manager 222 ASM Administration Using F P 223 ASM Administration Using ASMCMD 224

Monitoring ASM 231

Dynamic Performance Views 231 Fixed Views 231

Summary 233

CHAPTER 10 Installing Linux 235

Linux Software Selection 235 Hardware Requirements 236 Networking Requirements 236

CONTENTS

Installing Red Hat Enterprise Linux 4 AS 237 Starting the Installation 237 Installation Media Check 238 Anaconda Installation 238

Language Selection 238

Keyboard Configuration 238

Upgrade Examine 238 Disk Partitioning 239 Boot Loader Configuration 243 Network Configuration 243

Firewall Configuration 244 Additional Language Support 244 Time Zone Selection 244

Root Password Configuration 245

Package Installation Defaults 245 Package Group Selection 245

Start Installation 247

Installing Packages 247

Installation Complete 247

Welcome 247 License Agreement 247 Date and Time 248 Display 248 Red Hat Login 249

Why Register? 249 System User 249 Additional CDs 249 Finish Setup 249 Manual Package Installation 249

Installing SUSE Linux Enterprise Server 250 Starting the Installation 250 License Agreement 251 Select Your Language 251 Previous Installation 251 Installation Settings 251 Suggested Partitioning 251 Preparing Hard Disk 251

Expert Partitioner 253

Software Selection 253 Time Zone 255 Warning 256 Root Password 256 Network Configuration 256 Test Internet Connection 257

xii M^ONTENTS

Service Configuration 257

User Authentication Method 257

Add a New Local User 257

Release Notes 258

Summary 258

CHAPTER 11 Configuring and Verifying Linux 259

Operating System and RPM Package Checks 259 Hostnames and Name Resolution 260 NTP 261

Hangcheck-Timer 263

Kernel Parameters 264

Shared Memory 265

Semaphores 268 Network 269

Open Files 271 Kernel Parameter Example 271

Oracle User Configuration 272 Creating the dba and oinstall Groups 272

Creating the oracle User 272

Setting the Password for the oracle User 273 Setting Environment Variables 273 Creating the Oracle Software Directories 278 Setting Security Limits 278

Secure Shell Configuration 279 Shared Storage Configuration 281

Partitioning 281 Preparing the Oracle Clusterware Partitions 285

Channel Bonding 287 Cluster Verification 290

Java Runtime Environment 291 Syntax 292

Stage Checks 292

Component Checks 295 Summary 299

BHAFTER 12 Installing and Configuring Oracle Software 301

Preparing to Install 301 Installation Media 302

Oracle Clusterware 303 Preinstallation Tasks 303 Verifying the Configuration 304

IBONTENTS XIII

Running the Installer 305 Verifying the Oracle Clusterware Installation 314

Oracle Database Software Installation 315 Verifying the Configuration 315 Running the Installer 317

Troubleshooting 320

Cluster Verification Utility 321

Common Problems 321 Other Information Sources 322

Configuration Files 323

Inventory 323 OCR 324 /etc/inittab 324

Deinstallation 324 Summary 325

IICHAFTER 13 Creating and Configuring Databases 327

Using DBCA As a GUI Tool 327

Configuring ASM 329

Creating a Database 332 Deleting a Database 345 Creating a Template 345 Managing Instances 349 Managing Services 349

Using DBCA at the Command Line 350 Creating a Database Using Scripts 351

Primary node 352 Secondary Nodes 360 Running the Scripts 361

Summary 362

ICHAPTER14 Installing and Configuring Enterprise Manager 363

Manageability Infrastructure 364 Database Control 365

Grid Control 366 Planning 367

Preparation 371

Management Service Installation and Configuration 372

Starting and Stopping the Management Service 380 Management Agent Installation 381

Summary 384

XIV •CONTENTS

CHAPTER 15 Linux Administration 385

Run Levels 385

Services 387

Terminal Sessions 388 Manual Pages 388 Bash Shell 389

Commands 392 Packages 393 Kernel Parameters 394 Swap Space and File Systems 395

Swap Space 395 General File Systems 396 NFS 397

Booting the System 398

Bootloaders 398

Initial Ramdisk Image 399 Log Files, Tracing, and Debugging 399

Log Files 400 Tracing 400 Debugging 400

Summary 400

CHAPTER 16 Oracle Clusterware 401

Oracle Clusterware Components 401 Cluster Synchronization Services (CSS) 402 Cluster Ready Services (CRS) 402 Event Manager (EVM) 402

Oracle Clusterware Files 402 Oracle Cluster Registry (OCR) 402 Voting Disk 403

Oracle Clusterware High Availability Framework 403 Oracle Clusterware Installation and Configuration 404

Virtual Internet Protocol Configuration Assistant (VIPCA) 404

OIFCFG 407

Administering Oracle Clusterware 408 CRSCTL 408 CRS_STAT 413 OCRCONFIG 416 OCRCHECK 421 OCRDUMP 422

Administering Voting Disks 422

•CONTENTS XV

Oracle Clusterware Logging 423

Summary 423

CHAPTER 17 Application Development 425

RAC-Specific Development Considerations 425 Instances and Database Services 425

Multiple SGAs 426

Local and Shared Storage 427

Node Affinity 428 Transparent Application Failover 430

Fast Application Notification 437

Oracle Notification Service 439 Server-Side Callouts 441 Fast Connection Failover 442

Oracle Clusterware High Availability Framework 446 Example Application 447 Implementing the HA Framework 452 HA Framework Commands 454

Summary 463

CHAPTER 18 Using Administrative Tools 465

Using EM 465

Starting and Stopping Databases and Instances 467

Setting Parameters 470 Performance Options 471 Administrative Options 472 Maintenance Options 474

Using SRVCTL 475 SRVCTL Syntax 476 Administering Databases 477 Administering Instances 480 Administering Node Applications 483 Administering the Listener 486

Using SQL*Plus 487 Setting the SQL*Plus Prompt 487 Connecting to a RAC Instance Using SQL*Plus 487 Starting an Instance 488 Stopping an Instance 488

Setting Parameters in an Initialization Parameter File (PFILE) 489 Setting Parameters in a Server Parameter File (SPFILE) 490

Summary 492

CONTENTS

CHAPTER 19 Workload Management 493

Workload Distribution 493 Client-Side Connection Balancing 494 Server-Side Connection Balancing 494

Database Services 497 Implementing Database Services 499 Administering Database Services 501 Monitoring Services 511

Summary 519

CHAPTER 20 Adding and Deleting Nodes and Instances 521

Running GUI Tools 521 Adding a Node 522

Plan the Installation 522 Install and Configure the New Hardware .523 Configure the Network 523 Install the Operating System 523 Configure the Storage 524 Install Oracle Clusterware 525 Install Oracle Database Software 528 Configure the Listener 530

Adding an Instance 535 Deleting an Instance 540 Deleting a Node 544

Delete ASM Instance 544 Delete the Listener Process 544 Delete the Oracle Database Software 547 Update Inventories on Remaining Hosts 548 Remove Node-Specific Interface Configuration 548 Disable Oracle Clusterware Applications 549 Delete the Node from the OCR 550 Delete the Oracle Clusterware Software 550 Update the Inventories on the Remaining Hosts 551 Verify Node Deletion Using CLUVFY 552

Summary 552

CHAPTER 21 Trace and Diagnostics 553

Trace File Locations 553 Alert Log 553 Trace Files 554

CONTENTS XVII

DBMS_MONITOR 555 Tracing Applications 556 Tracing Multitier Applications 557 DBA_ENABLED_TRACES 558

ORADEBUG 559 LKDEBUG 563

Events 572 Summary 574

CHAPTER 22 RAC Internals 575

Global Enqueue Services 576 Background Processes 576 Resources and Enqueues 576

Lock Types 577 Library Cache Locks 577 Row Cache Locks 580

Tracing GES Activity 581

Optimizing Global Enqueues 581

Global Cache Services 581 Global Resource Directory (GRD) 582

Cache Coherency 584 Cache Fusion 584 Cache Fusion Examples 586 Disk Writes 599

System Change Numbers (SCNs) 603 Optimizing the Global Cache 604

Instance Recovery 604 Summary 605

CHAPTER 23 Benchmarking Tools: Hammerora and Swingbench 607

Hammerora 607 Installation 608 Bespoke Application Load Tests 610 TPC-C Simulation 622

Swingbench 633 Installation and Configuration 633 Calling Circle 635

Developing Swingbench Benchmarks 644 Summary 647

xviii CONTENTS

KHAPTER 24 Performance Monitoring 649

Oracle Performance Monitoring 649

Performance Monitoring with Enterprise Manager 649

AWR Reports 652

Active Session History 660

Automatic Database Diagnostic Monitor 661

Performance Monitoring Using SQL*Plus 662 GV$ Views 663 System Statistics 663 Segment Statistics 663 Global Cache Services 664 Global Enqueue Service 669 Library Cache 669 Dictionary Cache 670 Lock Conversions 671

Linux Performance Monitoring 672

ps 672 free 673 top 675 vmstat 676

Summary 681

KHAPTER 25 Backup and Recovery 683

Backup and Recovery Strategy 683 Recovery Manager (RMAN) 685

RMAN Utility 686 RMAN Repository 689 Backup Sets 692 Tags 693 Parameters 693 Flashback Recovery Area 698 Performing Backups 700

Image Copies 703 Incremental Backups 705 Block Change Tracking 706

Incrementally Updated Backups 707 Housekeeping 707 Performing a Restore 710 Performing Recovery 712

RAC Considerations 714 Archived Redo Logs 715

Summary 716

IKONTENTS xix

•CHAPTER 26 Disaster Recovery 717

Oracle Data Guard 718 Data Protection Modes 719

Maximum Performance Mode 719 Maximum Availability Mode 719 Maximum Protection Mode 719

Setting the Data Protection Mode 720

Redo Transport Services 720

ARCn Background Process 721

LGWR Background Process 721 Asynchronous Network I/O (ASYNC) 721

Synchronous Network I/O (SYNC) 722

Standby Redo Logs 722 Log Apply Services 723

Role Management 723

Read-Only Mode 723 Switchover 723

Failover 724

RAG and Physical Standby 724 RMAN 725

NOLOGGING Attribute 725

Archive Log Gaps 726 Initialization Parameters 726

LOG_ARCHIVE_DEST_n 726 LOG_ARCHIVE_DEST_STATE_n 727 LOG_ARCHIVE_START 727 LOG_ARCHIVE_FORMAT 727 DB_UNIQUE_NAME 728 LOG_ARCHIVE_CONFIG 728 REMOTE_ARCHIVE_ENABLE 728 STANDBY_ARCHIVE_DEST 728 STANDBY_FILE_MANAGEMENT 729 FAL_CLIENT 729 FAL_SERVER 729

Creating a Physical Standby Database 729 Enable Archiving on the Primary Database 730

Create Password Files on Primary Nodes 731 Force Logging on the Primary Database 732

Back Up the Primary Database 732 Set Parameters on the Primary Database 733 Create Directories on Standby Nodes 733 Create Password Files on Standby Nodes 734 Create Server Parameter File for Standby Database 734

XX IKONTENTS

Create Initialization Parameter Files on Standby Nodes 734 Copy the RMAN Backup from Primary to Standby 735 Update /etc/oratab on Standby Nodes 735 Add the Standby Database to the OCR 735 Update the Listener Configuration Files on Standby Nodes 735 Update Oracle Net Configuration Files on All Nodes 736 Set Parameters on the Standby Database 737 Create Standby Database 737 Enable Managed Recovery on the Standby Database 738 Check the Standby Configuration 738 Verify Log Transportation 738

Role Management 739 Read-Only Mode 739 Switchover 739 Failover 740

Summary 741

APPENDIX Third-Party Clustering Solutions 743

Clusterware 743 Third-Party Certification Programs 745

PolyServe Matrix Server 745 Central Cluster Management Console 747 SAN Management Layer 747 Sophisticated Fencing 747 Dynamic Volume Manager 748 CFS 748 Matrix Server ODM 748

Summary 751

INDEX 753

About the Authors

JULIAN DYKE is an independent consultant specializing in Oracle Data-base technology. He has over 20 years of database experience, including more than 15 years as an Oracle DBA, developer, and consultant. He is chair of the UK Oracle User Group Real Application Clusters Special Interest Group (UKOUG RAG SIG) and a member of the Oak Table Network. He regularly presents at conferences, seminars, and user-group meetings in the UK, Europe, and the US. He also maintains www.juliandyke.com, which specializes in Oracle diagnostics, opti-mization, and internals. He is an Oracle Certified Professional and holds a bachelor of science degree in computation from the University of Manchester Institute of Science and Technology (UMIST), UK.

STEVE SHAW is the lead Oracle technologist for Intel Corporation in EMEA (Europe, the Middle East, and Africa). Steve has over 12 years of commercial IT experience with 8 years dedicated to working with the Oracle Database, including a period of time with Oracle Corporation. Steve is the author of Hammerora, the open source Oracle load-test tool, and an expert on Oracle benchmarks and performance. Steve has contributed articles to many Oracle publications and web sites and presents regularly at Oracle seminars, conferences, and special-interest group meetings. He is an Oracle Certified Professional and holds a mas-ter of science degree in computing from the University of Bradford, UK.

About the Technical Reviewers

KEVIN CLOSSON is a chief software architect at PolyServe, Inc. His 18-year career has included engineering, technical marketing, and support positions specializing in Oracle and clustered platforms. Kevin's positions within PolyServe, IBM, Sequent, and Veritas focused on the scalability and availability of the Oracle server on high-end SMP and clustered systems. Kevin holds patents for SMP locking algorithms and database caching methods. He is a frequent speaker at Oracle user groups and a member of the Oracle Oak Table Network. In addition to book collab-orations, Kevin's written work has appeared in Oracle Magazine, Oracle Internals Magazine, IBM Redbooks, and SELECT.

In a previous life, WALLY PEREIRA was a hippie, a surfer, and a moun-taineer. During this period, he graduated from University Without Walls in Berkeley, California (one of those experimental colleges of the late 1960s), became a high school teacher, and obtained a master of arts degree in education and an master of business administration degree from California State University, Dominguez Hills. He decided to get serious and started working on microcomputers in 1979 and hasn't looked back. He began working with Oracle in 1988 and developed his own expertise though working with some of the best Oracle experts, including the authors of this book and his fellow reviewers.

Wally credits his sanity to his lovely wife Lois and is immensely proud of his four children: Layla, Irena, Ben, and Sam. He still loves the surf and the mountains. When Wally isn't out walking the seaside bluffs near his

home or refereeing teenage soccer games, he is a senior consultant with Intel Solution Services.

BRETT A. COLLINGWOOD has been using Oracle products for over 18 years. Brett started his IT career as an application developer using Oracle version 5. He has used Oracle on almost every general-purpose computer and operating system, including PCs, proprietary midrange systems, large SMPs, open systems, and mainframes. Along the way, Brett became frustrated with the perceived problems that his co-workers and customers were having with their databases. In many situations, problems were caused by a poorly architected system, botched imple-mentation, or ignored warning signs presented by the database, operating system, or peripheral devices.

Since that time, Brett has focused his efforts on architecting data-bases, operating systems, and applications in a manner where expec-

tations are correctly set while at the same time making an efficient use of the computer platform. Brett can usually be found implementing large clustered systems and complex applications, extending the infrastructure, and occasionally reengineering sites that have poorly architected sys-tems, botched implementations, ignored warning signals, and so forth.

XXII

Acknowledgments

We f e would like to thank all of those people who assisted in the writing of this book, including our customers, managers, colleagues, and contacts.

We would like to thank the Apress production team, including our technical editors, Matthew Moodie and Tony Davis; our project managers, Elizabeth Seymour and Beckie Brand; our technical reviewers, Kevin Closson, Wally Pereira, and Brett CoUingwood; and our copy editors. Heather Lang, Nicole LeClerc, and Jennifer Whipple.

We would also like to thank Phil Davies, Joel Goodman, Dominic Giles, John Nangle, Dave Storey, Bruce Carter, Phil Newlan, Erik Petersen, Paul Schuster, Paul Nothard, Stuart Pass, Fiona Godley, Jeff Browning, and Chris Gale for their assistance with the research and development of this book.

Julian Dyke and Steve Shaw

I would like to thank a few people who have helped me throughout my career, contributing either directly or indirectly to this book.

Most important, thank you to my wife, Isabel, and my children, Emma and Roberto, for their patience, tolerance, and support over the past 18 months.

I would also like to thank Steve Adams, James Anthony, Christian Antognini, Mike Bedford, Stephan Bendall, Lee Cashmore, Nigel Chapman, Carel-Jan Engel, Dave Ensor, Michael Erwin, Lex de Haan, Justin Hudd, Jeff Hunter, Howard Jones, Anjo Kolk, David Kurtz, Tom Kyte, Jonathan Lewis, Niall Litchfield, Richard Miller, Gary Millsap, Michael MoUer, Dave Moore, James Morle, Graham Murray, Mogens Norgaard, Tanel Poder, Rob Philpotts, David Phizacklea, Joze Senegacnik, John Skarin, Kevin Smith, Billy Taylor, Clive Taylor, Raies Uddin, MuraliVallath, Tim Waterton, and Plamen Zyumbyulev.

Finally, thank you to members of the Oak Table Network and the UKOUG I^C SIG for your assistance and inspiration over the years.

Julian Dyke

Above all, I owe an immeasurable debt of gratitude to my wife, Angela, my daughter, Evey, and my son, Lucas, for their boundless enthusiasm, motivation, and support for my writing of this book.

I would also like to thank Peter MacNamara, Neil Blecherman, Keith Shea, Alastair McKeeman, Eias Daka, Nigel Wayman, and Martinus JC Marx.

Finally, 1 would like to thank Todd Helfter, Jeff Hobbs, Zoran Vasiljevic, Donal Fellows, John LoVerso, Csaba Nemethi, Steve Cassidy, Brett Bercich, Bob Vance, Andy Duncan, Geoff Ingram, and everyone who has contributed time and skills to the development of Hammerora and Oracle on open source and Linux.

Steve Shaw

XXIII

Introduction

I his book is about deploying Oracle Real Application Clusters (RAC) databases on the Linux oper-ating system. RAC enables multiple Oracle instances running on separate servers to access a single database located on shared storage. RAC technology ensures that all instances can physically mod-ify any block in the database while maintaining a single logical copy of the data.

RAC was introduced as part of Oracle 9/ in 2001 and has been improved significantly in each subsequent major release. In Oracle lOg Release 2, RAC has become a highly stable production environment on all platforms, particularly on Linux.

Linux has been available as an open source operating system for many years. However, robust, fully supported, enterprise-level versions from Red Hat and Novell SUSE have only been available since around 200 L Linux offers almost all of the features included in proprietary UNIX offerings; it also offers additional functionality in many areas.

The number of Oracle customers implementing RAC databases has grown exponentially over the last couple of years. The cost of deploying RAC has decreased because of its use of industry-standard hardware.

RAC has four main objectives:

• Increased availability

• Increased scalability

• Improved manageability

• Reduced total cost of ownership

Achieving at least one of the aforementioned objectives is essential for a successful Oracle RAC deployment, and a few customers may achieve two or even three. It is very rare to encounter a cus-tomer who achieves all four goals, as to some extent, the objectives conflict with each other. In this book, we describe how to achieve each objective and we outline some of the pitfalls.

Oracle RAC may not be the optimum choice as a high-availability solution for all users. While we have met hundreds of Oracle customers who have successfully deployed Oracle RAC, we also know a handful for whom RAC has not been appropriate. Before you design an Oracle RAC architec-ture, you should investigate other options that might be available, including Oracle features such as Data Guard and Streams, and third-party solutions such as active/passive clusters, third-party clus-ter file systems, and SAN-level replication. Any high-availability solution should be considered within the context of your business needs, not the sales targets of your Oracle account manager.

Oracle documentation has greatly improved over the last few releases. Taken together with MetaLink web site notes and other information available on the Internet, there is plenty of information in the public domain for new Oracle RAC users. We have tried to consolidate as much information as possible into this book. Where we describe procedures such as installation, configuration, and adding and deleting nodes, we illustrate these with screenshots when possible to help you visualize the required steps.

We also venture into many areas that are not always covered by Oracle RAC books, including hardware and storage configuration. Enterprise Manager Grid Control, backup and recovery using RMAN, and disaster recovery using Data Guard. We also include a chapter on application development describing some of the latest RAC technologies.

XXV

xxvi ilNTRODUCTION

Coverage and Audience The book is intended for an audience of DBAs, developers, managers, and consultants—in short, anybody concerned with the installation, configuration, administration, or tuning of RAC databases on the Linux operating system. A basic knowledge of Linux or a background in other Unbc variants would be useful but is not essential.

We concentrate on the two main Linux platforms on which the Oracle Database lOgRAC option is supported, namely Red Hat Enterprise Linux (http: //www. redhat. com) and SUSE Linux Enterprise Server (http: //www. novell. com). We cover both Oracle lOgRelease 1 and Release 2 run-ning on Red Hat 4 and SUSE 9. However, some material does cover earlier releases of both Oracle RAC and the Linux operating system.

Of course, other high-availability products and solutions are available. One such product is PolyServe Matrix Server, about which one of our technical reviewers, Kevin Closson, has written an appendix. We hope this appendix will give you insight into the PolyServe solution and demonstrate that in some cases third-party high-availability architectures might address your business needs more appropriately.

Standards and Conventions We adopt a number of standards and conventions throughout this book. We generally avoid using the Oracle marketing names for releases and versions and use the numeric version numbers instead. The major releases discussed in this book are listed in Table L

Table 1. Oracle Version Numbers

Version Name Version Number

Oracle 9/ Release 1 Oracle 9.0.1 Oracle 9/ Release 2 Oracle 9.2.0 Oracle lOg Release 1 Oracle 10.1 Oracle lOgRelease 2 Oracle 10.2

We also use a standard configuration for all the examples in the book. As we are both based in the UK, we invented an environment in which our primary site is in London and our disaster-recovery site is in Reading, which is around 40 miles (60 kilometers) west of the capital. Our primary nodes, therefore, are called londonl, london2, londonB, and so on, and our standby nodes are called readingl, reading2, and so on.

We also attempt to use standard prompts throughout the book. Most are self-explanatory, but we deviate from our standards on a couple of occasions and abbreviate the prompts to avoid com-mands overflowing onto subsequent lines. All operating system-level commands in this book are run by the root user, for which we occasionally use the # prompt, or the oracle user, for which we use the $ prompt.

Errata and Additional IVIaterial Despite our best endeavors, there may be errors and omissions throughout the book. While we can only apologize for these in advance, we will try to maintain up-to-date errata on the Apress web site and on our own sites. Errata for this book can be found on the Apress web site (http: //www. apress. com). Additional material can be found on our personal web sites, which are http: //www. juliandyke. com for Julian Dyke and http: //www. sourceora. com for Steve Shaw.