Introduction

Presentation 37007Optimizing Infrastructure for Oracle 9i Implementations

William BatailleBristol-Myers Squibb http://www.bms.com

Introduction

Database / Unix System Engineer at Bristol-Myers Squibb

Responsible for large single instance SAP / Oracle Implementation

– thousands of concurrent users– approaching 3 TB of data

Participant in Gartner Group SAP / Best Practices Advance training from Oracle SAP Solution Center,

Walldorf, Germany Knowledge of HP-UX, SANs, and EMC

Topics Infrastructure, what is it, why should I care ? Where can problems occur ? Planning for data placement Storage arrays uncovered Oracle 9i multiple block sizes OS implications (release and configuration) Evaluating your current implementation Trends and directions Questions, comments, discussions

Infrastructure, what is it, why should I care ? The hardware, firmware, and operating

system The configuration of these environments Infrastructure can affect recoverability Infrastructure can affect scalability Optimal Oracle Configuration requires

knowledge of the environment

Infrastructure Models

Simple, self contained– CPUs, memory and disk all in one server

Complex, separate disk storage– Network Attached Storage (NAS)– Storage Area Network (SAN)

Elaborate, partitioned – Hardware partition, separate disk storage– Virtual partition, floating memory, bound or

unbound CPUs

Complex Infrastructure Model

Storage Area Network Shared Disk Array

Gigabit Ethernet

100BaseT Dedicated MC/SG Heartbeat

HP HASSHigh AvailabilityStorage enclosures32 GB disk (16 GBmirrored to 16 GB)

4 Fibre channelinterfaces for BCV

10 fibre channel interfacesfor Primary and Secondary cluster nodes

Prod. DB server16 CPUs, 16GB RAM

16 port FC Switch

EMC

16 native Fibre channel ports

Disk Array32GB Cache18 TB Raw

5.5 TB usableRAID 1

BCV/R1

Backup ServerFC Switches

165.89.31.96

Standby DB server16 CPUs, 16GB RAM

Where can problems occur ?

Location of Data Physical Logical

Server memory / Oracle SGA Yes ?

Server I/O buffer cache Yes ?

Storage Network / Fabric Yes Yes

Disk Array Cache Yes ?

Logical Volume N / A Yes

Physical Disk Yes N / A

Data Block Yes Yes

Top - Down Problem tracking Oracle Alert / Trace log points to SGA

– Logical: check MetaLink for known problems– Physical: run diagnostics on memory

System log points to I/O subsystem– Server’s fiber channel interface– Physical cables– Fiber Channel switches– Disk Array’s fiber channel ports– Disk Array’s cache– Physical disk

Storage Focus

DBA’s concerned with data placement– recovery– performance

Hardware vendor interests– proprietary tools– other hardware solutions (e.g. cache)– RAID configurations

System Administrator – ease of configuration– ease of maintenance

Data Placement

Redo Logs on separate disks Segregate Data, Index, and Undo Tablespaces Isolate Archive Logs Document growth plan Sounds good, however:

– Disk hardware vendor doesn’t see the need– System Administrator doesn’t want to “waste storage”

Challenge: Is disk cheap ?

Planning for JBOD

Just a Bunch Of Disk How many disks on the system ? Are the used only by Oracle or shared ? What would happen if you lose a disk ? Enough disks to mirror ? Mirroring software available ?Plan to segregate by disk

Sample Data Placement on JBOD

/u03

/u05

/u01

/u02

/u04

/u01 - software, archive logs/u02 - SYSTEM, control 1, mirr redo/u03 - orig redo, RBS, control 2/u04 - TEMP, control 3/u05 - INDEX/u06 - DATA

No data loss if one disk failsProtection against controller failure

/u02

Planning for Disk Array How are the drives configured ?

– Raid 0+1 for performance– Raid n for cost savings

Hardware Stripe And Mirror Everything (SAME) Does the disk support revectoring ?

– If yes, has bad block reallocation been disabled ? What type of volume is presented to the server ?

– Entire disk ?– Hyper Volume ?– Meta volume ?

Plan to segregate by volume group

Sample Data Placement on Disk Array

Cache

/u01/u03/u05/u07

/u02/u04/u06/u08

Disk Adapter 1a

Fiber Adapter 1a

/u01/u03/u05/u07

/u02/u04/u06/u08

/u01/u03/u05/u07

/u02/u04/u06/u08

Disk Adapter 1b

Fiber Adapter 1b

/u01/u03/u05/u07

/u02/u04/u06/u08

/u01/u03/u05/u07

/u02/u04/u06/u08

Disk Adapter 3a

Fiber Adapter 3a

/u01/u03/u05/u07

/u02/u04/u06/u08

/u01/u03/u05/u07

/u02/u04/u06/u08

Disk Adapter 3b

Fiber Adapter 3b

/u01/u03/u05/u07

/u02/u04/u06/u08

VG1

Storage Terminology

Volume - physical disk, example 36GB Hyper Volume - slice of a volume, ex. 9GB Meta Volume - group of striped Hyper

Volumes example:– 4 * 9GB hyper volumes@1MB stripesize = 36GB

Volume Groups - collection of Meta Volumes Logical Volume - portion of a Volume Group Striped Logical Volume

Storage Relationships

Volume(disk)

VolumeGroup

LogicalVolume

HyperVolume

MetaVolume

FileSystem

Divided

Into

Combined

Into

Divided

Into

Mounted

On

Storage uncovered (Volumes)

Cache

V1

Disk Adapter 1a

Fiber Adapter 1a

Disk Adapter 1b

Fiber Adapter 1b

Disk Adapter 3a

Fiber Adapter 3a

Disk Adapter 3b

Fiber Adapter 3b

V2

V8 V7

V3 V4

V6 V5

V5 V6

V4 V3

V7 V8

V2 V1

Storage uncovered (Hyper Vols)

Cache

H1H9

H17H25

H5H13H21H29

Disk Adapter 1a

Fiber Adapter 1a

H32H24H16H8

H28H20H12H4

H2H10H18H26

H6H14H22H30

Disk Adapter 1b

Fiber Adapter 1b

H31H23H15H7

H27H19H11H3

H3H11H19H27

H7H15H23H31

Disk Adapter 3a

Fiber Adapter 3a

H30H22H14H6

H26H18H10H2

H4H12H20H28

H8H16H24H32

Disk Adapter 3b

Fiber Adapter 3b

H29H21H13H5

H25H17H9H1

Storage uncovered (Meta Vols)

Cache

M1M3M5M7

M2M4M6M8

Disk Adapter 1a

Fiber Adapter 1a

M1M3M5M7

M2M4M6M8

M1M3M5M7

M2M4M6M8

Disk Adapter 1b

Fiber Adapter 1b

M1M3M5M7

M2M4M6M8

M1M3M5M7

M2M4M6M8

Disk Adapter 3a

Fiber Adapter 3a

M1M3M5M7

M2M4M6M8

M1M3M5M7

M2M4M6M8

Disk Adapter 3b

Fiber Adapter 3b

M1M3M5M7

M2M4M6M8

Storage uncovered (Volume Groups)

Cache

M1M3M5M7

M2M4M6M8

Disk Adapter 1a

Fiber Adapter 1a

M1M3M5M7

M2M4M6M8

M1M3M5M7

M2M4M6M8

Disk Adapter 1b

Fiber Adapter 1b

M1M3M5M7

M2M4M6M8

M1M3M5M7

M2M4M6M8

Disk Adapter 3a

Fiber Adapter 3a

M1M3M5M7

M2M4M6M8

M1M3M5M7

M2M4M6M8

Disk Adapter 3b

Fiber Adapter 3b

M1M3M5M7

M2M4M6M8

VG1

Disk Array Monitoring Does an up-to-date schematic diagram exist? Are there multiple access paths to the disk? Is the I/O balanced across controller paths?

– Do pvchange scripts exist? How is performance monitored ? Preemptive support agreement ?

– Who is notified when maintenance occurs ?

File System Configurations

Create one Logical Volume per meta volume use Journaled File Systems (JFS) Set filesystem blocksize = database objects

– Consider 9i multiple blocksize capabilities use large files only when necessary

– some OS utilities still don’t work with > 2GB Choose your JFS mount options wisely

Journaled File Systems

Extent based allocation of disk Fast file system recovery thanks to logging Intent log holds completed “transactions” Mount options control use of this log Balance system integrity with performance Does your disk supports bad block revectoring?

– Internally in disk arrays– JBOD specified during pvcreate

Choosing JFS mount options

Ultra conservative– full logging of all structural changes– do not store any data in log (nodatainlog)

Conservative methodology– full logging for Oracle Redo Logs– delayed logging, datainlog for non Redo

Moderate methodology– delayed logging, datainlog for all filesystems

Online JFS allows dynamic changes

Oracle 9i Multiple Blocksizes

Useful in OLTP Environment to reduce block contention

Can free up I/O bandwidth– small blocksize where appropriate

Can remedy wrong initial choice of blocksize Synchronize with OS block sizeQuestion: What is the best OS block size?

2K, 4K, 8K ?OS blocksize tablespace blocksize ?

Oracle Tablespace Blocksize vs OS Filesystem BlocksizeTime in seconds to: Create 10 GB tablespace,

Insert 50 million rows, Select all rows:

0

500

1000

1500

2000

2500

Create Insert Select

2K TS on 2K OS

2K TS on 4K OS

2K TS on 8K OS

8K TS on 2K OS

8K TS on 4K OS

8K TS on 8K OS

32K TS on 2K OS

32K TS on 4K OS

32K TS on 8K OS

Storage Summary

Know and document your hardware environment and Vendor contacts

Establish firmware upgrade policies Keep Oracle and OS blocksizes the sameQuestion: Do I still need to mirror redo logs? Absolutely, doesn’t hurt performance and will save

you when mistakes occurQuestion: What is the best I/O?

No I/O, found everything needed in SGA

OS ImplicationsPatching strategy Oracle specified OS patchesOracle specified software

– example JAVA SDK 1.3Kernel Configurations

OS related installation problems inadequate kernel resources

– memory segments– file system handles

missing X-library symbolic links missing software

– c compiler– JDK (was version 1.3.1 July 2003)– PERL (was version 5.6.1 July 2003)

inadequate file system size missing entry in /etc/hosts, pfs_mount hangs SQLNet Session Data unit (SDU) set too high

OS related performance problems Dynamic file system buffer cache disabled

– set bufpages = 0 and nbuf = 0 I/O buffer set too high

– default dbc_max_pct 50% recommend value 8%set dbc_min_pct = dbc_max_pct

Psuedoswap is disabled– set swapmem_on = 1

unlockable memory set– set unlockable_mem = 0

OS related CPu performance problem

You are CPU bound and don’t know why You set up the server by applying a “Tuned

Parameter Set” Scheduling timeslice interval too low

– forces a process to check for pending signals– templates set incorrectly to 1– set timeslice = 10 (10 x 10 millisecond clicks)

Evaluating your current environment Is file system utilization > 95 percent ?

– Control files may need to grow– JFS performance issues– potential restore issues

Check recoverability – Same disk used by multiple file systems– Backup on same disk as data

Review system log / diagnostic messages– power failed errors and / or retries

Review Oracle and OS performance stats

Evaluating your current environment (cont.)

Check JFS mount options (more /etc/mnttab)– “nodatainlog” decreases write performance 50%

Check the clock time– syslog for network time protocol daemon recycle

Review oracle alert log for “checkpoint not complete”

Check if tablespace blocksize * multiblock read count is > OS capabilities (128K HP 11i)

Trends and directions

Faster CPUs following Moore’s Law Larger Disks and Cache Raid to the nTH degree Storage consolidation SANs for everything, even boot disks Server consolidation “Virtualization”: pooled IT assets across

storage systems, servers, networks...

AQ&Q u E S T I O N SQ u E S T I O N SA N S W E R SA N S W E R S

Discussion

Additional Resources

Oracle Technology Networkhttp://otn.oracle.com/deploy/availabilityhttp://otn.oracle.com/deploy/performance/content.html

Optimal Storage Configuration Made EasyDiagnosing Performance using StatsPack, Part I, II

Send an [email protected]

Reminder – please complete the OracleWorld session survey

Thank you. Presentation 37007

Documents

Introduction