Presentation 37007 Optimizing Infrastructure for Oracle 9i Implementations William Bataille Bristol-Myers Squibb

Presentation 37007Optimizing Infrastructure for Oracle 9i Implementations

William BatailleBristol-Myers Squibb http://www.bms.com

Introduction

Database / Unix System Engineer at Bristol-Myers Squibb

Responsible for large single instance SAP / Oracle Implementation

– thousands of concurrent users– approaching 3 TB of data

Participant in Gartner Group SAP / Best Practices Advance training from Oracle SAP Solution Center,

Walldorf, Germany Knowledge of HP-UX, SANs, and EMC

Topics Infrastructure, what is it, why should I care ? Where can problems occur ? Planning for data placement Storage arrays uncovered Oracle 9i multiple block sizes OS implications (release and configuration) Evaluating your current implementation Trends and directions Questions, comments, discussions

Infrastructure, what is it, why should I care ?

The hardware, firmware, and operating system

The configuration of these environments Infrastructure can affect recoverability Infrastructure can affect scalability Optimal Oracle Configuration requires

knowledge of the environment

Infrastructure Models

Simple, self contained– CPUs, memory and disk all in one server

Complex, separate disk storage– Network Attached Storage (NAS)– Storage Area Network (SAN)

Elaborate, partitioned – Hardware partition, separate disk storage– Virtual partition, floating memory, bound or

unbound CPUs

Complex Infrastructure Model

Storage Area Network Shared Disk Array

Gigabit Ethernet

100BaseT Dedicated MC/SG Heartbeat

HP HASSHigh AvailabilityStorage enclosures32 GB disk (16 GBmirrored to 16 GB)

4 Fibre channelinterfaces for BCV

10 fibre channel interfacesfor Primary and Secondary cluster nodes

Prod. DB server16 CPUs, 16GB RAM

16 port FC Switch

EMC

16 native Fibre channel ports

Disk Array32GB Cache18 TB Raw

5.5 TB usableRAID 1

BCV/R1

Backup ServerFC Switches

165.89.31.96

Standby DB server16 CPUs, 16GB RAM

Where can problems occur ?

Location of Data Physical Logical

Server memory / Oracle SGA Yes ?

Server I/O buffer cache Yes ?

Storage Network / Fabric Yes Yes

Disk Array Cache Yes ?

Logical Volume N / A Yes

Physical Disk Yes N / A

Data Block Yes Yes

Location of Data Physical Logical

Server memory / Oracle SGA Yes ?

Server I/O buffer cache Yes ?

Storage Network / Fabric Yes Yes

Disk Array Cache Yes ?

Logical Volume N / A Yes

Physical Disk Yes N / A

Data Block Yes Yes

Top - Down Problem tracking

Oracle Alert / Trace log points to SGA– Logical: check MetaLink for known problems– Physical: run diagnostics on memory

System log points to I/O subsystem– Server’s fiber channel interface– Physical cables– Fiber Channel switches– Disk Array’s fiber channel ports– Disk Array’s cache– Physical disk

Storage Focus

DBA’s concerned with data placement– recovery– performance

Hardware vendor interests– proprietary tools– other hardware solutions (e.g. cache)– RAID configurations

System Administrator – ease of configuration– ease of maintenance

Data Placement

Redo Logs on separate disks Segregate Data, Index, and Undo Tablespaces Isolate Archive Logs Document growth plan Sounds good, however:

– Disk hardware vendor doesn’t see the need– System Administrator doesn’t want to “waste

storage” Challenge: Is disk cheap ?

Planning for JBOD

Just a Bunch Of Disk

How many disks on the system ? Are the used only by Oracle or shared ? What would happen if you lose a disk ? Enough disks to mirror ? Mirroring software available ?

Plan to segregate by disk

Sample Data Placement on JBOD

/u03/u03

/u05/u05

/u01/u01

/u02/u02

/u04/u04

/u01 - software, archive logs

/u02 - SYSTEM, control 1, mirr redo

/u03 - orig redo, RBS, control 2

/u04 - TEMP, control 3

/u05 - INDEX

/u06 - DATA

No data loss if one disk fails

Protection against controller failure

/u02/u02

Planning for Disk Array

How are the drives configured ?– Raid 0+1 for performance– Raid n for cost savings

Hardware Stripe And Mirror Everything (SAME) Does the disk support revectoring ?

– If yes, has bad block reallocation been disabled ?

What type of volume is presented to the server ?– Entire disk ?– Hyper Volume ?– Meta volume ?

Plan to segregate by volume group

Sample Data Placement on Disk Array

CacheCache

/u01/u01/u03/u03/u05/u05/u07/u07

/u02/u02/u04/u04/u06/u06/u08/u08

Disk Adapter 1aDisk Adapter 1a

Fiber Adapter 1aFiber Adapter 1a

/u01/u01/u03/u03/u05/u05/u07/u07

/u02/u02/u04/u04/u06/u06/u08/u08

/u01/u01/u03/u03/u05/u05/u07/u07

/u02/u02/u04/u04/u06/u06/u08/u08

Disk Adapter 1bDisk Adapter 1b

Fiber Adapter 1bFiber Adapter 1b

/u01/u01/u03/u03/u05/u05/u07/u07

/u02/u02/u04/u04/u06/u06/u08/u08

/u01/u01/u03/u03/u05/u05/u07/u07

/u02/u02/u04/u04/u06/u06/u08/u08



/u01/u01/u03/u03/u05/u05/u07/u07

/u02/u02/u04/u04/u06/u06/u08/u08

/u01/u01/u03/u03/u05/u05/u07/u07

/u02/u02/u04/u04/u06/u06/u08/u08



/u01/u01/u03/u03/u05/u05/u07/u07

/u02/u02/u04/u04/u06/u06/u08/u08

VG1

Storage Terminology

Volume - physical disk, example 36GB Hyper Volume - slice of a volume, ex. 9GB Meta Volume - group of striped Hyper

Volumes example:– 4 * 9GB hyper volumes@1MB stripesize = 36GB

Volume Groups - collection of Meta Volumes Logical Volume - portion of a Volume Group Striped Logical Volume

Storage Relationships

Volume(disk)

Volume(disk)

VolumeGroup

VolumeGroup

LogicalVolumeLogicalVolume

HyperVolumeHyper

VolumeMeta

VolumeMeta

Volume

FileSystem

FileSystem

Divided

Into

Combined

Into

Divided

Into

Mounted

On

Storage uncovered (Volumes)

CacheCache

V1V1









V2V2

V8V8 V7V7

V3V3 V4V4

V6V6 V5V5

V5V5 V6V6

V4V4 V3V3

V7V7 V8V8

V2V2 V1V1

Storage uncovered (Hyper Vols)

CacheCache

H1H1H9H9

H17H17H25H25

H5H5H13H13H21H21H29H29



H32H32H24H24H16H16H8H8

H28H28H20H20H12H12H4H4

H2H2H10H10H18H18H26H26

H6H6H14H14H22H22H30H30



H31H31H23H23H15H15H7H7

H27H27H19H19H11H11H3H3

H3H3H11H11H19H19H27H27

H7H7H15H15H23H23H31H31



H30H30H22H22H14H14H6H6

H26H26H18H18H10H10H2H2

H4H4H12H12H20H20H28H28

H8H8H16H16H24H24H32H32



H29H29H21H21H13H13H5H5

H25H25H17H17H9H9H1H1

Storage uncovered (Meta Vols)

CacheCache

M1M1M3M3M5M5M7M7

M2M2M4M4M6M6M8M8



M1M1M3M3M5M5M7M7

M2M2M4M4M6M6M8M8

M1M1M3M3M5M5M7M7

M2M2M4M4M6M6M8M8



M1M1M3M3M5M5M7M7

M2M2M4M4M6M6M8M8

M1M1M3M3M5M5M7M7

M2M2M4M4M6M6M8M8



M1M1M3M3M5M5M7M7

M2M2M4M4M6M6M8M8

M1M1M3M3M5M5M7M7

M2M2M4M4M6M6M8M8



M1M1M3M3M5M5M7M7

M2M2M4M4M6M6M8M8

Storage uncovered (Volume Groups)

CacheCache

M1M1M3M3M5M5M7M7

M2M2M4M4M6M6M8M8



M1M1M3M3M5M5M7M7

M2M2M4M4M6M6M8M8

M1M1M3M3M5M5M7M7

M2M2M4M4M6M6M8M8



M1M1M3M3M5M5M7M7

M2M2M4M4M6M6M8M8

M1M1M3M3M5M5M7M7

M2M2M4M4M6M6M8M8



M1M1M3M3M5M5M7M7

M2M2M4M4M6M6M8M8

M1M1M3M3M5M5M7M7

M2M2M4M4M6M6M8M8



M1M1M3M3M5M5M7M7

M2M2M4M4M6M6M8M8

VG1

Disk Array Monitoring

Does an up-to-date schematic diagram exist? Are there multiple access paths to the disk? Is the I/O balanced across controller paths?

– Do pvchange scripts exist?

How is performance monitored ? Preemptive support agreement ?

– Who is notified when maintenance occurs ?

File System Configurations

Create one Logical Volume per meta volume use Journaled File Systems (JFS) Set filesystem blocksize = database objects

– Consider 9i multiple blocksize capabilities

use large files only when necessary– some OS utilities still don’t work with > 2GB

Choose your JFS mount options wisely

Journaled File Systems

Extent based allocation of disk Fast file system recovery thanks to logging Intent log holds completed “transactions” Mount options control use of this log Balance system integrity with performance Does your disk supports bad block revectoring?

– Internally in disk arrays– JBOD specified during pvcreate

Choosing JFS mount options

Ultra conservative– full logging of all structural changes– do not store any data in log (nodatainlog)

Conservative methodology– full logging for Oracle Redo Logs– delayed logging, datainlog for non Redo

Moderate methodology– delayed logging, datainlog for all filesystems

Online JFS allows dynamic changes

Oracle 9i Multiple Blocksizes

Useful in OLTP Environment to reduce block contention

Can free up I/O bandwidth– small blocksize where appropriate

Can remedy wrong initial choice of blocksize Synchronize with OS block size

Question: What is the best OS block size?

2K, 4K, 8K ?OS blocksize tablespace blocksize ?

Oracle Tablespace Blocksize vs OS Filesystem Blocksize

Time in seconds to: Create 10 GB tablespace, Insert 50 million rows, Select all rows:

0

500

1000

1500

2000

2500

Create Insert Select

2K TS on 2K OS

2K TS on 4K OS

2K TS on 8K OS

8K TS on 2K OS

8K TS on 4K OS

8K TS on 8K OS

32K TS on 2K OS

32K TS on 4K OS

32K TS on 8K OS

Storage Summary

Know and document your hardware environment and Vendor contacts

Establish firmware upgrade policies Keep Oracle and OS blocksizes the same

Question: Do I still need to mirror redo logs? Absolutely, doesn’t hurt performance and will

save you when mistakes occur

Question: What is the best I/O? No I/O, found everything needed in SGA

OS Implications

Patching strategy Oracle specified OS patchesOracle specified software

– example JAVA SDK 1.3

Kernel Configurations

OS related installation problems inadequate kernel resources

– memory segments– file system handles

missing X-library symbolic links missing software

– c compiler– JDK (was version 1.3.1 July 2003)– PERL (was version 5.6.1 July 2003)

inadequate file system size missing entry in /etc/hosts, pfs_mount hangs SQLNet Session Data unit (SDU) set too high

OS related performance problems Dynamic file system buffer cache disabled

– set bufpages = 0 and nbuf = 0 I/O buffer set too high

– default dbc_max_pct 50% recommend value 8%set dbc_min_pct = dbc_max_pct

Psuedoswap is disabled– set swapmem_on = 1

unlockable memory set– set unlockable_mem = 0

OS related CPu performance problem

You are CPU bound and don’t know why You set up the server by applying a “Tuned

Parameter Set” Scheduling timeslice interval too low

– forces a process to check for pending signals– templates set incorrectly to 1– set timeslice = 10 (10 x 10 millisecond clicks)

Evaluating your current environment Is file system utilization > 95 percent ?

– Control files may need to grow– JFS performance issues– potential restore issues

Check recoverability – Same disk used by multiple file systems– Backup on same disk as data

Review system log / diagnostic messages– power failed errors and / or retries

Review Oracle and OS performance stats

Evaluating your current environment (cont.)

Check JFS mount options (more /etc/mnttab)– “nodatainlog” decreases write performance 50%

Check the clock time– syslog for network time protocol daemon recycle

Review oracle alert log for “checkpoint not complete”

Check if tablespace blocksize * multiblock read count is > OS capabilities (128K HP 11i)

Trends and directions

Faster CPUs following Moore’s Law Larger Disks and Cache Raid to the nTH degree Storage consolidation SANs for everything, even boot disks Server consolidation “Virtualization”: pooled IT assets across

storage systems, servers, networks...

AQ&Q u E S T I O N SQ u E S T I O N S

A N S W E R SA N S W E R S

Discussion

Additional Resources

Oracle Technology Networkhttp://otn.oracle.com/deploy/availability

http://otn.oracle.com/deploy/performance/content.html

Optimal Storage Configuration Made Easy

Diagnosing Performance using StatsPack, Part I, II

Send an email

[email protected]

Reminder – please complete the OracleWorld session survey

Thank you. Presentation 37007

Documents

Presentation 37007 Optimizing Infrastructure for Oracle 9i Implementations William Bataille Bristol-Myers Squibb