45
© 2016 IBM Corporation Oracle Performance with Linux on System z David Simpson - Oracle Technical Specialist, IBM ([email protected])

Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

Embed Size (px)

Citation preview

Page 1: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

Oracle Performance with Linux on System z

David Simpson - Oracle Technical Specialist, IBM ([email protected])

Page 2: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

Copyright and Trademark Information

� For IBM – can be found at http://www.ibm.com/legal/us/en/copytrade.shtml

� For Oracle – can be found at

http://www.oracle.com/us/legal/index.html

� Any performance results/observations in this presentation are purely for education and planning purposes. No Test results should be construed as indicative of any particular customer workload or benchmark result.

Page 3: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

Oracle April PSU 2016: Security & Bug/Performance Fixes

3

Linux on System z – one of the first

Page 4: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

Upgrade 11.2.0.4 -> 12.1.0.1 - CPU for Concurrency Workload

18.9% improvement in response time between 11.2.0.4 & 12.1

Running Parallel Processes: 32real 0m10.12suser 0m0.16ssys 0m0.14s

Running Parallel Processes: 64real 0m20.05suser 0m0.34ssys 0m0.27s

Running Parallel Processes: 32real 0m12.01suser 0m0.20ssys 0m0.13s

Running Parallel Processes: 64real 0m23.84suser 0m0.40ssys 0m0.26s

Oracle 11.2.0.4

Oracle 12.1.0.1

Page 5: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

Start Planning for Oracle 12.1.0.2 Upgrade’s – preupgrd.sql

Source: Oracle Mike Dietrich’s Blog: https://blogs.oracle.com/UPGRADE/entry/oracle_april_2016_psu_and5

� With the latest PSU now included are the latest preupgrd.sql with utluppkg.sql scripts that can help with Oracle 12c upgrades!

� Pre-Upgrade utility provides a list of items which should be reviewed prior to the actual upgrade.

� 3 years newer than the original version you get with a fresh install of Oracle 12.1.0.2!

� MOS: How to Download and Run Oracle's Database Pre-Upgrade Utility (Doc ID 884522.1)

– Shared_pool_size has gone from 472M to 660M for non-CDB.

– Memory_target algorithm more robust

– added pga_aggregate_target to its sizing.

– Container Database sizing also included.

Page 6: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

Sample preupgrd.sql output with Linux on System z

6

Page 7: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

Oracle Data pump Export Recommendations

7

� Exclude statistics on Export expdp userid=xxx/xxx full=y EXCLUDE=STATISTICS directory=DBDIR dumpfile=expApr15_%U.dmp parallel=4 logfile=ex_Apr15.log

� If using Transportable Tablespaces – you need to use EXCLUDE=TABLE_STATISTICS,INDEX_STATISTICS

� Consider exporting statistics to a table then migrating the statistics table, to speed up migration time.

Oracle Admin Guide: https://docs.oracle.com/database/121/TGSQL/tgsql_astat.htm#TGSQL510

1. Create staging table in source database with DBMS_STATS.CREATE_STAT_TABLE

2. Export local stats into staging table using DBMS_STATS.EXPORT_SCHEMA_STATS

3. Export the staging table and import it into your destination database with Data Pump

4. Import the statistics held in the staging table by using DBMS_STATS.IMPORT_SCHEMA_STATS

Page 8: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

Oracle Data pump Import Recommendations

8

� Datapump import can take a really long time, when importing STATISTICS consuming large amount of TEMP (sorting) tablespace.

� If Datapump Statistics were not excluded on export you can try and exclude on import with:– EXCLUDE=TABLE_STATISTICS,INDEX_STATISTICS

� EXCLUDE=STATISTICS Or EXCLUDE=INDEX_STATISTICS During Datapump Import Still

Analyzes The Indexes (MOS ID: 793585.1)

– Had to set alter system set "_optimizer_compute_index_stats” =FALSE;

Hat Tip: http://www.oraclebuffer.com/oracle/why-datapump-import-impdp-is-not-ignoring-index-statistics/

Page 9: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

Oracle Data pump Import Recommendations

9

� Datapump import can take a really long time, when importing STATISTICS consuming large amount of TEMP (sorting) tablespace.

� If Datapump Statistics were not excluded on export you can try and exclude on import with:– EXCLUDE=TABLE_STATISTICS,INDEX_STATISTICS

� EXCLUDE=STATISTICS Or EXCLUDE=INDEX_STATISTICS During Datapump Import Still

Analyzes The Indexes (MOS ID: 793585.1)

– Had to set alter system set "_optimizer_compute_index_stats” =FALSE;

� Also watch your parallelization parameters when moving from a system with many CPU cores to fewer.

Hat Tip: http://www.oraclebuffer.com/oracle/why-datapump-import-impdp-is-not-ignoring-index-statistics/

Page 10: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

Import Performance

� Apply patch for bug 21539301

Old behavior: build one index at a time with PARALLEL n keyword

New behavior: build n indexes at a time,each without parallelism

12.1.0.2.0 without patch:

• PARALLEL=4: 5681 seconds (total import 4:14:23)

12.1.0.2.3 with patch:

• PARALLEL=4: 1308 seconds (total import 2:38:29)

• PARALLEL=8: 719 seconds (total import2:03:11)

� PARALLEL=32: 347 seconds (total import 1:44:41)

10Source: https://apex.oracle.com/pls/apex/f?p=202202:2::APPLICATION_PROCESS=downloadFile:::F20225_ID:164

Page 11: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

DataPump Performance Tips Summary

� Use PARALLEL=n, Typically n=2x <number of CPU cores>

� EXCLUDE=STATISTICS

� On export Enable parallel import of indexes:

– Apply patch for bug 21539301

– Available for 11.2.0.4 and 12.1.0.2

� Large Package Bodies Bug 16018315 (fixed in 11.2.0.4, 12.1.0.2)

– 11.2.0.3: 22179 seconds

– 11.2.0.4/12.1.0.2: 2810 seconds

� New feature in 12c: impdp … transform=disable_archive_logging:y (disables logging for import)

– Apply patch for bug 20778442

� Connecting as SYS (or SYSDBA) also gives worse performance!

– Use id with DATAPUMP_IMP_FULL_DATABASE

11

Page 12: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

Import Performance Results

12

Page 13: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

Deprecation of Non-CDB Architecture after 12.2

13

Source: http://docs.oracle.com/database/121/UPGRD/deprecated.htm#BABDBCJI

Page 14: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation14

Working with Multitenant Databases

� CDB is the multitenant container database and PDB – is the pluggable database.

� Local user ids are specific to the pluggable DB only.

Page 15: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation15

Connecting to Pluggable Database

� Application users connect to the database as before…

� Application users do not have access to other Pluggable databases in the CDB.

Page 16: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation16

Performance of Multitenant Databases

� Performance of a pluggable and a non-pluggable database about the same.

DB Name & Test 10 Minutes

Total Trans(swingbench)

User avgutilization %

Sys avgutilization %

I/O wait avg%

test5_1_pdb + test5_2_pdb 3,658,385 88.7 5.2 4.8

test3 + test4 (non pluggable) 3,647,106 89.9 5.4 2.8

Page 17: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

Oracle Pluggable Databases

17

- PDB hot clone/refresh/relocate. - Need to put the source in read-only for cloning, - Refresh a clone, can relocate a PDB to another CDB Online

Page 18: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

Performance:“Diagonal” Scalability & “Horizontal” Scalability

What can you do if you need more capacity or transaction throughput?

• System z can scale both “Vertically” and “Horizontally” dynamically:– Cpu (both virtual and physical cpus)– Memory (add to guest & LPAR)– I/O (cards/channels)– Network (link aggregation for bandwidth)

• System z can scale “Horizontally” using Oracle’s addNode.sh RAC topology.

RAID-5 or RAID-10

Horizontal

Vert

ical

Page 19: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

AWR Lite Snapshots – 12.1.0.2

� AWR Lite Snapshots - MOS Note: 1993045.1

� Reducing AWR resource consumption using LITE mode snapshots

– Automatic snapshots

– _AWR_SNAPSHOT_LEVEL = BASIC | LITE | TYPICAL | ALL | BESTFIT

� On-demand snapshots

SQL> exec dbms_workload_repository.create_snapshot('LITE');

� Introduced with Oracle Database 12.1.0.2

19

Page 20: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

Turning off Unneeded Services

� Keep the golden image as lean as possible in terms of processor usage, some of these services can be turned off with chkconfig command:

20

Source: http://www.redbooks.ibm.com/abstracts/sg248147.htmlThe Virtualization Cookbook for IBM z Systems Volume 2: Red Hat Enterprise Linux Server 7.1, SG24-8303The Virtualization Cookbook for IBM z Systems Volume 3: SUSE Linux Enterprise Server 12, SG24-8890

Red Hat 6.7# chkconfig iptables off# chkconfig ip6tables off# chkconfig auditd off# chkconfig abrtd off# chkconfig atd off# chkconfig cups off# chkconfig mdmonitor off

SuSe 11 SP4# chkconfig fbset off# chkconfig network-remotefs off# chkconfig postfix off# chkconfig splash off# chkconfig splash_early off# chkconfig smartd off# chkconfig xinetd off

Page 21: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

Linux paging / swappiness Update

� Default Linux swappiness setting of 60 Linux does proactive paging

– Recommendation: set swappiness to 1 (previously 0)

– /etc/syctl.conf add vm.swappiness=1 or 10

–Avoids any unnecessary waits for swap

� Kernel version 3.5 and over, as well as Red Hat kernel version 2.6.32-303 (6.4) and over.

21

Page 22: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

Large Linux Guests - Write Improvements

� The following settings may be relevant when tuning for database workloads for large Linux guests. (/etc/sysctl.conf) with write performance.

vm.dirty_ratiovm.dirty_ratiovm.dirty_ratiovm.dirty_ratio=10 =10 =10 =10 ---- Maximum percentage of dirty system memory (default 40Maximum percentage of dirty system memory (default 40Maximum percentage of dirty system memory (default 40Maximum percentage of dirty system memory (default 40))))

vm.dirty_background_ratiovm.dirty_background_ratiovm.dirty_background_ratiovm.dirty_background_ratio = 3 = 3 = 3 = 3 ---- Percentage of dirty system memory at which background Percentage of dirty system memory at which background Percentage of dirty system memory at which background Percentage of dirty system memory at which background writebackwritebackwritebackwriteback will start (default 10will start (default 10will start (default 10will start (default 10).).).).

vm.dirty_expire_centisecsvm.dirty_expire_centisecsvm.dirty_expire_centisecsvm.dirty_expire_centisecs = = = = 500 500 500 500 ---- Duration Duration Duration Duration after after after after dirty dirty dirty dirty system memory is considered old & eligible for system memory is considered old & eligible for system memory is considered old & eligible for system memory is considered old & eligible for writebackwritebackwritebackwriteback (default (default (default (default 3333000000000000))))

vm.dirty_writeback_centisecsvm.dirty_writeback_centisecsvm.dirty_writeback_centisecsvm.dirty_writeback_centisecs=100 =100 =100 =100 ---- ( default 500)( default 500)( default 500)( default 500)

vm.vfs_cache_pressurevm.vfs_cache_pressurevm.vfs_cache_pressurevm.vfs_cache_pressure =200 =200 =200 =200 ---- Helps performance of backups to diskHelps performance of backups to diskHelps performance of backups to diskHelps performance of backups to disk

vm.dirty_expire_centisecsvm.dirty_expire_centisecsvm.dirty_expire_centisecsvm.dirty_expire_centisecs = 3000 = 3000 = 3000 = 3000 ---- Default is 500Default is 500Default is 500Default is 500

� Effects particularly large Linux Guests > 8GB that write to NFS devices. (e.g. RMAN backups)

22Hat Tip SUSE: Hat Tip SUSE: Hat Tip SUSE: Hat Tip SUSE: https://www.SUSE.com/support/kb/doc.php?id=7010287https://www.SUSE.com/support/kb/doc.php?id=7010287https://www.SUSE.com/support/kb/doc.php?id=7010287https://www.SUSE.com/support/kb/doc.php?id=7010287

Page 23: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

VKTM with Oracle 12c & 11gR2

_disable_highres_ticks TRUE

_timer_precision 2000

23

% time seconds usecs/call calls errors syscall------ ----------- ----------- --------- --------- ----------------100.00 0.069437 1 125092 nanosleep0.00 0.000000 0 1 restart_syscall

------ ----------- ----------- --------- --------- ----------------100.00 0.069437 125093 total

% time seconds usecs/call calls errors syscall------ ----------- ----------- --------- --------- ----------------99.81 0.002063 1 1496 nanosleep0.19 0.000004 4 1 restart_syscall

------ ----------- ----------- --------- --------- ----------------100.00 0.002067 1497 total

_disable_highres_ticks False

_timer_precision 10

Default VKTM Oracle Parms: VKTM Changes to Help Reduce CPU***:

*** Get Oracle support approval before using.

Page 24: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

Summary of “tuneable” Oracle Parameters

� Work with Oracle support to get approval for setting underscore Oracle parameters…

*._disable_highres_ticks=TRUE #vktm cpu reduction

*._timer_precision=2000 #vktm cpu reduction

*._fastpin_enable=1 # enable reference count based fast pins

24

Page 25: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

Linux Huge Pages

� Consider Using Linux Huge Pages for Oracle Database Memory

→In general 10-15% can be gained by the reduction in CPU usage as well as having a lot more memory for applications that would be consumed in Linux Page Tables…

25

Page 26: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

Linux HugePage Considerations:

� Can not use MEMORY_TARGET with Huge Pages.

– Set large page memory with SGA_TARGET not including the PGA_AGGREGATE_TARGET.

– Oracle parameter use_large_pages = true | false | only

� HugePages are not swappable, and reduces your real memory of Linux guest size.

� General guideline consider when combined Oracle SGA’s are greater than 8 GB(particularly if a lots of connections)

� Decreased page table overhead; more memory can be freed up for other uses. i.e. more Oracle SGA memory, and less physical I/O (See also Oracle Note: 361468.1)

26

Page 27: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

Suggestions to Implement Large Pages:

� Update the following kernel parameters: /etc/sysctl.conf

vm.hugetlb_shm_groupvm.hugetlb_shm_groupvm.hugetlb_shm_groupvm.hugetlb_shm_group=10001 =10001 =10001 =10001 –––– set to the Group ID of Oracle user idset to the Group ID of Oracle user idset to the Group ID of Oracle user idset to the Group ID of Oracle user id

vm.nr_hugepagesvm.nr_hugepagesvm.nr_hugepagesvm.nr_hugepages = <#pages> = <#pages> = <#pages> = <#pages> ---- set to the Oracle SGA size (1 set to the Oracle SGA size (1 set to the Oracle SGA size (1 set to the Oracle SGA size (1 mbmbmbmb pages)pages)pages)pages)

� Update the Oracle ulimit values - /etc/security/limits.conf# Use memlock for Huge Pages support (kbytes)

* soft memlock unlimited

* hard memlock unlimited

27

Page 28: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

Putting it all Together

� Customer Customer Customer Customer was suffering was suffering was suffering was suffering –––– ffffrrrroooommmm OOOOrrrraaaacccclllleeee ““““FFFFrrrreeeeeeee BBBBuuuuffffffffeeeerrrr WWWWaaaaiiiittttssss””””,,,, ccccuuuussssttttoooommmmeeeerrrr rrrreeeedddduuuucccceeeedddd tttthhhheeeesssseeee wwwwaaaaiiiittttssss 50% by setting the 50% by setting the 50% by setting the 50% by setting the following>following>following>following>

Linux Parameter Changes (Linux Parameter Changes (Linux Parameter Changes (Linux Parameter Changes (sysctl.confsysctl.confsysctl.confsysctl.conf))))####fs.aiofs.aiofs.aiofs.aio----maxmaxmaxmax----nrnrnrnr = 1048576 = 1048576 = 1048576 = 1048576 fs.aiofs.aiofs.aiofs.aio----maxmaxmaxmax----nrnrnrnr = 3145728 = 3145728 = 3145728 = 3145728 ---- allow more asynchronous I/Oallow more asynchronous I/Oallow more asynchronous I/Oallow more asynchronous I/O

####vm.hugetlb_shm_groupvm.hugetlb_shm_groupvm.hugetlb_shm_groupvm.hugetlb_shm_group=10001 =10001 =10001 =10001 –––– set the Group ID for Oracle large pages to Oracle useset the Group ID for Oracle large pages to Oracle useset the Group ID for Oracle large pages to Oracle useset the Group ID for Oracle large pages to Oracle use

####vm.swappinessvm.swappinessvm.swappinessvm.swappiness = 60= 60= 60= 60vm.swappinessvm.swappinessvm.swappinessvm.swappiness = = = = 1111

####vm.dirty_ratiovm.dirty_ratiovm.dirty_ratiovm.dirty_ratio = 40= 40= 40= 40vm.dirty_ratiovm.dirty_ratiovm.dirty_ratiovm.dirty_ratio = 10 = 10 = 10 = 10

####vm.dirty_background_ratiovm.dirty_background_ratiovm.dirty_background_ratiovm.dirty_background_ratio = 10= 10= 10= 10vm.dirty_background_ratiovm.dirty_background_ratiovm.dirty_background_ratiovm.dirty_background_ratio = 3= 3= 3= 3

Oracle Parameter Changes (Oracle Parameter Changes (Oracle Parameter Changes (Oracle Parameter Changes (init.orainit.orainit.orainit.ora))))

# Make vktm processes use less cpualter system set "_disable_highres_ticks"=TRUE scope=spfile sid='*';alter system set "_timer_precision"=2000 scope=spfile sid='*';# Enable Fast Path buffer getsalter system set "_fastpin_enable"=1 scope=spfile sid='*';alter system set db_writter_processes=2;

28

Page 29: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

Oracle High Availability Network Options:

� Link Aggregation – (Active / Active ) Allow up to 8 OSA-Express adapters to be aggregated per virtual switch Each OSA-Express feature must be exclusive to the virtual switch (e.g. OSA’s can now be shared).

� Linux Bonding – create 2 Linux interfaces – e.g. eth1 & eth2 and create a bonded interface bond0 made up of eth1 and eth2.

� Oracle HAIP – Oracle 11gR2+ can now have up to 4 Private interconnect interfaces to load balance interconnect traffic.

Page 30: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

Oracle RAC with z/VM Multi VSWITCH LAG

Linux Oracle RAC Node 1A

NetworkSwitch 1

PUBLIC VSWITCHPROD1.PUBLIC

Dynamic Add:•Additional OSAs can be added dynamically to Multi VSWITCH Link Aggregation z/VM Logical Partition

PROD2

z/VM Logical Partition PROD1

PUBLIC VSWITCHPROD2.PUBLIC

NetworkSwitch 2

PRIVATE VSWITCHPROD1.PRIVATE

eth0 eth1

OSA and TCP/IP

Global VSWITCH PUBLIC v

Global VSWITCH PRIVATE

PRIVATE VSWITCHPROD2.PRIVATE

Linux Oracle RAC Node 2A

eth0

eth1

Linux Oracle RAC Node 1B

Linux Oracle RAC Node 2B

eth1

eth0

eth0

eth1

Page 31: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

I/O Tuning

� Typical AWR with I/O Wait:

31

Page 32: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

Changing I/O scheduler with UDEV RuleI/O settings you may want to consider adjusting (based on storage vendor recommendations):

scheduler I/O scheduler (noop or deadline)

nr_requests number of I/O requests default 128 (may want to increase to 1024 for Flash Storage devices)

rotational 1 default for spinning disk, 0 for Flash

nomerges default 0, 1 for Flash

max_sectors_kb default 1024, 4096 for Flash

rq_infinity default 1, 2 for Flash

add_random default 1, 0 for Flash

Sample UDEV rules (/etc/udev/rules.d) – Don’t forget to set both the single path and the multipath devices!

/etc/udev/rules.d/99-udev-oracle.rules

KERNEL=="sda",OWNER="oracle",GROUP="dba",MODE="660",RUN+="/bin/sh -c ‘echo noop > /sys/$devpath/queue/scheduler && echo 1024 > /sys/$devpath/queue/nr_requests'"

KERNEL=="dasd*1",ID=="0.0.0300",OWNER="oracle",GROUP=“dba”,MODE="0660",SYMLINK+="ASM0300“

/etc/udev/rules.d/12-dm-permissions.rules

#ENV{DM_UUID}=="mpath-36005076303ffcbbf000000000000ef01", OWNER:="oracle", GROUP:="dba", MODE:="660"

ENV{DM_NAME}=="ASM*",OWNER:="oracle",GROUP:="dba",MODE:="660",RUN+="/bin/sh -c 'echo noop > /sys/$devpath/queue/scheduler && echo 1024 > /sys/$devpath/queue/nr_requests'"

Restarting UDEV rules (Sles 11 sp3/Red Hat 6):

udevadm control --reload-rulesudevadm trigger32

Page 33: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

Linux multipathing Recommendations

� /etc/multipath.conf parameters can impact your FCP/SCSI performance particularly with I/O bound workloads:

� Check with your OS distributor & Storage vendors for recommended settings

rr_min_io_rq=1 #Number of I/Os per Path (for Newer Linux Kernels) SLES 11 SP3, RH 6.x+

#rr_min_io #Use for Older Linux Distros (Red Hat 5.x,SLES 11sp1 & older)

path_selector "service-time 0" #default for SLES 11 SP3+, RedHat 6.7+

#path_selector "round-robin 0" #round-robin for SLES 11 SP2+ and RedHat 6.x

path_grouping_policy "multibus" # SLES 11 SP1+ and Red Hat 6.x+

# path_grouping_policy "failover" # SLES 10, Red Hat 5.x

33

Page 34: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

Customer experience: multipath.conf setting

34

Page 35: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

Oracle I/O Calibrate – To Verify Storage Configuration

SET SERVEROUTPUT ONDECLARElat INTEGER;iops INTEGER;mbps INTEGER;BEGIN

DBMS_RESOURCE_MANAGER.CALIBRATE_IO (2, 10, iops, mbps, lat);DBMS_OUTPUT.PUT_LINE ('max_iops = ' || iops);DBMS_OUTPUT.PUT_LINE ('latency = ' || lat);dbms_output.put_line('max_mbps = ' || mbps);end;/

35

Source: https://docs.oracle.com/database/121/ARPLS/d_resmgr.htm#ARPLS67598

Page 36: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

Oracle I/O Calibrate Results

36 Educational - Test Results will vary by workload and environment

0

2

4

6

8

10

12

Baseline (1 Path) 2 Paths 3 Paths 4 Paths 6 Paths 12 Paths MultipathTuning

IOPs

mbps

IOPS and mbps “X” Times Percent Improvement

Page 37: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

OS Level Currency

�Significant Performance & Security Improvements

when upgrading OS Distribution levels:

Red Hat Memory Performance:

37

Source Red Hat - A Performance Comparison Between RHEL 5 and RHEL 6 on System z

Page 38: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

File System Types & Noatime mount option

� Recommended file system types:

– Oracle ASM

– SUSE xfs file system recommended for database files for SUSE distribution, see Oracle Note for latest: 414673.1

– Red Hat 6.x - ext4 recommended for database files.

– see Oracle Note for latest: 414673.1

� Oracle db file systems on ext3/4 mount with the noatime, (nodiratime) options

– Reduces CPU usage and I/Os - see Oracle Note 1561740.1

38

Page 39: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

z/VM HyperPAV Setup:

39

• To utilize HyperPAV with z/VM the Dasd volumes must be defined as a Full Pack, cylinder 0 –end, with either:

1) DEDICATE vDEV rDEV statements for the Base and subsequent alias(es) or 2) mdisk vDEV 3990 DEVNO rDEV mr statement.

• Utilize the entire DASD volume including cylinder 0 which you may previously have not utilized for Linux.

• Typically Full Pack mdisk’s start at cylinder 1-end to preserve the z/VM volume label and drive characteristics.

Page 40: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation40

Have lots of DASD to Format? – create a Flash Copy script!

Page 41: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

z/VM 6.3 with SMT Enabled

# vmcp q mtMultithreading is enabled.

Requested Activated

Threads Threads

MAX_THREADS MAX 2

CP core MAX 1

IFL core MAX 2

ICF core MAX 1

zIIP core MAX 1

41

cat /proc/cpuinfovendor_id : IBM/S390# processors : 24bogomips per cpu: 20325.00features : esan3 zarch stfle msa ldisp eimm dfp etf3eh highgprs processor 0: version = FF, identification = 05DA97, machine = 2964processor 1: version = FF, identification = 05DA97, machine = 2964processor 2: version = FF, identification = 05DA97, machine = 2964processor 3: version = FF, identification = 05DA97, machine = 2964…processor 22: version = FF, identification = 05DA97, machine = 2964processor 23: version = FF, identification = 05DA97, machine = 2964

• Oracle is licensed by the # of physical CPU Cores (IFLs) in a Hard Partitioned LPAR

• With z/VM SMT enabled the number of processors will show as the number of virtual processor threads that have been allocated and not what is licensed.

Page 42: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

IBM z13 CPU Performance

42

� Published performance improvement with out SMT (threading) is 12% and 32% for workloads that can benefit from SMT.

� SMT - Pre-install guidance based on internal testing and eventual field experience (20% for IFLs, 25% for zIIPs)

� .For Oracle workloads were seeing performance gains consistent with these z13 SMT performance guidance.

Page 43: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

How to Run an Oracle Performance Test

- Silly Little Oracle Benchmark (SLOB) – (Kevin Closson – author)- Logical I/O (Random memory access to Oracle SGA) - Want to have 99% + DB CPU and 100% Buffer Hit Ratio for a clean test from Oracle

Automatic Workload Repository (AWR) Report.

Page 44: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

Summary

� Performance

– Oracle runs well on System z for both memory access (Logical I/O)– Integration with Flash Systems allows Oracle to run well with Physical I/O

� Consolidation

– z/VM can virtualize / overcommit resources well.– System z can run Oracle at very high cpu utilization rates with little degradation.– System z can dynamically add system resources (memory, network, cpu)

� Highly Available– System z runs Oracle workloads highly available (hardware) and in some cases can avoid

configuring Oracle RAC for availability.– Linux HA solutions can be leveraged to increase application availability

� Security– Oracle on System z can be ran highly secure with FIPs (US Govt.) 140-2 compliance at z/VM

and Oracle levels.– SSL Crypto card support for Oracle SQL*net network traffic.

Page 45: Oracle Performance with Linux on System z · © 2016 IBM Corporation Oracle Performance with Linux on System z ... The Virtualization Cookbook for IBM z Systems Volume …

© 2016 IBM Corporation

Questions?