Upload
phamngoc
View
253
Download
0
Embed Size (px)
Citation preview
Optimizing OLTP Oracle Database Performance using PCIe SSDs
Kai Yu
Dell Oracle Solutions Engineering
OUG Norway
Vårseminar 2013
HPC Engineering
About Author
• Kai Yu, Senior Architect, Dell Oracle Solutions Lab
18 years Oracle DBA and Solutions Engineering
Specializing in Oracle RAC, Oracle VM and Oracle EBS
Oracle ACE Director, Oracle papers author/presenter
Winner of 2011 OAUG Innovator of Year Award
Winner of 2012 Oracle Excellence Award: Technologist
of the Year: Cloud Architect
Oracle Blog: http://kyuoracleblog.wordpress.com/
HPC Engineering
3
Dell Global Solutions Engineering Lab
HPC Engineering
• Dell | Oracle Solutions Engineering : www.dell.com/oracle Dell | Oracle Partnership offers customers complete a solution Solutions Deliverables:
Tested and Validated Configuration Deployment Guild Oracle Advisor Detailed Oracle Solutions Deliverables List(SDL)
Dell Oracle Solutions Reference Configuration
Oracle Virtualization Technology Oracle Enterprise Manager & Cloud Oracle Applications Proof of Concepts
Dell Global Solutions Engineering Lab
HPC Engineering
Agenda
• Introduction to Express Flash PCIe SSD
• Configuring PCIe SSDs devices
• Use Cases of PCIe SSDs for Oracle Database
• Performance Study 1: PCIe to store Database objects
• Performance Study 2: PCIe as Oracle Smart Cache
• QAs
HPC Engineering
Introduction to Express Flash PCIe SSD • Introduction to Solid State Drive (SSD) Technology
Solid-State Drives (SSDs) use solid-state NAND flash memory to store data No rotating magnetic recording media, No mechanical moving parts of HDDs. No limited by mechanical constraints Two Type of NAND Flash: Single-level cell (SLC) NAND: one bit per Cell for high performance Multilevel cell (MLC) NAND: two bit per Cell for high capacity Different Types of SSD Formats and OS Driver Stacks: SATA Use SATA driver SAS Use SAS driver PCIe Connects to PCIe Bus:
• Express Cache PCIe SSDs Built with SLC NAND flash Used as an internal storage of Dell PowerEdge servers
6
HPC Engineering
Introduction to Express Flash PCIe SSD
7
Media Bay & Control
Panel SA
S 0
SA
S 1
SA
S 2
SA
S 3
SA
S 4
SA
S 5
SA
S 6
SA
S 7
PC
Ie S
SS
0
PC
Ie S
SS
1
PC
Ie S
SS
2
PC
Ie S
SS
3
Directly connected to PCIe bus Up to four PCIe SSDs per server 2.5 inch plugged into a device bay Capacity: 175GB/350GB Hot Swappable: add or remove a device without halting or rebooting the system
HPC Engineering
PCIe SSDs and Oracle Database Performance
• Dell Express Flash PCIe SSDs Built with SLC NAND flash Removes the performance bottleneck by the
mechanical parts of HDDs Eliminates the latency/bottleneck between server & external storage Designed for applications with low latency and heavy small random r/w IOs Performance: 1.8/1.2 GB/s for sequential r/w, 200k/320K IOPs Low Latency: < 50us, Capacity: 175GB/350GB NAND flash management by firmware on PCIe SSD, no burden for server CPU Durability:12.5 and 25 petabytes of drive life (175GB/350GB capacity,
respectively). 74898 times of the driver capacity.
8
HPC Engineering
Configuring PCIe SSD Devices
• Steps to configure Express Flash PCIe SSD drivers: Download: Express_Flash_PCIe-SSD_DRVR_RHEL6.2_1NY9K_A00_1.2.32-1.tar.gz
Unzip the driver: gunzip and tar –xvf Load it: $rpm -ivh kmod-mtip32xx-1.2.32-1.el6.x86_64_rhel6u2.rpm PCIe device partitions:
# more /proc/partitions major minor #blocks name 252 0 341873784 rssda PCIe1 252 256 341873784 rssdb PCIe2 252 256 341873784 rssdc PCIe 256 256 341873784 rssdd PCIe4
Partition these devices using fdisk /dev/rssda /dev/rssda1 Create Linux RAID configuration: mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 /dev/rssda1 /dev/rssdb1
mdadm --create --verbose /dev/md1 --level=1 --raid-devices=2 /dev/rssdc1 /dev/rssdd1
Set the proper ownership and permissions for devices in /etc/udev/rules.d
KERNEL=="md0",OWNER="grid", GROUP="asmadmin", MODE="0775"
KERNEL=="md1",OWNER="grid", GROUP="asmadmin", MODE="0775"
9
HPC Engineering
Use Cases of PCIe SSDs for Oracle Database • Use PCIe SSDs stores the entire database schema.
• PCIe SSDs stores portions of a database schema four PCIe SSDs combining with MD3220 external storage with 24 HDDs
• PCIe SSD used as Oracle Smart flash cache in a 11g R2 RAC Database
two PCIe SSDs combining with MD3220 external storage with 24 HDDs
10
PCIE SSDSAS/SATADell Power Edge R720Dell Power Edge R720
PCI-e SSD
Dell PowerVault MD3220 SAS
PCIE SSDSAS/SATA
PCI-e SSD
HPC Engineering
Use PCIe SSDs to store database objects • Single Node Database: use PCEI SSDs to store database objects
• MD3220 SAN to store the rest of the database: 24 Hard Drive Disks • Base line configuration: all the database objects are store in MD 3220 SAN
11
HPC Engineering
Use PCIe SSDs to store database objects • Create ASM diskgroup on PCIe SSDs: Two ASM disks: /dev/md01 and /dev/md02 each is 350GB, total PCIED diskgroup capacity is 700GB /dev/md01: mirroring of rssda1 and rsddb1 /dev/md02: mirroring of rssdc1 and rssdd1 • Possible Configurations: User Case 1: PCIe SSDs store the entire database User Case 2: PCIe SSDs store various portions of the database • Performance Test Tool: Quest Benchmark Factory: TPCC type workload testing. Generates concurrent workloads by incremental: 50-3000 users Measure the maximal OLTP throughput TPS (Transactions per Second) Measure the average responsible time at given # of the concurrent users
12
HPC Engineering
User Case 1: PCIe SSDs stores the Entire database • Performance Comparison of two configurations:
Baseline: the entire database schema stored in Hard Disk Drives (HDDS) Compared group: the entire database schema stored in PCIe SSDs TPS Comparison: 569% increase Response time (MS) Comparison: 250% increase of # of users for 1 second time 99 times reduce of query response time for 600 users
13
HPC Engineering
• Performance Improvement vs. Portions of database in PCIe SSDs: – Config1 : all the database objects stored in HDDs. (Base Line) – Config2: all indexes stored in PCI-e SSD Drives , – Config3: all indexes plus one most active table stored in PCI-e SSDs – Config4: all indexes plus four active tables stored in PCI-e SSD Response time Comparisons: TPS Comparisons: 12%-200% increase of # of users in 1 sec mark 14%-310% TPS Increases 2-16 times decrease of response time for 800 users
14
User Case 2: PC stores a portion of database
HPC Engineering
• Use PCIe as the smart flash cache: an extension of database buffer cache
• Configure PCEI SSD as Oracle Smart cache:
Set udev rule in /etc/udev/rules.d KERNEL=="rssda1",OWNER="oracle", GROUP="dba", MODE="0775"
15
PCIE SSDSAS/SATADell Power Edge R720Dell Power Edge R720
PCI-e SSD
Dell PowerVault MD3220 SAS
PCIE SSDSAS/SATA
PCI-e SSD
Database
Buffer Cache
Database
Buffer Cache
Smart Flash
Cache
Smart Flash
Cache
3. :Read Block from
Smart Flash Cache to
Buffer Cache
1: Read a data
block from disk
to Database
buffer
2: The block is
written to flash
cache
PCIe SSDs as Smart Cache for RAC
HPC Engineering
• Configure PCEI SSD as Oracle Smart cache: Set smart cache: alter system set db_flash_cache_file='/dev/rssda1' sid='*' scope=spfile alter system set db_flash_cache_size=20G sid='*' scope=spfile; Solve Oracle Bug with 11.2.03 on Linux: SQL> startup ORA-00439: feature not enabled: Server Flash Cache need to apply patch Patch 12949806; Oracle metalink note [ID 1473149.1] ORA-00439 during Startup of Database with Oracle Flash Cache
• Performance comparisons with baseline (no flash cache) Baseline: no flash Cache Compared groups: 20GB, 40GB, 60GB and 80GB flash cache Oracle buffer cache Size: 8GB Oracle database size: 500GB
• Monitor the smart cache usage: SELECT name, value FROM v$sysstat WHERE name IN ('physical read flash cache hits', 'physical reads', 'consistent gets', 'db block gets', 'flash cache inserts');
16
PCIe SSDs as Smart Cache for RAC
HPC Engineering
PCIe SSDs as Smart cache for RAC SELECT name, value FROM v$sysstat
WHERE name IN ('physical read flash cache hits', 'physical reads', 'consistent gets', 'db block gets', 'flash cache inserts'); NAME VALUE ---------------------------------------------------------------- ---------- db block gets 10912661 consistent gets 194736483 physical reads 4514407 physical read flash cache hits 2047180 flash cache inserts 246469
• 44.8% increase of TPS (Transactions Per Second)
17
HPC Engineering
PCIe SSD as Smart cache for RAC • 5.8 times decrease of Query Average Response Time
18
HPC Engineering
PCIe SSD as Smart cache for RAC • Observed different database wait patterns
19
HPC Engineering
PCIe SSD as Smart Cache for RAC • Free buffer wait event
– Process A: Fast data read from smart flash cache – Process B: Slow process to free a slot in memory due to the slow writing a dirty block to disk – Fast process A has to wait for slow process B Free buffer wait
20
HPC Engineering
Summary • Three user cases of PCIe SSDs for Oracle Database
1) Use PCIe SSDs to store the entire application database schema;
2) Use PCIe SSDs to store portion of application database schema.
3) Use PCIe SSDs as Oracle database smart flash cache • Performance Results: 1) Use Case1: 569% TPS increase and 99 times response time reduction, database size is limited as 700GB. 2) User Case2: have some different level of performance gains: 14% to 410% TPS increases and 41.3% to 20 times query response time reductions depending on how much is stored in PCIe SSDs. 3) Use case 3: achieved 44.5% TPS increase and 5.8 times response time reduction. Size of Smart cache increase doesn’t translate the further improvement due to the “free buffer wait”
21
HPC Engineering
A Customer Case Study of SSDs on Oracle • Optimizing Oracle Database Performance using SSDs Case study completed in Oslo, Norway The databases were serving an application from software vendor DIPS that is used at several hospitals in Norway. 20,000 concurrent users connecting to the five databases Four node RAC databases on Oracle Linux 6.3 Tested performance by replaying the real applications workloads using Oracle Real Application Testing (RAT)
• Test Results: transactions/queries continued with the same response time even after an enormous increase of load The response time was better in our PoC than the actual production system at the hospital. I/O wait was very low, and reserve CPU capacity was very high even with only three nodes in cluster. In other words the solution had a lot of spare capacity
22
HPC Engineering
A Customer Case Study of SSDs on Oracle
23
HPC Engineering
Questions?
Comments?
HPC Engineering
Contact me at [email protected] or visit my Oracle Blog at http://kyuoracleblog.wordpress.com/