Upload
secure-24
View
411
Download
9
Embed Size (px)
Citation preview
Session ID:
Prepared by:
Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP
1197
@jmjgangler
Mike Gangler – Senior Database Specialist, DBA Team Lead Secure-24 - @mjgangler [email protected]
Palm D Thu, Apr 14, 2016 08:30 AM - 09:30 AM
About Mike Gangler
• Oracle ACE with robust database credentials • DBA for over 28 years, working with Oracle since
version 4 • Team Lead and Senior Database Specialist at Secure-24 • Currently serving on the board of the Southeast Michigan
Oracle Professionals (SEMOP) group – www.meetup.com • Charter member of the Board of Directors for the
International Oracle Users Group (IOUG) – www.ioug.org • Follow me on my Blog http://mjgangler.wordpress.com and
on twitter! @mjgangler
2
About Secure-24
3
FOUNDED
HEADQUARTERS GLOBAL
OPERATION CENTERS
DATA CENTERS
Secure-24 was founded in 2001 and since then has grown
to 500+ employees and has received
recogniPon as one of Computerworld’s Best
Places to Work in IT, 3-years running.
Secure-24 is headquartered in
Southfield, MI
Serving customers around the globe,
Secure-24 has two (2) OperaPon Centers in Michigan, one (1) in
Nevada and one (1) in Hyderabad India.
Secure-24 has three (3) data centers in
Michigan, one (1) in Nevada, plus several global partnerships. We only choose the safest locaPons for
our data centers.
Secure-24 has 15 years of experience delivering managed IT operaPons, applicaPon hosPng and cloud services to enterprises worldwide. We manage SAP, Hyperion, PeopleSo^, JD Edwards, Oracle E-Business Suite and other
mission criPcal applicaPons across all industries for businesses of every size.
Communi'es Educa'on
Join for as low as $150 SELECT Journal Resource Center IOUG Press Webinars & Podcasts IOUG Forum 5 Minute Briefing
Plus get access to IOUG’s content library, peer-to-peer networking, and more! Corporate options also available!
Oracle Conferences in Detroit Area
Southeast Michigan Oracle Professionals
http://www.meetup.com/SouthEast-Michigan-Oracle-Professionals/ Meet monthly – 2nd Tuesday of the month
Michigan Oracle User Summit November 3, 2016 http://www.mous.us
Great Lakes Oracle Conference
• 2016 Great Lakes Oracle Conference (GLOC)
• May 18 & 19, 2016 Cleveland Public Auditorium
Cleveland, OH
https://www.neooug.org/gloc/
Today’s Discussion
7
Learn how Secure-24 leverages Oracle ZFS Storage, Oracle Intelligent
Storage Protocol (OISP), Templates and Oracle 12C RAC to implement a
Private Cloud for Oracle 12c RAC applications. This pairing of Database and
Storage hardware through Oracle’s VCA has enabled Secure-24 to take
advantage of the unique integration capabilities such as OISP which enables
automatic and dynamic tuning of storage settings such as record size.
Secure-24 Confidential – Internal – Copyright © 2015 Secure-24 and/or its affiliates. All rights reserved.
• Simplicity – Simplify the environment with new technology
• Scalability – Technology should be able to grow and scale as the company grows.
• Automation – Technology should have the ability to be automated and repeatable process
• Overall reduction of cost – Technology should drive down the overall cost
• Solution driven by Customer requirements and demands – This is the reason for looking at the technology
Secure-24 Technology Goals
Business Case
• Simplify Oracle 12c RAC deployments • Eliminate complexity around storage tuning • Decrease support time through transparency and
simplicity • Fully Oracle Support technology stack
10
Hardware Specifications Overview
Oracle Private Cloud Appliance
OracleX86Servers OracleZFSStorageAppliance• Redundantcontrollers,highperformancevirtualstorage
NetworkInfrastructure• OracleFabricInterconnectF1-15(VirtualNetwork)• OracleSwitches• Fullyconfiguredfabricnetworking,nosetuprequired• Integrateswithexis'ngEthernetandstoragenetworks
Fabric Interconnect
Fabric Interconnect
Compute Nodes
Compute Nodes
Mgmt Nodes
Management Storage
InfiniBand Expansion
Ethernet Switch
OracleX86Servers• Upto25nodes• CPU:Upto900cores• Memory:Upto6.4TBaggregate• SupportsanyLinux,SolarisandWindowsapplica'ons
Netw
orking M
anagement
40Gb/sServer-to-Server
InfiniBand Expansion
• HighspeedInfiniBandnetwork–mul'ple40Gbps
• InfiniBandconnectedstorage
• Dedicatedmanagementsystemsincluded
• OracleVMandOracleLinuxincluded
• TestedandCer'fiedbyOracle
• So^wareDefinednetworkingthroughFabricInterconnect
• Fast “Power-on to Production” - Automation – 2 hours vs. 1 month on legacy infrastructure
• Mixed workloads - Scalable – Oracle DB, Oracle Applications, Microsoft, Custom apps
• More efficient Licensing – Reduction of cost – Trusted Partitions Licensing
• Rapid Application Deployment – Simplicity and automation – Pre-built Oracle VM templates
• Simplified Support - Simplicity – One Hand to Shake (One Throat to Choke)
• Lower Acquisition and Deployment Costs • Impressive performance – Customer requirements
Why We Selected Private Cloud Appliance
Oracle 12C RAC Design Overview
• Oracle PCA and ZS3-2 Storage • Single Node or multiple nodes with RAC • Oracle 12c Enterprise Edition • Oracle ZFS storage array with OS 8.3+ • Database connections made from Oracle
using Direct NFS • ZFS Storage – Flash enabled performance
storage • OISP (Oracle Intelligent Storage Protocol)
for storage optimization • No “ASM”
X3-2
VIR
TUA
L CO
MP
UTE
AP
PLIA
NC
E
S PREARPS
MAJ OR
MINOR
US E R
C RIT IC AL
Z FS S TORAGEZ S 3-2
S TORAGEDE 2-24P
HDDMAP12-23
0-11
S TORAGEDE 2-24P
HDDMAP12-23
0-11
S TORAGEDE 2-24P
HDDMAP12-23
0-11
S TORAGEDE 2-24P
HDDMAP12-23
0-11
S TORAGEDE 2-24P
HDDMAP12-23
0-11
S TORAGEDE 2-24P
HDDMAP12-23
0-11
S TORAGEDE 2-24P
HDDMAP12-23
0-11
S TORAGEDE 2-24P
HDDMAP12-23
0-11
S TORAGEDE 2-24P
HDDMAP12-23
0-11
S TORAGEDE 2-24P
HDDMAP12-23
0-11
S TORAGEDE 2-24P
HDDMAP12-23
0-11
S TORAGEDE 2-24P
HDDMAP12-23
0-11
S TORAGEDE 2-24P
HDDMAP12-23
0-11
S TORAGEDE 2-24P
HDDMAP12-23
0-11
S TORAGEDE 2-24P
HDDMAP12-23
0-11
S TORAGEDE 2-24P
HDDMAP12-23
0-11
S PREARPS
MAJ OR
MINOR
US E R
C RIT IC AL
Z FS S TORAGEZ S 3-2
Design Overview
• Local OS disks Stored on Oracle VM data repository
• Data, Logs, and Voting disks presented via NFS and dNFS
VM 2
dNFS
VM
NAS,SAN,iSCSI
OracleVMServers
OracleVMServerPool
VM VMVM
SIVM 1
RAC RAC
ZFSStorage
ZFS Powered
• OISP MOS Doc ID 1943618.1 • ZFS Admin Guide http://
docs.oracle.com/cd/E51475_01/pdf/E52872.pdf
• ZFSHybridArraywith3typesofcaching• DRAM,SLCWriteFLASH,
MLCReadFLASH
• OISPOp'malRecordSize&Latency• TheOracledNFSclientpassestheop'mal
recordsizetotheZFSStorageApplianceforeachNFSv4writerequest
• TheOracledNFSclientpassestheop'malwritebiastotheZFSStorageApplianceforeachNFSv4writerequest..
ZFS Setup • Create our 3 shares
• OCR Voting Disk
– This will be raw NFS v3
– https://docs.oracle.com/database/121/CWADD/votocr.htm#CWADD92368
– We leave Write Bias as Latency and Record Size as 128K (Generic)
• Data and Logs files
– This is where the magic of OISP comes in.
– Instead of having to create multiple different volumes for redo, archive, control, data, temp etc. We simple create 2 volumes for Logs and Data
– Write Bias and Record Size will be set via database cue’s through OISP and optimized for the database.
STANDARD NFS ARCHITECTURE
dNFS and OISP
Linux Configuration Setup Direct NFS
– Fairly simple example
• # cat oranfstab.example
• server: 10.10.10.10
• nfs_version: nfsv4
• export: /export/U02 mount: /u02
• export: /export/U03 mount: /u03
– dNFS is fairly straight forward in 12c setup guide
• https://docs.oracle.com/database/121/CWLIN/storage.htm#CWLIN279
– This is where NFSv4 is setup – Many other options are
possible, but simple is possible too • Old paper, still valid - http://www.oracle.com/technetwork/articles/directnfsclient-11gr1-twp-129785.pdf
– Most dNFS guides are for 11g, 12c is the same except dNFS is Natively part of 12c
– Troubleshooting • (Doc ID 1464567.1)
dNFS Linux Configuration Setup
dNFS (11G) – Add 12C dnfs_on Enable:
Ln –s $ORACLE_HOME/lib/libnfsodm11.so $ORACLE_HOME/lib/libodm11.so
Disable
Ln –s $ORACLE_HOME/lib/libodmd11.so $ORACLE_HOME/lib/libodm11.so
dNFS Linux Configuration Setup
dNFS (12C) Enable:
Change directory to $ORACLE_HOME/lib
>>make –f ins_rdbms.mk dnfs_on
Disable Change directory to $ORACLE_HOME/lib
>>make –f ins_rdbms.mk dnfs_off
dNFS Linux Configuration Setup
dNFS Verify• bash-3.2$ ps -ef | grep dbw
oracle 23126 1 0 06:53 ? 00:00:00 ora_dbw0_MYINST1oracle 23128 1 0 06:53 ? 00:00:00 ora_dbw1_MYINST1oracle 23246 22377 0 07:04 pts/0 00:00:00 grep dbw
• bash-3.2$ /usr/sbin/lsof -p 23126COMMAND PID USER FD TYPE DEVICE SIZE NODE NAMEoracle 23126 oracle cwd DIR 104,3 4096 14811513 /u02/app/oracle/product/12.1.0/dbsoracle 23126 oracle rtd DIR 104,3 4096 2 /oracle 23126 oracle txt REG 104,3 210824714 15892486 /u02/app/oracle/product/12.1.0/bin/oracle
• ...oracle 23126 oracle 32u IPv4 322388 TCP mydbhost:64553->mynfshost:nfs (ESTABLISHED) oracle 23126 oracle 33u IPv4 322389 TCP mydbhost:21549->mynfshost:nfs (ESTABLISHED)
-- dNFS setup guide • https://docs.oracle.com/database/121/CWLIN/storage.htm#CWLIN279
Linux Configuration Setup
Oracle Linux 6 – uek Kernel – Utilize dtrace in Linux
– Ksplice for non-disruptive kernel patching
– UEK kernel optimization for Oracle DB
Setup Mount Points – Setup Voting disk
• Must be NFSv3
• Not Part of dNFS
• Mount options important (Doc ID 359515.1)
• rw,bg,hard,nointr,rsize=32768, wsize=32768,tcp,noac,vers=3,timeo=600,actimeo=0
– Setup Logs and Data
• Must be NFSv3 (dNFSv4 setup to come later)
• Mount options important (Doc ID 359515.1)
• rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,vers=3,timeo=600
Linux Configuration Setup - #1
Install Oracle Pre-Install RPM
• Optional but makes life easier • >> yum install oracle-rdbms-server-12cR1-preinstall • This will download all required dependencies
• create the oracle user id • modify sysctl.conf • modify limits.conf
• https://blogs.oracle.com/wim/entry/easily_install_oracle_rdbms_12cr1
Linux Configuration Setup - #2
TEST SNMP – REQUIRED FOR OISP MOS ID – 1943618.1 # SNMPGET –v1 –c public 10.10.10.10 1.3.6.1.4.1.42.2.225.1.4.2.0 - Looking for the following:
SNMPv2- SMI::enterprises. 42.2.225.1.4.2.0 = STRING: "Sun ZFS Storage 7330"
Linux Configuration Setup - #3
Setup for dNFS Oracle wants us to tune for dNFS, so we will
>> vi /etc/sysctl.conf ##7.3.4 Checking TCP Network Protocol Buffer for Direct NFS Client • net.ipv4.tcp_rmem = 4096 87380 4194304 • net.ipv4.tcp_wmem = 4096 16384 4194304
• https://docs.oracle.com/database/121/CWLIN/storage.htm#CHDEFHFC
Linux Configuration Setup
Install Oracle Pre-Install RPM
– Optional but makes life easier
– # yum install oracle-rdbms-server-12cR1-preinstall
– This will download all required dependencies, create the oracle user id, modify sysctl.conf and modify limits.conf.
– https://blogs.oracle.com/wim/entry/easily_install_oracle_rdbms_12cr1
Test snmp (needed for OISP)
– Doc ID 1943618.1
– # snmpget -v1 -c public 10.10.10.10 1.3.6.1.4.1.42.2.225.1.4.2.0
SNMPv2-SMI::enterprises.42.2.225.1.4.2.0 = STRING: "Sun ZFS Storage 7330"
Setup for dNFS – Oracle wants us to
tune for dNFS, so we will
• # vi /etc/sysctl.conf • # • #
https://docs.oracle.com/database/121/CWLIN/storage.htm#CHDEFHFC
• #7.3.4 Checking TCP Network Protocol Buffer for Direct NFS Client
• # • net.ipv4.tcp_rmem =
4096 87380 4194304
• net.ipv4.tcp_wmem = 4096 16384 4194304
Oracle 12c Enterprise Edition Install Oracle 12.1.0.2 - If using a private heartbeat IP that contains .255 - add the following patch (Before root.sh) - 19777496(PatchId)
- Or dont use .255 in the heartbeat network
Oracle 12c Installation - #1
Confirm dNFS and OISP setup $ egrep I '"Sun ZFS"|Direct' alert_*.log
Found NAS server Sun ZFS Storage 7420 Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 3.0 Direct NFS: channel id [0] path [10.80.216.18] to filer [10.80.216.18] via local [] is UP Direct NFS: channel id [1] path [10.80.216.19] to filer [10.80.216.18] via local [] is UP Direct NFS: mount complete dir /export/oisp/redo on 10.80.216.18 nfsport 2049 using NFS version 4 Direct NFS: mount complete dir /export/oisp/redo on 10.80.216.18 nfsport 2049 using NFS version 4
u “Found NAS server” indicates SNMP has found the ZFSSA. Requirement for OISP.
u ODM/Direct NFS indicates that dNFS has been activated using NFSv4.
Oracle 12c Installation - #2
Applyrequiredpatch(OISP)Patch19630940ResolvesissuewithDBAnaly'cs
Oracle 12c Installation - #3
Oracle12cEnterpriseEdi'onInstallOracle12.1.0.2
ConfirmdNFSandOISPsetup$egrepI'"SunZFS"|Direct'alert_*.log
FoundNASserverSunZFSStorage7420
OracleinstancerunningwithODM:OracleDirectNFSODMLibraryVersion3.0
DirectNFS:channelid[0]path[10.80.216.18]tofiler[10.80.216.18]vialocal[]isUP
DirectNFS:channelid[1]path[10.80.216.19]tofiler[10.80.216.18]vialocal[]isUP
DirectNFS:mountcompletedir/export/oisp/redoon10.80.216.18nfsport2049usingNFSversion4
DirectNFS:mountcompletedir/export/oisp/redoon10.80.216.18nfsport2049usingNFSversion4
“FoundNASserver”indicatesSNMPhasfoundtheZFSSA.RequirementforOISP.ODM/DirectNFSindicatesthatdNFShasbeenac'vatedusingNFSv4.
ApplyrequiredpatchPatch19630940
ResolvesissuewithDBAnaly'cs
Oracle 12c Installation
Typical Deployment
Typical Process For Creating and Deploying New Applications
1to3weeks
1-2days1day1-5days1-2days1-5days Start
PCA Rapid Deployment
• Pre-made, certified virtual machine templates, including application, operating system, and database available at https://edelivery.oracle.com
• Save Weeks/Days in Configuration Time – Pre-built, pre-configured Templates
– Application, Middleware, and DB
– Oracle DB, JD Edwards, EBS, PeopleSoft, and many more…. 90+ templates
Start-upinOracleVMPool
ImporttoOracleVMManager
OracleVMServers
ServerPool2Customize&Save
asGoldenImage
E-DeliveryFile
JDEdwards
DownloadfromOracle.com
File
JDE
VM1
JDE
Application Templates & Assemblies
DB
WebLogicServer
DeploymentPoliciesConfigura'onMetadata
SOA_PROD_1
OracleVMAssemblyProduc'onEnvironment:SOA_PROD_1
VM
VM
VM
VM
VM
VM
Deploy
Deployapplica'onintoOracleVMManageras‘GoldenImage’• Clone,ini'alize,andstart-upinstance• Upandrunninginhoursvs.daysorweekswithlitlerisk
OracleVMServers
StorageRepository
ServerPool2
Deployed:Produc'onEnvironmentSOA_PROD_1
GuestAddi'ons• Allowsdirectintegra'onbetweenguestso^wareandvirtualiza'onlayer• Assistsinorchestra'onofcomplex,mul'-VMdeployments
OVF-based
3
Notes
• Oracle Database 11g RAC with ZFS Storage Appliance and dNFS on SPARC: http://www.oracle.com/technetwork/server-storage/sun-unified-storage/documentation/oracledb-rac-sparc-zfssa-2414324.pdf
• Optimizing Storage for Oracle Database 11gR2 with ZFS Storage Appliance: http://www.oracle.com/technetwork/server-storage/sun-unified-storage/documentation/oracle11gr2-zfssa-bestprac-2255303.pdf
• How to enable OISP on ZFS Storage Appliance for 12c: https://support.oracle.com/epmos/faces/DocumentDisplay?id=1943618.1
ANALYST DATA
• htp://www.oracle.com/us/corporate/analystreports/industries/zs3-storage-appliance-2033590.pdf
Please complete the session evaluation Paper – 1197 Author – Mike Gangler We appreciate your feedback and Insight
You May complete the session evaluation via the mobile app