Optimizing Oracle on VMwareBert Scalzo PhD & Oracle ACEQuest Software: Database Expert

Intro

Many people swore that databases on virtual machines would not fly – or at best, wouldbe accepted very, very slowly. I was not one of those skeptics – so I wrote a book on justthat subject: Oracle on VMware: Expert Tips for Database Virtualization.

http://www.rampant-books.com/book_2008_1_oracle_vmware.htm

However I also don’t expect people in these tough economic times to rush out to buyevery new book, so this paper will attempt to provide a very brief synopsis of Oracle onVMware book’s overall thesis and some high value proposition recommendations.

Other Sources

One of the first and most robust papers I’ve seen on this subject is from VMware itself:Deployment of Oracle Databases on VMware Infrastructure.

http://www.vmware.com/partners/alliances/technology/oracle-database.html

This fine paper is 119 pages of highly useful DBA information for both effectively andefficiently deploying Oracle databases on a VMware infrastructure.

At first Oracle was a little slow to fully embrace the new concept of databases on virtualmachines. But then they débuted their own virtual machine solution – and things quicklychanged. There are now papers that show benchmarks of virtual machines versus baremetal that are achieving acceptable results.

http://www.oracle.com/technologies/virtualization/docs/ovmbenchmark.pdf

There are now even papers for successfully deploying RAC (Oracle Real ApplicationClusters) on virtual machines.

http://www.oracle.com/technology/products/database/clusterware/pdf/oracle_rac_in_oracle_vm_environments.pdf

So the trend winds have changed, virtualization is no longer verboten for databases

So Let’s Begin

At Oracle Open World 2008, I presented some materials that show that deploying Oracleon VMware with “default configuration and settings across the board” could cost you asmuch as 440% in terms of achievable database server performance. However, we mustapproach configuration, optimization and tuning in a slightly different manner – becausewe must now fully consider the four core, shared resources shown here more deeply:

This is really nothing new – DBA’s have always cared about the CPU, memory, disk andnetwork bandwidths versus utilization. But generally speaking in the past, DBA’s tendedto have one database instance per physical server – thus they historically didn’t worry toomuch about “shared resource” consumption. Of course with SAN, NAS and iSCSI sharedstorage servers – DBA’s began to feel, spot and optimize around such shared resources.

Virtualization merely extends that same paradigm across all of the key component layersthat the database engine relies upon – so we must now account and accommodate for that.Suppose that we have a SQL Server and Oracle database sharing the same resources andrunning radically different workloads concurrently. We must now then setup, configure,optimize and tune everything with the assumption that every resource of value is sharedand thus not ours alone. This means that there will be even less room for error than ever.

What Was Tested

Of course the very best possible workload to test with would be that from productionwhich one needs to deploy onto a virtualized server. For the sake of this paper, I simplyneeded to choose something that a general readership population might either alreadyknow or at least easily understand. Therefore I chose the TPC-C benchmark – a well-known and fairly respected industry standard OLTP benchmark (although soon to bereplaced by the newer TPC-E benchmark). I ran the TPC-C benchmark for 50 to 300concurrent users – and not larger due to my hardware size limitations. My goal was tomeet a Service Level Agreement (SLA) of no transaction taking more than 2 seconds,and to determine just how many concurrent users my server could ideally accommodate.

Luckily I work for a software vendor (i.e. Quest Software) – so it’s easy for me to obtaintools to make this testing and optimization effort trivial. I don’t mention these tools as asales pitch, but rather merely to inform you as to what I used. There are of course severalother tools out there that offer similar capabilities. So feel free to utilize those that bestsuit your needs, budget and/or preferences. Again, I only list these so that you know howI performed all of these tests. The 440% improvement results would have probably beenalso fairly easily obtained using different tools – because it’s really the concepts here thatapply, and not the tools per se used to arrive at and/or derive them.

1. Load Generator = Benchmark Factory for Databases

2. Virtualization Monitor = Foglight for VMware

3. Database Performance = Foglight Database Performance Analysis

4. Database Ad-Hoc Diagnostics = Spotlight for Oracle

So returning to the technique used, I incrementally and in an evolutionary style modifiedall the default settings across the board to obtain that 440% improvement for the TPC-COLTP database benchmark (your mileage will of course vary). Look at the chart below(Figure 1) at how the response time for 300 concurrent users decreased from 1080 ms to200. And while 1080 ms (i.e. 1 second) may have been acceptable (i.e. met the requiredSLA), what would happen if the user load increased to 1000 or more users. Obviously thetuned scenario would scale much better to thus to a much higher concurrent user load.

So what black magic was applied to get these results? The ten simple steps covered inthis paper are all that’s required to see the 440% improvement. Note that the VM serverused was running Windows 2003 Server Enterprise Edition Release 2 64-bit, but similarissues would also apply had it been Linux or some other OS.

Figure 1 – Performance Achieved for Various Default Settings

Ten Tricks to Try

Sometimes the best things are quite often obvious and easy. So here’s a simple plan athow to incrementally optimize all the relevant database virtualization default settings:

1. Obtain a baseline test for relative comparisons

2. On the VM host, exclude VM clients from active, online virus scans

3. Remove Windows Indexing Service(because don’t need fast file searches)

4. Remove other extraneous Windows services

5. Change the VM host registry settings to improve file system IO for databases

6. Optimize the VM host configuration and options

7. Optimize the VM client OS configuration and options for Oracle database

8. Remove other extraneous VM client OS services and daemons

9. Change VM client file system settings to improve IO performance for databases

10. Adjust VM client file system block size to more closely match Oracle block size

My key contention is that as long as there are sufficient resources and/or bandwidth tohandle the net requests, requiring database servers to be islands unto themselves is nowpassé’. Even if we say that virtualization adds a 10-20% overhead (which I’m inclined tosay is far closer to 10 than 20), with cheap hardware these days the benefits far outweighthe negatives – just spend 10% more and get a bigger server to handle multiple databases.But it may be a few years before that opinion is generally or more widely accepted.

The remainder of this paper focuses on just three somewhat less obvious key steps fromthe list above (which are highlighted in green): #5 and #9 (which kind of go together asone concept), and #10. All the rest are somewhat more obvious and thus in need of lessexplanation. Plus they also tend to be more subjective (i.e. what services don’t we need).

Optimize All File Systems for Databases

Remember, a database ultimately makes IO requests – and we all know that IO is theslowest part of the database equation. So look again above at what choices #5 and #9really are and/or mean. As a reminder:

#5 = Change the VM host registry settings to improve file system IO fordatabases

#9 = Change VM client file system settings to improve IO performance fordatabases

In reality, this is one and the same improvement simply being applied to two differentvirtualization levels: the host and each client. Thus I’m simply going to present how toaccomplish this technique for both Windows and Linux, and then leave it to the reader tomake sure to apply it properly across all their various virtualization layers.

Both the Windows NTFS and Linux ext2/3 file systems maintain multiple “meta-data”information related to file access – such as date created, last time updated, etc. So an IOrequest might actually generate multiple physical IO’s – one for the data file and one ormore for updating the related meta-data. In the case of the VMware server, we reallydon’t care to keep OS detailed file system information about the hosted clients’ data fileaccess – it’s simply neither useful nor critical (unless doing snapshots). And at the Oracledatabase level, we know Oracle accesses its files every so many seconds – so why pay toupdate the meta-data with that information. Oracle keeps it’s own timestamps (i.e. SCN)in the file headers.

For Windows, we simply adjust the following registry entry:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\FileSystem\NtfsDisableLastAccessUpdate = 1

By the way, Quest Software’s Toad for Oracle offers a screen to do this for Windows:

For Linux, there are several ways to accomplish the same result. We can set the arrtibutefor the individual Oracle files as follows:

chattr +A file_name

Or we can do it for an entire directory:

chattr –R +A directory_name

However the best method (because it automatically handles any file additions) is to editthe /etc/fstab and add the NOATIME attribute:

/dev/sda6 / ext3 defaults,noatime 1 1/dev/sda1 /boot ext2 defaults,noatime 1 2/dev/cdrom /mnt/cdrom iso9660 noauto,owner,ro 0 0/dev/sda5 swap swap defaults 0 0

Many people ask if I know similar settings for other operating systems such as AIX,HPUX and Solaris – I don’t. But please email me and share that information, because Iget asked this one all the time – and would love to have an answer for those people.

Match File System and Database Block Sizes

explain what choice #10 is. As a reminder:

#10 = Adjust VM client file system block size to more closely match Oracle blocksize

The default block size for both the host and client OS file systems’ is generally not thesame as your Oracle block size (although hopefully the Oracle block size is a multiple ofthe OS file block size). Since the host may be servicing multiple Oracle databases withdifferent block sizes, different database platforms (e.g. MySQL), or being used as to hostother applications (e.g. web server) – we cannot always make this adjustment at the hostlevel. But generally we can make it for each of the Oracle database clients.

Let’s assume we have a Linux host running a Linux based Oracle database client. Let’sexamine Scenario #2 from the chart below. Start by assuming that we create the host filesystem using the default block size: 2K. Let’s further assume that we do the same thingon the client – but that we size the database blocks at 4K. Thus each Oracle physical IOrequests asks the client OS for two IO’s, and each client IO asks the host for one IO.That’s a total of four logical IO requests (although only two physical IO’s in reality –but note that there is overhead for each logical IO request, so larger numbers are worse).Now if the database block size had instead been 8K while the other factors had remainedat 2K (i.e. Scenario #3), then the logical IO’ would instead now be DB -> Client = 4 andClient -> Host = 4 for a grand total of eight logical IO requests. Here’s a basic chart forsome common combinations and their sometimes surprising overheads:

Obviously there are some very bad choices in the chart above. So a seemingly good 16Kdatabase block size might actually result in a total of 12 to 16 logical IO requests acrossthe virtualization infrastructure under the worst case scenario. While it’s still effectivelyjust 16 K being read, the extra overheads added by this mismatch only serve to multiplythe bad effects. So choose wisely.

The Results & Conclusion

Look again back above at Figure 1 above where response time for 300 concurrent usersdecreased from 1080 ms to 200. With those initial default settings, the server could onlyscale to some 500+ users and still meet the under 2 second SLA. I was able to scale wellabove 2000 users on the very same hardware – and with better average response times.

The first question people ask is how did the incremental improvements pan out? That iswhat “measurable” percentage improvement went with each of colored line in Figure 1?I have included Figure 2 below to show that the tests (and their resulting graph lines) doin fact have a one to one correspondence to the recommended 10 steps or tricks to try.But I very purposefully left the graph color versus performance ramifications results keyoff Figure 1 so that you would be inclined to try them all. They’re too easy to skip evenone of them – and they’re all worth it.

Figure 2 - Breakdown of the Steps to Achieve the Results

This paper attempts to demonstrate that a database can perform up to 440% better simplyby adjusting various virtualized infrastructure default settings. Of course “your mileage”will vary, but the key point is that by simply taking the time to properly layer your virtualinfrastructure settings can have substantial impacts – which are magnified by the net orcumulative effect of a virtual server hosting multiple database servers. And while myexample was specific to Oracle, these same principles generally apply to any databasebeing deployed on a virtual server.

So don’t hesitate to deploy your database servers on virtualized platforms – it works.

Lessons Learned: (i.e. Best Practices)

Tons of “low hanging fruit” (i.e. easy no-brainer stuff)

Optimize each of the four major VM platform stacks

o Optimize the Host Machine (BIOS too)

o Optimize the Host VMware / OS Setup

o Optimize each Guest VM Configuration

o Optimize each Guest Operating System

Remember: 440% improvement for practically free – really

About the Author

Bert Scalzo is a Database Expert for Quest Software and a member of the TOAD devteam. He has worked with Oracle databases for over two decades, including time spentworking at both Oracle Education & Oracle Consulting. Mr. Scalzo is an Oracle ACE,holds several Oracle Masters certifications, plus an extensive academic background -including a BS, MS and PhD in Computer Science, an MBA and several insuranceindustry designations. He is an accomplished speaker and has presented at numerousOracle conferences - including OOW, ODTUG, IOUGA, OAUG, RMOUG and Hotsos.Mr. Scalzo's key areas of DBA interest are Data Modeling, Database Benchmarking,Database Tuning & Optimization, "Star Schema" Data Warehouses, Linux and VMware.He has also written many articles, papers and blogs - including for Oracle TechnologyNetwork (OTN), Oracle Magazine, Oracle Informant, PC Week (eWeek), Dell PowerSolutions Magazine, The LINUX Journal, linux.com, Oracle FAQ and Toad World. Plushe has written six books: Oracle DBA Guide to Data Warehousing and Star Schemas,TOAD Handbook, TOAD Pocket Reference (2nd Edition),Database Benchmarking:Practical Methods for Oracle & SQL Server, Oracle on VMware: Expert tips forDatabase Virtualization, and Advanced Oracle Utilities: The Definitive Reference. Mr.Scalzo’s email addresses are bert.scalzo@quest.com or bert.scalzo@yahoo.com.