A Performance Comparison of Microsoft® SQL Server®

2008 Transparent Data Encryption® and NetLib®

Encryptionizer®

Phil Colbert

CSCI 693 Fall 2008

Presentation Overview

Introduction Data Set Related Work Experimental Procedures Experimental Results Conclusions Future Work

SQL Server, Microsoft, Transparent Data Encryption, Virtual PC, Transact-SQL, T-SQL, and Windows XP are registered trademarks of the Microsoft Corporation, Inc. NetLib and Encryptionizer are registered trademarks of Communication Horizons LLC. VMware is a registered trademark of VMware, Inc.

Introduction Data and information security is a critical element

of our modern world Relational database management systems

(RDBMS) key component Microsoft SQL Server 2008 Enterprise introduces

Transparent Data Encryption (TDE), or whole database encryption

NetLib Encryptionizer has been available for nearly a decade, and works on all versions of Microsoft SQL Server since version 6

Benchmark performance comparisonSources: The Canadian Press [1], C. Boss [2], D. Migoya [3], S. Hsueh [4], NetLib [5]

Introduction: Acronyms

TDE: Transparent Data Encryption NLE: NetLib Encryptionizer AES: Advanced Encryption Standard DES: Data Encryption Standard TSQL: Microsoft Transact-SQL MSSQL: Microsoft SQL Server 2008

Enterprise BigNW: Expanded Northwind database

Introduction: Encryption

128-bit AES AES replaced DES in 2002 for the United

States government as the cryptographic security standard for all sensitive data

TDE and NLE both support AES

Source: FIPS [6]

Data Set

Microsoft Northwind database Expanded by a factor of 100 by BigNW

TSQL script authored by Scott Mauvais BigNW TSQL modified to work with

MSSQL

Sources: Microsoft [7], S. Mauvais [11]

Related Work

No significant benchmarks comparing these two products

Microsoft estimates 3-5% reduction in overall performance for low processor and low input/output systems using TDE

Microsoft estimates up to 28% reduction in overal performance for high load systems using TDE

Industry standard TPC benchmark cost-prohibitive Virtual environments viable benchmark platform

Sources: S. Hsueh [4], Transaction Processing Performance Council [9], M.L. Catalan et al [8]

Experimental Procedures: Overview

Development and implementation of a proprietary benchmark methodology based on ANSI SQL Standard Scalable and Portable (AS3AP) benchmark guidelines

Batch file, table-based, and TSQL script-based solution Experimental results saved in real-time to test results

database and tables Three MSSQL instances:

– Base: unencrypted baseline

– TDE: TDE encrypted

– NLE: NLE encrypted

Utilized MSSQL cache clearing methods

Source: C. Turbyfill et al [10]

Experimental Procedures: Platform and Software

Intel® Core™2 Duo 2.13 GHz Processor with 3.25 GB RAM running Microsoft Windows XP Professional operating system with Service Pack 3

Microsoft Virtual PC used version 6.0.156.0 with 1024 MB RAM running Microsoft Windows XP Professional operating system with Service Pack 3

VMware Workstation used version 6.5.0 build-118166 with 1024 MB RAM running Microsoft Windows XP Professional operating system with Service Pack 3

Microsoft SQL Server 2008 Enterprise code name "Katmai" (CTP) version 10.0.1075.23

NetLib Encryptionizer version was 2007.101.20.8.3.4a Hardware virtualization enabled All software packages and operating systems were 32-bit

Experimental Procedures: TSQL Benchmark

Fig. 1 Benchmark flow diagram

Experimental Procedures:Benchmarks

Ten benchmarks in five categories:1. Data retrieval, 2. Insertion, 3.Deletion, 4. Update, 5. Backup

Category Benchmark Description

Retrieve Select orders ordered on or after 1/1/1996 using index seek

Retrieve Select orders shipped to Germany using index scan

Retrieve Select all orders joined with customers table

Retrieve Select all invoices from Invoices view

Retrieve Execute stored procedure Employee Sales by Country inserted into temp table

Add Select all orders and insert into temporary table

Add Insert new orders

Update Update orders information

Delete Delete orders

Backup Add a backup device and backup the entire database

Fig. 2 Benchmark descriptions by category

Experimental Procedures: Unbalanced

Benchmark executions with no attempt to account for indeterminate system anomalies

Executions by benchmark by instance Record script execution times using TSQL timing

syntax Performed on both virtual environments, and dual-

processor non-virtual environment Derived baseline number of iterations per script to

execute within a 5-20 second time interval

Experimental Procedures:Balanced

Sequential execution methodology of each script across each instance

A measurable unit is the iterative execution of a single script, on a single database instance

Each measurable execution unit repeated 1000 times for dual processor environment, 100 times for each virtual environment

Reduce data outliers

Benchmark Iterations

1 12

2 19

3 32

4 2

5 7

6 12

7 143

8 10

9 100

10 3

Fig. 3 Derived iterations

Experimental Results: BalancedVirtual PC

Mean Execution Time (ms) ± % At 95%Confidence Level

Benchmark Base TDE NLE Base TDE NLE

1 7963 9524 8523 2.8 1.1 0.9

2 6531 8118 6685 1.6 1.2 1.6

3 5202 6974 6757 0.8 3.1 3.5

4 6838 7881 7679 0.4 1.0 2.7

5 6092 8796 7894 3.4 2.5 1.0

6 6314 7684 7163 1.5 0.5 4.0

7 7414 10151 8741 3.6 5.1 3.4

8 3532 4698 4298 1.0 0.8 2.2

9 8691 11953 11501 8.9 6.4 9.3

10 8135 7968 8624 3.0 2.7 2.9

Fig. 4 Balanced Virtual PC benchmark results, with an average sample mean decrease in execution speed for NLE compared to TDE of 6.9% (± 2.8% at the 95% confidence level).

Experimental Results: BalancedVirtual PC

Fig. 5 Balanced Virtual PC mean execution time (ms) at 95% confidence level

0 2000 4000 6000 8000 10000 12000 14000

BaseTDENLE

BaseTDENLE

BaseTDENLE

BaseTDENLE

BaseTDENLE

BaseTDENLE

BaseTDENLE

BaseTDENLE

BaseTDENLE

BaseTDENLE

BM 1

BM 2

BM 3

BM 4

BM 5

BM 6

BM 7

BM 8

BM 9

BM 10

Experimental Results: BalancedVMware

Mean Execution Time (ms) ± % At 95%Confidence Level

Benchmark Base TDE NLE Base TDE NLE

1 6301 8719 8138 1.9 1.1 1.3

2 4886 7083 6313 1.0 0.5 0.7

3 5398 6435 6630 1.3 0.9 0.8

4 6112 6600 6882 1.0 1.0 1.1

5 4999 8079 7227 0.6 0.4 0.4

6 4904 6896 6373 1.6 1.3 1.3

7 7820 8039 7002 4.7 5.5 3.1

8 10565 13606 12576 1.0 0.9 0.8

9 16283 15785 15493 0.4 1.5 0.9

10 5751 5694 6246 1.0 0.9 0.9

Fig. 6 Balanced VMware benchmark results, with an average sample mean decrease in execution speed for NLE compared to TDE of 4.1% (± 1.3% at the 95% confidence level).

Experimental Results: BalancedVMware

0 5000 10000 15000 20000

BaseTDENLE

BaseTDENLE

BaseTDENLE

BaseTDENLE

BaseTDENLE

BaseTDENLE

BaseTDENLE

BaseTDENLE

BaseTDENLE

BaseTDENLE

BM 1

BM 2

BM 3

BM 4

BM 5

BM 6

BM 7

BM 8

BM 9

BM 10

Fig. 7 Balanced VMware mean execution time (ms) at 95% confidence level

Experimental Results: BalancedNon-virtual dual-processor

Mean Execution Time (ms) ± % At 95%Confidence Level

Benchmark Base TDE NLE Base TDE NLE

1 7066 9324 7014 0.4 0.2 0.3

2 6763 11173 7172 0.6 0.3 0.5

3 6580 9000 7715 0.2 0.2 0.3

4 6618 7318 6749 0.1 0.1 0.1

5 7093 8283 7107 0.2 0.2 0.2

6 7108 9317 7280 0.4 0.2 0.3

7 3958 9653 6580 1.1 0.7 1.7

8 15377 15581 15568 0.1 0.2 0.1

9 4651 10684 7345 3.4 0.3 1.5

10 8783 8564 9419 0.1 0.1 0.1

Fig. 8 Balanced dual-processor benchmark results, with an average sample mean decrease in execution speed for NLE compared to TDE of 17.2% (± 0.4% at the 95% confidence level).

Experimental Results: BalancedNon-virtual dual-processor

0 5 10 15 20

BaseTDE

NetLibBaseTDE

NetLibBaseTDE

NetLibBaseTDE

NetLibBaseTDE

NetLibBaseTDE

NetLibBaseTDE

NetLibBaseTDE

NetLibBaseTDE

NetLibBaseTDE

NetLib

BM 1

BM 2

BM 3

BM 4

BM 5

BM 6

BM 7

BM 8

BM 9

BM 10

Time (seconds)

Fig. 9 Balanced dual-processor mean execution time (ms) at 95% confidence level

Conclusions

Experimental results indicate that– Both TDE and NLE increase mean execution

time– NLE is 17.2% (± 0.4%1) faster than TDE on a

non-virtual dual-processor system, 4.1% (± 1.3%) faster on VMware, and 6.9% (± 2.8%) faster on Virtual PC

– TDE mean execution time increased by 25.2% (± 0.4%) over the baseline instance on a non-virtual dual-processor system

1 All ± % at the 95% confidence level

Future Work Benchmark on a more diverse pool of computer hardware and software

– Processor count and configurations– Total memory– 64-bit and 32-bit operating systems– 64-bit and 32-bit MSSQL and NLE

Increased database size Increased benchmark query complexity Simulated load Industry standard TPC benchmarks Stronger encryption bit strength (256-bit) CPU ticks and IO busy ticks Increased iterations Completely isolated computer system with minimal software to

prevent indeterminate anomalies

References

[1] The Canadian Press, "National Bank reports theft of laptop with mortgage loan database," September 23, 2008.

[2] C. Boss, "Reynoldsburg student information stolen," The Columbus Dispatch, August 28, 2008.

[3] D. Migoya, "Stolen state database puts 1.4 million at ID-theft risk," Denver Post, November 2, 2006.

[4] S. Hsueh, "Database Encryption in SQL Server 2008 Enterprise Edition," SQL Server Technical Article, Feb. 2008. Retrieved from the World Wide Web Sep. 24, 2008, http://msdn.microsoft.com/en-us/library/cc278098.aspx.

[5] NetLib, "Performance Benchmarks – Whole Database Encryption," Communications Horizon, LLC., Retrieved from the World Wide Web Sep. 24, 2008, http://www.netlib.com/files/performance_benchmarks_wholedb.pdf.

[6] National Institute of Standards and Technology (NIST), "Announcing the Advanced Encryption Standard (AES)," Federal Information Processing Standards Publication (FIPS) 197, United States of America Federal Government, November 26, 2001. Retrieved from the World Wide Web Oct. 4, 2008, http://csrc.nist.gov/publications/fips/fips197/fips-197.pdf.

[7] Microsoft Corporation, Inc., "Access 2000 Tutorial: Northwind Traders Sample Database," June 22, 1999. Retrieved from the World Wide Web Sep. 24, 2008, http://www.microsoft.com/downloads/details.aspx?familyid=C6661372-8DBE-422B-8676-C632D66C529C&displaylang=en..

[8] M.L. Catalan, R. Ludena, A. Dennis, H. Umeno, "VM-Based Benchmark and Analysis System for Testing Online Transaction Processing," Second International Conference on Innovative Computing, Information and Control, ICICIC 2007, 2008, p. 4427665.

[9] Transaction Processing Performance Council, "TPC Benchmark E, Standard Specification, Version 1.6.0," Technical White Paper, 2008.

[10] C. Turbyfill, C. Orji, D. Bitton, "AS3AP – A Comparative Relational Database Benchmark," IEEE Computer Society International Conference, pp. 560-64, Feb. 1989.

[11] S. Mauvais, "Big Northwind Sample Code," May 23, 2007. Retrieved from the World Wide Web Sep. 24, 2008. http://www.mauvais.com/Download/ZD-BigNW.htm.

Thank You. Questions?

Benchmark Schema

Fig. 10 Benchmark schema

Benchmark Installation and Execution

Step 1: Create instances (Base, TDE, NLE) Step 2: Install NetLib Step 3: Configure NetLib with NLE instance Step 4: Create Northwind database within each instance Step 5: Create BigNW database from Northwind for each each

instance Step 6: Configure TDE for BigNW within TDE instance Step 7: Verify BigNW database in NLE instance is encrypted Step 8: Verify BIgNW database in TDE instance is encrypted Step 9: Verify benchmarks execution Step 10: Create TestResults database with each instance Step 11: Execute benchmarks Step 12: Evaluate benchmark data

TSQL Driver: Page 1-- CPU and timing variablesDECLARE @CPUStart int, @CPUEnd intDECLARE @StartTime datetime, @EndTime datetime-- Loop variablesDECLARE @InnerLoopCount int SET @InnerLoopCount = 1DECLARE @OuterLoopCount int SET @OuterLoopCount = 1DECLARE @InnerLoops intDECLARE @OuterLoops int-- Cursor variablesDECLARE @ScriptID int, @Benchmark int, @Description nchar(300), @Script nchar(300), @Iterate int, @Repeat int, @DisableCache bit,

@UseTable nchar(50)

-- Clean buffersCHECKPOINTDBCC DROPCLEANBUFFERS WITH NO_INFOMSGSDBCC FREESYSTEMCACHE ('ALL') WITH NO_INFOMSGS

-- Disable interactive output during processingSET NOCOUNT ON

-- Retrieve scriptsDECLARE curScripts CURSOR

FORSELECT [ScriptID], [Benchmark], [Description], [Script], [Iterate], [Repeat], [DisableCaching], [UseTable] FROM [TestResults].[dbo].[Scripts] ORDER BY [ScriptID]

OPEN curScripts

TSQL Driver: Page 2-- Loop over and execute scriptsFETCH NEXT FROM curScripts INTO @ScriptID, @Benchmark, @Description, @Script, @Iterate, @Repeat, @DisableCache, @UseTableWHILE (@@FETCH_STATUS = 0)BEGIN

-- Setup looping variablesSET @OuterLoops = @Repeat IF @OuterLoops <= 0Begin

SET @OuterLoops = 1End

SET @InnerLoops = @IterateIF @InnerLoops <= 0Begin

SET @InnerLoops = 1End

SET @OuterLoopCount = 1

PRINT 'Processing ' + Convert(varchar,@ScriptID) + ': ' + Convert(varchar(300),@Description)PRINT ' Instance = ' + Convert(varchar,'$(WhichInstance)')PRINT ' Benchmark = ' + Convert(varchar,@Benchmark)PRINT ' Iterations = ' + Convert(varchar,@Iterate)PRINT ' Repeat = ' + Convert(varchar,@Repeat)PRINT ' DisableCache = ' + Convert(varchar,@DisableCache)PRINT ''

TSQL Driver: Page 3-- Start outer loop block

While @OuterLoopCount <= @OuterLoops

BEGIN

-- Start performance block

SET @InnerLoopCount = 1

SET @CPUStart = @@CPU_BUSY

SET @StartTime = GETDATE()

WHILE @InnerLoopCount <= @InnerLoops

BEGIN

-- Begin dynamic execution

-- Select database to use for running scripts against and

-- execute script

EXEC ('USE ' + @UseTable + ';' + @Script)

-- Caching, if enabled

If (@DisableCache = 1)

BEGIN

CHECKPOINT

DBCC DROPCLEANBUFFERS WITH NO_INFOMSGS

DBCC FREESYSTEMCACHE ('ALL') WITH NO_INFOMSGS

END

-- End dynamic execution

SET @InnerLoopCount = @InnerLoopCount + 1

END

TSQL Driver: Page 4Set @EndTime = GETDATE()

SET @CPUEnd = @@CPU_BUSY

-- End performance block

-- Insert results into test table

INSERT INTO [TestResults].[dbo].[Results] ([Benchmark], [WhichInstance], [Cipher],[KeySize],[CPUTicks],[RunTime],[Query],[Iterate], [Repeat], [DisableCaching]) Values (@Benchmark,'$(WhichInstance)','AES','128',@CPUEnd-@CPUStart,DATEDIFF(ms,@StartTime,@EndTime),@ScriptID,@Iterate,@Repeat,@DisableCache)

-- Debug line

-- PRINT 'O: ' + Convert(varchar,@OuterLoopCount) + ', I: ' + Convert(varchar,@InnerLoopCount) + ', Diff: ' + Convert(varchar,DATEDIFF(ms,@StartTime,@EndTime))

--PRINT Convert(varchar,DATEDIFF(ms,@StartTime,@EndTime))

SET @OuterLoopCount = @OuterLoopCount + 1

END

FETCH NEXT FROM curScripts INTO @ScriptID, @Benchmark, @Description, @Script, @Iterate, @Repeat, @DisableCache, @UseTable

END

DEALLOCATE curScripts

SET NOCOUNT OFF

Benchmark Scripts: Page 1 Benchmark 1: Select orders ordered on or after 1/1/1996

– Select * INTO #Temp FROM [dbo].[Orders] WHERE [OrderDate] >= '1/1/1996'; IF OBJECT_ID('TempDB..#Temp') IS NOT NULL DROP TABLE #Temp;

Benchmark 2: Select orders shipped to Germany– Select * INTO #Temp FROM [dbo].[Orders] WHERE [ShipCountry] = 'Germany'; IF

OBJECT_ID('TempDB..#Temp') IS NOT NULL DROP TABLE #Temp;

Benchmark 3: Select all orders joined with customers– Select [dbo].[Orders].[OrderID], [dbo].[Customers].[ContactName] INTO #Temp FROM

[dbo].[Orders] LEFT JOIN [dbo].[Customers] ON [dbo].[Orders].[CustomerID] = [dbo].[Customers].[CustomerID]; IF OBJECT_ID('TempDB..#Temp') IS NOT NULL DROP TABLE #Temp;

Benchmark 4: Select invoices from Invoices view– Select * INTO #Temp FROM [dbo].[Invoices]; IF OBJECT_ID('TempDB..#Temp') IS NOT

NULL DROP TABLE #Temp;

Benchmark 5: Execute stored procedure Employee Sales by Country inserted into temp table

– EXECUTE [dbo].[Employee Sales by Country2] '1/1/1997','12/31/1997‘;

Benchmark Scripts: Page 2 Benchmark 6: Select ORDERS table into temporary table

– SELECT * INTO #Temp FROM [dbo].[Orders] IF OBJECT_ID('TempDB..#Temp') IS NOT NULL DROP TABLE #Temp;

Benchmark 7: Insert into Orders table– INSERT INTO [dbo].[Orders] ([CustomerID], [EmployeeID], [OrderDate]) Values

('VINET',5,'1/1/2005');

Benchmark 8: Updates Orders table– UPDATE [dbo].[Orders] SET [dbo].[Orders].[EmployeeID]=6 WHERE [dbo].[Orders].

[CustomerID]='VINET' AND [dbo].[Orders].[OrderDate]='1/1/2005‘;

Benchmark 9: INSERT and DELETE records from Orders table– INSERT INTO [dbo].[Orders] ([CustomerID], [EmployeeID], [OrderDate]) Values

('VINET',5,'1/1/2005'); DELETE FROM [dbo].[Orders] WHERE [dbo].[Orders].[CustomerID]='VINET' AND [dbo].[Orders].[OrderDate]='1/1/2005‘;

Benchmark 10: Add a backup device and backup the complete database– IF EXISTS (SELECT * FROM master.dbo.sysdevices WHERE NAME = 'BigNWBackupTest'

AND Status = 16) EXEC('sp_dropdevice BigNWBackupTest'); EXECUTE sp_addumpdevice 'disk', 'BigNWBackupTest','BigNWBackupTest.bak'; BACKUP DATABASE BigNW TO BigNWBackupTest WITH INIT;

Sample Graph:Raw Data Comparison

Benchmark 1 Run Chart - Raw Data Comparison

4000

6000

8000

10000

12000

14000

Tim

e (m

s)

Base TDE NetLib

Sample Graph:Delta Comparison

Benchmark 1 Run Chart - Delta Comparison

Base - TDE Base - TDE TDE - NetLib

Sample Chart:Scatter

Benchmark 1 Scatter Run Chart - Raw Data Comparison

4000

6000

8000

10000

12000

14000

Tim

e (m

s)

Base TDE NetLib