Sofia, Bulgaria | 9-10 OctoberSofia, Bulgaria | 9-10 October

Advanced Data Access Patterns

Advanced Data Access Patterns

with ADO.NET 2.0with ADO.NET 2.0

Julie Lerman

The Data Farm

Julie Lerman

The Data Farm

Sofia, Bulgaria | 9-10 October

About MeAbout Me● .NET Consultant and Mentor

● 20+ years developing

● Microsoft .NET MVP

● ASPInsider

● INETA Speaker

● Various publications & conferences● Blogs: thedatafarm.com/blog, blog.ziffdavis.com/devlife

● Founder and leader of Vermont .NET

● INETA Speaker Committee, VTSDA Board

● .NET Consultant and Mentor

● 20+ years developing

● Microsoft .NET MVP

● ASPInsider

● INETA Speaker

● Various publications & conferences● Blogs: thedatafarm.com/blog, blog.ziffdavis.com/devlife

● Founder and leader of Vermont .NET

● INETA Speaker Committee, VTSDA Board

Sofia, Bulgaria | 9-10 October

AgendaAgenda

● ADO.NET 2.0 DataSet performance

● Pushing huge data back to SQL Server

● Caching huge amounts of data

● Querying huge amounts of data in memory

● ADO.NET 2.0 DataSet performance

● Pushing huge data back to SQL Server

● Caching huge amounts of data

● Querying huge amounts of data in memory

Sofia, Bulgaria | 9-10 October

New in ADO.NET 2.0New in ADO.NET 2.0

● New features used in this session

● Batch Updating with DataAdapter

● SqlBulkCopy Class

● SqlDependency Class

● New features used in this session

● Batch Updating with DataAdapter

● SqlBulkCopy Class

● SqlDependency Class

Sofia, Bulgaria | 9-10 October

ADO.NET 2.0 PerformanceADO.NET 2.0 Performance

● DataSet Scalability across the board

● Can handle lots of rows

● Indexing Engine completely re-written

● Incremental indexing updates

● much more

● Low overhead for using Data Views

● Serialization: Faster, more compact

● Supports Binary & Schema-less serialization

● DataSet Scalability across the board

● Can handle lots of rows

● Indexing Engine completely re-written

● Incremental indexing updates

● much more

● Low overhead for using Data Views

● Serialization: Faster, more compact

● Supports Binary & Schema-less serialization

Sofia, Bulgaria | 9-10 OctoberSofia, Bulgaria | 9-10 October

DemonstrationDemonstration

Indexing Engine PerformanceIndexing Engine Performance

Sofia, Bulgaria | 9-10 October

Moving BIG DataMoving BIG Data

● Getting data is easy and fast

● Uploading is a different story

● 1.1 Update Options

● DataAdapter.Update – one row at a time

● Controlled stored procedure update – one row at a time

● Getting data is easy and fast

● Uploading is a different story

● 1.1 Update Options

● DataAdapter.Update – one row at a time

● Controlled stored procedure update – one row at a time

Sofia, Bulgaria | 9-10 October

Updating in 2.0Updating in 2.0

● DataAdapter Batch Update

● Pushes groups of rows up to SQL Server

● SQL Server still updates one at a time

● Find sweet spot for UpdateBatchSize value

● SqlDataAdapter & OracleDataAdapter

● Bulk Copy

● Streamed Insert using SQL Server Bulk Copy

● DataAdapter Batch Update

● Pushes groups of rows up to SQL Server

● SQL Server still updates one at a time

● Find sweet spot for UpdateBatchSize value

● SqlDataAdapter & OracleDataAdapter

● Bulk Copy

● Streamed Insert using SQL Server Bulk Copy

*slide slightly modified from original printed version

Sofia, Bulgaria | 9-10 OctoberSofia, Bulgaria | 9-10 October

DemonstrationDemonstration

Batch Update & Bulk Copy ClassBatch Update & Bulk Copy Class

Sofia, Bulgaria | 9-10 October

Fast Updates w/BCPFast Updates w/BCP

● Use BCP to stream Inserts & Updates and Deletes to temptables

● Create & run a DML TSQL command to process inserts, updates and deletes from the temptables into the real tables

● Wrap it all in a transaction

● End to End streaming with DataReader

● Use BCP to stream Inserts & Updates and Deletes to temptables

● Create & run a DML TSQL command to process inserts, updates and deletes from the temptables into the real tables

● Wrap it all in a transaction

● End to End streaming with DataReader

DML: Database Manipulation Language

Sofia, Bulgaria | 9-10 OctoberSofia, Bulgaria | 9-10 October

DemonstrationDemonstration

Using BCP & DML to process huge amounts of data

Using BCP & DML to process huge amounts of data

Sofia, Bulgaria | 9-10 October

DataSet CachingDataSet Caching

● DataSet improvements enable storing lots of data into cache

● SqlDependency enables automated notification of changes to .NET

● Combine SqlDependency and Caching for serious resource reduction

● DataSet improvements enable storing lots of data into cache

● SqlDependency enables automated notification of changes to .NET

● Combine SqlDependency and Caching for serious resource reduction

Sofia, Bulgaria | 9-10 OctoberSofia, Bulgaria | 9-10 October

DemonstrationDemonstration

SqlDependency & DataSet CachePart 1

SqlDependency & DataSet CachePart 1

Sofia, Bulgaria | 9-10 October

Break up Big cacheBreak up Big cache

● SqlDependency is all or nothing

● Create separate dependencies for logical sections of your cache

● SqlDependency is all or nothing

● Create separate dependencies for logical sections of your cache

Sofia, Bulgaria | 9-10 OctoberSofia, Bulgaria | 9-10 October

DemonstrationDemonstration

SqlDependency & DataSet CachePart 2: Achieve granularity

SqlDependency & DataSet CachePart 2: Achieve granularity

Sofia, Bulgaria | 9-10 October

Query BIG DataSetQuery BIG DataSet● Query a huge DataSet in cache● In-Memory Querying is coming!

● LiNQ● VB9/C#3

● What about today?● ADO.NET 2.0 has everything you need

to build a simple Query Processor● Great example of QP from ADO.NET

Team Technical Lead Pablo Castro (“for demo only”)

● Query a huge DataSet in cache● In-Memory Querying is coming!

● LiNQ● VB9/C#3

● What about today?● ADO.NET 2.0 has everything you need

to build a simple Query Processor● Great example of QP from ADO.NET

Team Technical Lead Pablo Castro (“for demo only”)

Sofia, Bulgaria | 9-10 October

Sample Query ProcessorSample Query Processor

● Takes advantage of new ADO.NET 2.0 tools

● Ability to query huge datasets

● Uses Caching

● Super Fast indexing

● DataTable.Select

● DataView

● Very simplistic compared to T-SQL

● *NOT* for production use

● Takes advantage of new ADO.NET 2.0 tools

● Ability to query huge datasets

● Uses Caching

● Super Fast indexing

● DataTable.Select

● DataView

● Very simplistic compared to T-SQL

● *NOT* for production use

Sofia, Bulgaria | 9-10 OctoberSofia, Bulgaria | 9-10 October

DemonstrationDemonstration

A look at a sample Query Processor written with ADO.NET 2.0

A look at a sample Query Processor written with ADO.NET 2.0

Sofia, Bulgaria | 9-10 October

SummarySummary● ADO.NET 2.0’s performance and

functional improvements put new power in our hands

● You can work with huge amounts of data efficiently

● Functionality like BatchUpdate gives you easy access to some of this power

● With a little more code, you can do wonders!

● ADO.NET 2.0’s performance and functional improvements put new power in our hands

● You can work with huge amounts of data efficiently

● Functionality like BatchUpdate gives you easy access to some of this power

● With a little more code, you can do wonders!

Sofia, Bulgaria | 9-10 October

ResourcesResources

● PDC05 Session DAT408, Pablo Castro, ADO.NET Team Technical Lead

● ADO.NET Team Blog: blogs.msdn.com/dataaccess

● MSDN Data Access Dev Center

● msdn.microsoft.com/data

● What’s New in ADO.NET 2.0, Julia Lerman, MSDN Magazine April 2005

● PDC05 Session DAT408, Pablo Castro, ADO.NET Team Technical Lead

● ADO.NET Team Blog: blogs.msdn.com/dataaccess

● MSDN Data Access Dev Center

● msdn.microsoft.com/data

● What’s New in ADO.NET 2.0, Julia Lerman, MSDN Magazine April 2005

Sofia, Bulgaria | 9-10 October

ADO.NET 2.0 BookshelfADO.NET 2.0 Bookshelf

● ADO.NET and System.Xml v 2.0 – The Beta Version: Alex Homer, Dave Sussman & Mark Fussell, Addison-Wesley

● PRO ADO.NET 2.0, Sahil Malik, APress

● Professional ADO.NET 2: Programming with SQL Server 2005, Oracle, and MySQL, Wallace B. McClure, Gregory A. Beamer, IV John J. Croft, J. Ambrose Little, Bill Ryan, Phil Winstanley, David Yack, Jeremy Zongker, WROX Press

● Programming Microsoft ADO.NET 2.0 Applications:

Advanced Topics, Glenn Johnson, Microsoft Press

● Microsoft ADO.NET 2.0 Step by Step, Rebecca Riordan, Microsoft Press

● ADO.NET and System.Xml v 2.0 – The Beta Version: Alex Homer, Dave Sussman & Mark Fussell, Addison-Wesley

● PRO ADO.NET 2.0, Sahil Malik, APress

● Professional ADO.NET 2: Programming with SQL Server 2005, Oracle, and MySQL, Wallace B. McClure, Gregory A. Beamer, IV John J. Croft, J. Ambrose Little, Bill Ryan, Phil Winstanley, David Yack, Jeremy Zongker, WROX Press

● Programming Microsoft ADO.NET 2.0 Applications:

Advanced Topics, Glenn Johnson, Microsoft Press

● Microsoft ADO.NET 2.0 Step by Step, Rebecca Riordan, Microsoft Press

Sofia, Bulgaria | 9-10 October

Contact InfoContact Info

Julie Lerman

jlerman@thedatafarm.com

www.thedatafarm.com

Blogs

www.thedatafarm.com/blog

blogs.ziffdavis.com/devlife

Julie Lerman

jlerman@thedatafarm.com

www.thedatafarm.com

Blogs

www.thedatafarm.com/blog

blogs.ziffdavis.com/devlife

Sofia, Bulgaria | 9-10 OctoberSofia, Bulgaria | 9-10 October

Please fill out the survey forms!

They are the key to amazing prizes that you can get at the end of each day

Please fill out the survey forms!

They are the key to amazing prizes that you can get at the end of each day

Thank you!Thank you!

Sofia, Bulgaria | 9-10 OctoberSofia, Bulgaria | 9-10 October