NoCOUG Journal Latest (1)

8/9/2019 NoCOUG Journal Latest (1)

1/28

Vol. 29, No. 1 FEBRUARY 2015 $15

Knowledge HappensAdapt or PerishJanis Griffin expostulates.

See page 15.

How to Avoid aSalted Banana

Lothar Flatz elucidates.

See page 17.

Five DatabaseTechnology TrendsGuy Harrison prognosticates.

See page 4.

Much more inside . . .


2/28

Catri e Antoinette Meredith Daisy Nada Liz Amie Jeannine Joh

elin Robbi Kathyrn StevenGisele Tim Laura Myron Elizbeth

aym Sun Darell Karan Janine BertLarry Rodrigo Rachel Todd

Sus n Britt

osso Mar g

Daria Brian

Krish a Fran

a Cherri Sa

y Nicholle

ine heron l

Bet y Elizb

ie Jea r

Antoinette

lina obbi K

Jeannine John Sherrill Hellen

2014 Dell, Inc. ALL RIGHTS RESERVED. Dell, Dell Software, the Dell Software logo and productsas identified in this documentare registered trademarks of Dell, Inc. in the U.S.A. and/or othercountries. All other trademarks and registered trademarks are property of their respective owners. PrintAd-ToadConnected-VG-25308

The Toad World community? Its experience. Its knowledge.

Its tips, tricks, and real-time help straight from the experts.

And its inside the latest version of Toad for Oracle. Realizethe power of connected intelligence.

Learn more at software.dell.com/toadconnected

Toad enables connectionswith a community of millions


3/28The NoCOUG Journal

2015 NoCOUG BoardPresidentHanan Hit

Vice PresidentJimmy Brock

SecretaryEric Hutchinson

TreasurerGanesh Sankar Balabharathi

Membership DirectorVacant Position

Conference DirectorSai Devabhaktuni

Vendor CoordinatorOmar Anwar

Training DirectorVacant Position

Social Media DirectorVacant Position

WebmasterJimmy Brock

Journal EditorIggy Fernandez

Marketing DirectorGanesh Sankar Balabharathi

IOUG LiaisonKyle Hailey

Members at LargeJeff Mahe

Linda Yang

Board AdvisorTim Gorman

Board AssociateStephen Van Linge

Book ReviewerBrian Hitchcock

Publication Notices and Submission Format

The NoCOUG Journalis published four times a year by the Northern Cal iforniaOracle Users Group (NoCOUG) approximately two weeks prior to the quarterlyeducational conferences.

Please send your questions, feedback, and submissions to the NoCOUG Journaleditor [email protected].

The submission deadline for each issue is eight weeks prior to the quarterly confer-ence. Article submissions should be made in Microsoft Word format via email.

Copyright by the Northern California Oracle Users Group except where other-wise indicated.

NoCOUG does not warrant the NoCOUG Journalto be error-free.

Special Feature ................................................. 4

NoSQL Corner .................................................. 10

Performance Corner........................................ 15

SQL Corner ........................................................17

Product Review ................................................ 20

Treasurers Report ...........................................23

Tips and Tricks.................................................. 24

ADVERTISERS

Dell Software .................................................... 2

Axxana ............................................................... 9

Embarcadero Technologies .......................... 22

Delphix ............................................................ 26

Database Specialists ...................................... 27

SolarWinds ...................................................... 28

Table of Contents

ADVERTISING RATESTheNoCOUG Journalis published quarterly.

Size Per Issue Per Year

Quarter Page $125 $400

Half Page $250 $800

Full Page $500 $1,600

Inside Cover $750 $2,400

Personnel recruitment ads are not accepted.

[email protected]

Professionals at Work

F

irst there are the IT professionals who write for the Journal. A very

special mention goes to Brian Hitchcock, who has written dozens of

book reviews over a 12-year period. The professional pictures on

the front cover are supplied by Photos.com.

Next, theJournalis professionally copyedited and proofread by veteran copy-

editor Karen Mead of Creative Solutions. Karen polishes phrasing and calls out

misused words (such as reminiscences instead of reminisces). She dots every

i, crosses every t, checks every quote, and verifies every URL.

Then, theJournalis expertly designed by graphics duo Kenneth Lockerbie and

Richard Repas of San Francisco-based Giraffex.

And, finally, Jo Dziubek at Andover Printing Services deftly brings theJournal

to life on an HP Indigo digital printer.

This is the 113thissue of the NoCOUG Journal. Enjoy!s

NoCOUG JournalEditor
http://localhost/var/www/apps/conversion/tmp/scratch_8/[email protected]:[email protected]://localhost/var/www/apps/conversion/tmp/scratch_8/Photos.comhttp://localhost/var/www/apps/conversion/tmp/scratch_8/Photos.commailto:[email protected]://localhost/var/www/apps/conversion/tmp/scratch_8/[email protected]


4/28

4February 2015

Ifirst started working with database technology in the mid-1980s. A lot has changed in database technology over those30 years, but I would have to say that there have been morefundamental changes in databases over the past f ive years

than in the 25 years preceding. I think these changes can beboiled down to a few key trends that are revolutionizing the

databases of today and the future. Those trends include: The end of the era in which a single type of database tech-

nologythe relational databaseserved as a one-size-fits-all-solution.

The emergence of big data technologies, most significant-ly Hadoop.

The explosion of non-relational operational databases(aka NoSQL) such as Cassandra and MongoDB.

The increasing capacity to store data in columnar ratherthan row-oriented format.

The diminishment of the spinning magnetic disk as theonly viable technology for persistently storing data.

Trend #1: The End of Relational Hegemony

Database technology arguably predates digital computing:Many of IBMs first business machines were tabulatorswhichwere able to summarize and analyze data held on punched cards.These punched cards arguably represented a paper database.However, what we think of as computer databases emerged fromthe combination of digital computing and persistent storagemedia such as magnetic disks and tape.

The first digital databasescirca 1950were based on se-quential files that would be scanned into memory on every ac-cess. However, during the 1960s, true database management

systems emerged that combined indexed access methods andstructured schemas. By the end of the 1960s, the hierarchical andnetwork database models were established and pervasive.

Edgar Codd first definedA Relational Model of Data for LargeShared Data Banksin 1970. The relational model had severaladvantages over the existing models: most significantly it sup-ported the concept of ad-hoc queries in which the data modelwas not required to anticipate all of the possible quer ies thatmight be supported. The relational model also required an inter-active query languageeventually standardized on SQLthatallowed non-programmers to execute query operations. Thisinteractive query language broke the logjam of report writingthat plagued IT departments of the time and was a major factor

in the rapid uptake of Relational Database Management Systems(RDBMS).

From the first release of Oracle in 1978 to the middle of the2000s, virtually every significant database released was eitherrelational or claimed to be relational. The database timeline inFigure 1 shows that virtually every database released from 1980

to 2000 wasor at least claimed to bea relational database.

Figure 1. Database timeline

While some non-relational systemsObject-Oriented Data-base Management Systems (OODBMSs) and XML databaseswere developed, none gained significant traction during thisperiod. For an entire generation of software professional, theRDBMS reigned supreme.

The First Cracks Appear

The end of the relational hegemony had been foretold since themiddle of the last decade, most significantly by relational databasepioneer Mike Stonebraker, whose 2005 paper One Size Fits All:

An Idea Whose Time Has Come and Goneargued that modernapplications would be better served by a mix of database systemsoptimized for specific workloads. At the time, the proposed newmodels of DBMS were largely theoretical. However, Stonebrakercorrectly identified the key trend: the complete dominance (thoughnot primary role) of the relational database was about to end.

As we can see from the timeline in Figure 1, between 2005and 2014 literally dozens of new databases were releasedal-most none of which were fully relational.

The Third Platform

There are many factors behind the end of the relational su-premacy, but at a high level, it is a result of the paradigm shifts in

S P E C I A LF E A T U R E

Five Database

Technology Trendswith Guy Harrison

Guy Harrison

Magnetic tape

flat (sequential) files

Pre-computertechnologies:

Printing pressDeweydecimalsystemPunched cards

Magnetic Disk

IMS

RelationalModeldefined

Indexed-SequentialAccess Mechanism(ISAM)

Network Model

IDMS

ADABAS

System R

Oracle V2

Ingres

dBase

DB2

Informix

Sybase

SQL Server

Access

Postgres

MySQL

Cassandra

Hadoop

Vertica

Riak

HBase

Dynamo

MongoDB

Redis

VoltDB

Hana

Neo4J

Aerospike

Hierarchical model

1960-701940-50 1950-60 1970-80 1980-90 1990-2000

2000-2010

Relational Era

Non relational Era

Pre Relational Era
http://www.seas.upenn.edu/~zives/03f/cis550/codd.pdfhttp://www.seas.upenn.edu/~zives/03f/cis550/codd.pdfhttp://cs.brown.edu/~ugur/fits_all.pdfhttp://cs.brown.edu/~ugur/fits_all.pdfhttp://cs.brown.edu/~ugur/fits_all.pdfhttp://cs.brown.edu/~ugur/fits_all.pdfhttp://www.seas.upenn.edu/~zives/03f/cis550/codd.pdfhttp://www.seas.upenn.edu/~zives/03f/cis550/codd.pdf



software application architecture. Weve seen any number ofparadigm shifts within the last three decades, including micro-computer to minicomputer and the introduction of the Internet.However, arguably all of the t hings weve seen fall into threemajor waves of application architecture.

IDC coined the term the third platform to describe the cur-rent application landscape and contrast it to what has come be-fore. In IDCs widely accepted model, software applications haveundergone three fundamental changes:

Figure 2. The third platform (source: IDC)

The first platform was based on mainframe systems ac-cessed through dumb terminals. All processing logic wascentralized. These systems typically supported only hun-dreds of users, although millions of users in aggregate ac-cessed this category of applications.

The second platform emerged with the client server modeland early web applications. These applications divided ap-plication logic into two or more tiers (one of which wastypically the database). As the Internet became a universal

WAN, these applications entered the realm of the main-stream consumer and attracted hundreds of millions ofusers.

The third platform emerged from the nexus of megatrendsthat weve all experienced over the past 10 years: smart-phones, social networking applications, and cloud com-puting have combined to revolutionize not just applicationarchitecture but digital lifestyles and our very societies.Big Data was arguably born out of the third platform butalso strongly influenced its growth. The increasing datagenerated by our mobile and social interactions createsnew avenues for user experience that in turn generatemore data. The increasing prevalence of Internet-connected

sensorsthe Internet of Things (IoT)further acceleratesthe paradigm shift.

A New Database World Order

Databases supporting the third platform need to deal withavailability, throughput, and performance requirements thatwere simply not in existence when the RDBMS was conceived.The demands on databases have included some of the followingpressures:

Global availability

Unstructured data and rapidly changing schemas

Massive volumesdata volumes had been increasingsteadily since the genesis of the database, but new signifi-cant pressure arose to store large amounts of unstructureddata at prices far lower than were possible with a rela-tional schema.

Diverging storage technologies that for the first time cre-ated an increasing divergence between the price of storingdata and the price of accessingdata. Solid-state disk tech-nologies and in-memory database options became avail-

able, which reduced the price of delivering an I/O opera-tion, but these technologies were far more expensive whenit came to storage. It became impossible to provide thebest value dollar/GB and best va lue dollar/IOPS in thesame database technology and storage architecture.

The economics of various components of the technologystack also increasingly demanded changes in core data-base architectures. CPU and memory capacity increasedexponentially in accordance with Moores law, while net-work and disk capacity increased geometrically at best. Asa result, it became increasingly more economical to moveprocessing to the data, rather than to move data to the pro-

cessing.Each of these will be discussed in upcoming sections, but in

overview we are currently in an era in which several differentdatabase technologies are employed for most moderately sizedbusinesses:

The relational system remains dominant as the underlyingstorage system for traditional business systemsin-houseERP and CRM. For instance, no non-relational system iscertified to support a SAP implementation and there islittle sign that this will change. However, as ERP and CRMsystems migrate to cloud-based SaaS alternatives, manywill be re-engineered and some may potentially adopt a

non-relational data store. The relational database remains entrenched as the basis

for virtually all in-house data warehouses, and BI tools ingeneral only deliver their full functionality when workingagainst relational systems that support the full range ofANSI SQL, including windowing functions, star schemaoptimizations, bitmap indexes, and so on. Again, there issome threat here from SQL-on-Hadoop, but for the mostpart the real-time data warehouse remains the province ofthe relational system.

Hadoop is increasingly deployed alongside the relationaldata warehouse as a data lake. Hadoop serves as the stag-

ing area for data that has yet to find its way into a fixed-schema DW and as a repository for fine-grained raw datathat may be the subject of data science projects.

Web applicationsparticularly those that in the pastwould have been based on the LAMP (Linux, Apache,MySQL & Perl/Python/PHP) stackare increasing basednot on a relational system such as MySQL but on a non-relational alternative such as MongoDB. These decisionsare largely being driven from the ground up as developersdecide on the NoSQL database in the absence of any cor-porate mandate and in projects that adopt the DevOpsstyle of application development.


6/28

6February 2015

In addition to the widespread use cases outlined above,there are increasingly viable niches for NewSQL or oth-erwise not-quite-relational database systems. These in-clude columnar variations on the data warehouse such asVertica, in-memory systems such as Hana and VoltDB,and graph-based databases such as Neo4J.

Figure 3. Modern enterprise data architectures combine multipletechnologies

Trend #2: Big Data and Hadoop

Every few years we seem to be subjected to a new buzzwordthat so rapidly dominates all conversation as to become mean-ingless. Many believe the term big dataseemingly added toany conversation that in any way relates to datahas becomesuch a buzzword. But in reality, big data encompasses two veryreal trends in modern data management:

We are required to store more, and more varied, data thanever before.

The use of data has shifted in importance: analysis of dataused to be primarily about operational efficiency; in themodern era it has become about competitive advantage.

The Data Revolution

While there are many arguably valid definitions of big data,my personal favorite is the Industrial Revolution metaphorcoined by Joe Hellerstein at OReilly. Before the IndustrialRevolution in the 1800s, all goods were hand produced. Likewise,before the industrial revolution of data, all data was produced byhand: entered by individuals paid to do so by the organization.Nowadays, of course, data comes in from everywhere: custom-ers, social networks and, increasingly, Internet-enabled devices.

From a database point of view, this involves more than juststoring qualitatively more data. We are being called upon tomanage the storage of fast amounts of unstructured, unprocessed

data. In the past data was cleansed, schematized and aggregatedprior to being loaded into a data warehouse. This still occurs, butwe are also required to store the original, unadulterated raw data.Why? Because often its the analysis of this original data that canbe used to unlock competitive advantage through advanced tech-niques of data science.

Google and Hadoop

Undeniably, Google is the pre-eminent pioneer of big data.From the very beginning, Google was faced with the challenge ofstoring and analyzing exponentially increasing volumes of data,and managing data that was inherently flexible and evolving informat.

No existing database management system had either the ca-pacity or capability of storing the volumes and varieties of datathat Google needed to master. So Google was forced to adopt itsown hardware and software stack for the purpose of building ascalable commercial solution.

This stack involved many components but most notably in-cluded the following building block technologies:

Google File System (GFS) is a distributed file system thatallows the directly attached storage in very large numbers

of commodity servers to be exposed as a single logical filesystem.

Map Reduce is a programming model that allows complexproblems to be broken up into simple parallelized mapfunctions together with reduce functions that combine theoutputs from each parallel stream. Multiple Map Reducepipelines can be constructed to solve a very wide varietyof problems. While somewhat simplistic (at least com-pared to modern alternatives) Map Reduce is an algorithmthat is applicable to a wide variety of problems.

Bigtable is a non-relational database system built on GFS.It uses a normalized data model in which rows are associ-

ated with a variable, sparse, and potentially very largenumber of columns, avoiding in many cases the need forseparate storage of detail records and for joins.

Google published details of the above three technologies dur-ing the latter part of the last decade. Consequently, an open-source project arose that implemented these and other keycomponents of the Google stack. This project is, of course,Hadoop, which is arguably the most significant new databasestorage system to be released in the last decade.

Hadoop was nurtured at Yahoo!, which allowed it to achievefairly rapidly the massive scalability for which it is well known. Itprovides compelling advantages over relational systems for bigdata storage:

It is significantly cheaper than all alternatives for storinglarge amounts of data online. This is what Forrester callsHadooponomics: the ability to linearly scale data storageand data-processing costs.

It supports the storage and processing of unstructured andsemi-structured data. There is no requirement to define theschema of a data item for storing it in Hadoop. While thisundoubtedly leads to some degree of Hadumpingthethoughtless dumping of all data without analysisit obviatesthe delay and cost involved with ETL into traditional stores.

Libraries of data science and advanced analysis exist forHadoop, providing data scientists with at least a starting

point for the analysis of data held in Hadoop.

OperationalRDBMS

(Oracle, SQLServer, )

In-memoryAnalytics(HANA,

Exalytics )

In-memoryprocessing

(Spark)

Hadoop

Web DBMS(MySQL,Mongo,

Cassandra)

ERP & in-house CRM

Analytic/BIsoftware

(SAS,

Tableau)

Web ServerData

WarehouseRDBMS

(Oracle,Teradata )

It takes all sorts

SQOOP

(RDBMS loader)Hive

(Query)

Pig

(Scripting)

Flume

(Log Loader)

Oozie (Workflow manager)

Hadoop File System (HDFS)

Map Reduce /

YARN

Hbase

(Database)

Zookeeper

(Locking)

Figure 4. Simplified Hadoop stack
http://oreil.ly/1DDVVb1http://bit.ly/1DE0IJyhttp://bit.ly/1DE0IJyhttp://oreil.ly/1DDVVb1



Today, Hadoop forms the background of most big dataprojects.

Trend #3: Distributed Non-Relational Databases (a.k.a.NoSQL)

Hadoop is an example of a non-relational database system.However, while Hadoop arose because of pressures on analysis ofmassive amounts of data, other non-relational systems were de-

veloped to meet other needs.

Traditionally, start-ups relied heavily on open-source softwarestacks: Linux, Apache, and MySQL were the key components ofweb start-ups such as Facebook and Twitter. However, as thesesites experienced rapid growth, it became increasingly difficultto scale systems built on these foundations.

Means of scaling the web server are well established, but thedatabase presented different challenges. Some degree of relativelyeasy scalability can be achieved by read offloading: using read onlyreplicas or distributed object cases such as Memcached. However,at some point the master database becomes the bottleneck.

The most common solution to this dilemma is sharding. Ina sharded database, the largest tables are partitioned across mul-tiple nodes. This partitioning is based on some key value, such as

a user ID. The application must be modified to know how to readfrom the appropriate shard as well as typically integrating code tohandle memory-cached copies of data and the read-only replicas.

Although sharding succeeded as a solution and is still in usein sites such as Facebook and Twitter, it is widely regarded as apoor solution. A sharded database has lost almost all the charac-teristics of a relational system: joins are not possibleand transac-tional logic can no longer be expressed simply in SQL.

It has been recognized for some time that the requirementsof massive scalability and availability common to the biggest webapplications are inconsistent with the transactional model ex-pressed in relational systems. This inconsistency is expressed inthe CAP (aka Brewers) theorem.

The CAP theorem states that you can only have two of thesethree properties in a system:

Consistency: Everyone always sees the same version of alldata.

Availability: The system can remain available when nodesfail.

Partition tolerance: The system can survive and remainavailable when split in two by a network partition.

While a system such as Oracle RAC could provide high avail-ability and strong consistency, a new model was needed for sys-tems that would prefer to sacrifice strong consistency in order tomaintain availability across multiple data centers and geographies.

Amazon published their model for highly available data stor-age in 2007, coining the term Eventually Consistent to describethis model. Together with the Google Bigtable model discussedearlier, these ideas inspired a variety of NoSQL databases toemerge, including Hbase, Riak, DynamoDB, and Cassandra.

Non-relational systems were also influenced by object-orient-ed and XML databases. Programmers have been frustrated for along time over the need to map program objects into relationalform. The so-called impedance mismatch between the twomodels creates programmer overhead and hard-to-maintaincode. As a consequence of the emergence of JSON (JavaScriptObject Notation) as a widely used object document model in web

applications and the validation of non-relational systems, a num-ber of JSON-based non-relational databases emerged, such asMongoDB and Couchbase.

Today these non-relational systemsusually referred to asNoSQL databasesare an increasingly common choice for back-ending web applications.

Trend #4: Columnar Databases

The natural way that most of us think about data is to imaginethe data organized into horizontal rows and vertical columnsthe same way that we might enter the data on index cards, apaper ledger, or Excel. This organization suited early databasesystems as well, since most operations were on individual rows/records: the well-known CRUD (Create, Read, Update, Delete)pattern.

However, in data analysis it is rare to perform an analysis onall the columns in a single row; rather, we tend to do operationson all the rows in a single column. This shift toward column-based processing created the opportunity for a new organiza-

tional model for data: the columnar storage model.

Figure 6. Row-oriented and column-oriented databases

In a column store, columns tend to be organized together ondisk rather than by clustering rows together on disk (Figure 6).This has several notable advantages:

Aggregate operations such as SUM(), AVG(), etc., typi-cally need to do less I/O, since all the data they need isgrouped together on disk.

Compression is improved, since there tends to be morerepetition across columns than across rows.

Consistency

Everyone always seesthe same data

Availability System stays up

when nodes fail

PartitionTolerance

System stays upwhen networkbetween nodesfails

NO

GO

Most NoSQL liveshere

Figure 5. CAP (Brewers) theorem

ID Name DOB Salary Sales Expenses1001 Dick 21/12/60 67,000 78980 3244

1002 Jane 12/12/55 55,000 67840 2333

1003 Robert 17/02/80 22,000 67890 6436

1004 Dan 15/03/75 65,200 98770 2345

1005 Steven 11/11/81 76,000 43240 3214

Block ID Name DOB Salary Sales Expenses

1 1001 Dick 21/12/60 67,000 78980 3244

2 1002 Jane 12/12/55 55,000 67840 2333

3 1003 Robert 17/02/80 22,000 67890 6436

4 1004 Dan 15/03/75 65,200 98770 2345

5 1005 Steven 11/11/81 76,000 43240 3214

Block

1 Dick Jane Robert Dan Steven

2 21/12/60 12/12/55 17/02/80 15/03/75 11/11/81

3 67,000 55,000 22,000 65,200 76,000

4 78980 67840 67890 98770 43240

5 3244 2333 6436 2345 3214

Row-oriented database

Column-oriented database

Dataset


8/28

8February 2015

Figure 7. Delta store within a column store

Weighed against these advantages is, primarily, a reduction inefficiency for typical OLTP operations. An insert, for instance,must perform as many block operations as there are columns inthe row. Continual inserts effectively require continual reorgani-zation of the column store, which is rarely practical. For thisreason, most column stores support the idea of a delta store: astaging area that is row organized. Inserts build up in this deltastore and are periodically merged with the column store. In this

interim period, queries must access both the delta store and thecolumn store in order to retrieve up-to-date data.Sybase IQ was one of the first commercial column-store data-

bases. Mike Stonebraker and his colleagues defined a theoreticalmodel for column stores called C-store that heavily influencedthe design of HP Vertica and other column-oriented systems.

Oracle uses column-oriented architecture within bothExadata Hybrid Columnar Compression (EHCC) and OracleDatabase 12cIn-Memory Option.

In EHCC, each block contains data for specific columns, butall columns for a specific row are held within a single compres-sion unittypically a 1 MB structure. EHCC allows Oracle to getthe higher compression rates allowed for by columnar storage

while limiting impact on single-row operations, which can stillbe satisfied by a single 1 MB I/O.

Well consider Oracle Database 12cIn-Memory Option in thenext section.

Trend #5: The Demise of the Spinning Disk

The magnetic disk device has been a ubiquitous presencewithin digital computing since the 1950s. The essential architec-

ture has changed very little over that time: one or more platterscontain magnetic charges that represent bits of information.These magnetic charges are read and written by an actuator arm,which moves across the disk to a specific position on the radiusof the platter and then waits for the platter to rotate to the ap-propriate location.

The time taken to read an item of information is the sum ofthe time taken to move the head into position (seek time), thetime taken to rotate the item into place (rotational latency), and

the time taken to transmit the item through the disk controller(transfer time).

Figure 8 illustrates that while the size and density of thesedevices have changed over the years, the architecture remains

virtually identical. While Moores law drives exponential growthin CPU, memory, and disk density, it does not apply to the me-chanical aspects of this performance; consequently, magneticdisks have become an increasing drag on database performanceover the years.

The promise of solid-state disks has led some to anticipate aday when all magnetic disks are replaced by solid-state disks.While this might someday come to pass, in the short term theeconomics of storage and the economics of I/O are at odds: mag-

netic disk technology provides a more economical medium perunit of storage, while flash technology provides a more eco-nomical medium for delivering high I/O rates and low latencies.

Figure 9. Economics of storage for solid-state and magnetic disktechnologies

Figure 9 illustrates the two competing trends: while the costof I/O is reduced with solid-state technology, the cost per TBincreases. Various flavors of SSD (PCI/SATA and MLC/SLC)offer different price and performance characteristics in mag-netic disks (15 K vs. 7 K RPM, for instance). The SSD devicesthat offer good economies of I/O offer poorer economies formass storage. Of course, the cost per gigabyte for SSD is drop-ping rapidly, but not faster than the falling cost of magneticdisks or the growth in database storage demandespecially inthe era of big data.

Since most databases include both hot and cold datasmallamounts of frequently accessed data as well as large amounts ofidle datamost databases will experience the best economicbenefit by combining both solid-state and traditional magneticdisk technologies. This is why Exadata combines both magneticdisks and flash disks to achieve the ability to provide the optimalbalance between storage economics and performance. If Exadatacontained only magnetic disks, it could not provide superior

Read Optimized Store

Columnar

Disk-based

Highly compressed

Bulk loadable

Delta store

Row oriented

Uncompressed

Single row inserts

Asynchronous merge

Bulk sequential loads

Continual parallel inserts

Merged

query

Figure 8. Disk devices over the years (Photo courtesy of Paul R.Potts)

1.27

0.50

0.05

0.12

0.06

0.04

0.83

2.93

13.41

26.83

0.00 5.00 10.00 15.00 20.00 25.00 30.00

0.00 0.20 0.40 0.60 0.80 1.00 1.20 1.40

Capacity HDD

Performance HDD

SATA SSD

MLC PCI SSD

PCI SLC SSD

Dollar/GB

Dollar/IO

Dollar/IOP

Dollar/GB



OLTP performance; if it contained only SSD, it could not offercompelling economical storage for large databases.

Moores law also continues to drive an increase in memorycapacity and decrease in memory cost. Many smaller databasescan now fit entirely within the memory capacity of a singleserver, and certainly within the memory capacity of a cluster. Forthese databases an in-memory solution may be even more attrac-tive than a full SSD architecture.

However, these advantages can only be realized fully by a non-

traditional database architecture. Simply creating a massive buf-fer case on top of an Oracle database, for instance, will notprovide benefits for all operations. Full table scans will still usedirect path reads to disk and checkpoints and redo operationsstill need to run to a persistent storage layer.

Therefore databases that seek to be truly in memory typicallyuse different architectural patterns. For instance in Times 10,data is guaranteed to always be in memory, and no user opera-tion ever waits for disk I/O (although disks are still used to storesnapshots and transaction logs). By default, a commit writesasynchronously to the transaction log, so that the user does notwait on the IO. Strictly speaking this violates the ACID transac-tion model, since a committed transaction could be lost if the

database failed before the data made it to disk.HANA employs a mix of columnar and row-based storage,

and also requires persistent disks (often SSDs) for transaction logand checkpoint writes.

One of the purest in-memory databases is VoltDB. VoltDBuses a cluster of in-memory partitions that replicate data acrossmachines. During a commit, data is written to multiple ma-chines, ensuring that data is not lost should there be a powerfailure. This approach is called K-safety.

VoltDB also eliminates latch waits by al lowing only a singlethread to access a partition at any moment. This is an interestingaspect of in-memory systems, since when you eliminate I/Owaits you are likely to find that latch waits, which prevent simul-

taneous memory accesses, become the next logical bottleneck.Oracle Database 12cIn-Memory Option uses a combination

of columnar and in-memory approaches. Data is still held in thetraditional buffer cache and data files, but it can also be held inan in-memory column format. As with other column stores,there is a row store for deltas (the SMU) that buffers modifica-tions.

Cloud: The Missing Trend?

Its been true for several years that you can add the phrase inthe cloud to almost any technology marketing phrase to in-crease its value. Since cloud is such a big trend, should data-bases in the cloud or Database as a Service (DBaaS) be a keydatabase technology trend?

Possiblyin that I tend to believe that the database architec-tures in the cloud are essentially driven by the database architec-tures that we see on premise, especially those being pioneered inthe largest Web 2.0 companies. Furthermore, DBaaS uptake willbe slower than other cloud categories, due to some of the uniquecharacteristics of databases:

When the database is remote from the application it serves,network latencies become a limiting performance factor thatcannot be overcome. A lot of work has gone into HTTP to makeit very responsive on web pages through things like staged ren-dering and batching of request/reply packets; less work has gone

into shaping database serverapplication server network trafficbecause its assumed that they are both on the same local areanetwork or on a dedicated interconnect.

For a similar reasonassumption that the application serverand database server are on a private networkthe database com-munication and authentication protocols are not as hardened asin HTTPS. Data items may be transmitted in plain text, and au-thentication is often restricted to one super account that can doanything. Consequently, opening the database listener port to

the Internet potentially exposes the entire database to attack orto successful data sniffing.

Data sovereignty and other considerations apply most strong-ly to the database, so its probably the last element of your on-premise servers to be moved to cloud.

None of this is to say that databases wont move into the cloudover time, but in my opinion they will follow applications intothe cloud rather than lead the way. s

Guy Harrison is an Oracle ACE and executive director of researchand development at Dell. He is the author of Oracle PerformanceSurvival Guide (Prentice Hall, 2009) and MySQL Stored ProcedureProgramming (OReilly, with Steven Feuerstein) as well as other

books, articles, and presentations on database technology. He alsowrites a monthly column for Database Trends and Applications(www.dbta.com). He is co-author of the upcomingOracle ExadataExpert Handbook(Pearson, 2015). Guy can be found on the Inter-net at www.guyharrison.netand on e-mail at [email protected], and he is @guyharrisonon Twitter.

Copyright 2015, Guy Harrison

[email protected] www.axxana.com

Axxanas award winning Phoenix Systemoffering unprecedented Data Protection,Cross-application consistency, Storage and

Replication agnostic.

SIMPLE!

DATABASE RECOVERY HAS NEVER BEEN SO
http://www.dbta.com/http://www.guyharrison.net/http://www.guyharrison.net/http://www.dbta.com/


10/28

0February 2015

N O SQ LC O R N E R

NoSQL is a disruptive innovation in the sense usedby Harvard professor Clayton M. Christensen. InThe Innovators Dilemma: When New TechnologiesCause Great Firms to Fail, Professor Christensen

defines disruptive innovations and explains why it is dangerousto ignore them:

Generally, disruptive innovations were technologically straight-forward, consisting of off-the-shelf components put together ina product architecture that was often simpler than prior ap-

proaches. They offered less of what customers in establishedmarkets wanted and so could rarely be initially employed there.They offered a different package of attributes valued only inemerging markets remote from, and unimportant to, the main-stream.

The personal computer was an example of a disruptive inno-vation. It was initially targeted only at the home computing seg-ment of the market. Established manufacturers of mainframecomputers and minicomputers did not see PC technology as a

threat to their bottom lines. Eventually, however, PC technologycame to dominate the market, and established computer manu-facturers such as Digital Equipment Corporation, Prime, Wang,Nixdorf, Apollo, and Silicon Graphics went out of business.

So where lies the dilemma? Professor Christensen explains:

In every company, every day, every year, people are going intosenior management, knocking on the door saying: I got a new

product for us. And some of those entail making better productsthat you can sell for higher prices to your best customers. Adisruptive innovation generally has to cause you to go after newmarkets, people who arent your customers. And the productthat you want to sell them is something that is just so much

more affordable and simple that your current customers cantbuy it. And so the choice that you have to make is: Should wemake better products that we can sell for better profits to ourbest customers. Or maybe we ought to make worse productsthat none of our customers would buy that would ruin ourmargins. What should we do? And that really is the dilemma.

Exactly in the manner that Christensen described, the e-commerce pioneer Amazon created an in-house product calledDynamo in 2007 to meet the performance, scalability, and avail-ability needs of its own e-commerce platform after it concludedthat mainstream database management systems were not capableof satisfying those needs. The most notable aspect of Dynamo

was the apparent break with the relational model; there was nomention of relations, relational algebra, or SQL.

Dynamo Requirements and Assumptions

Amazon started out by using Oracle Database for its e-com-merce platform but later switched to a proprietary database

management system called Dynamo that it built in-house.Dynamo is the archetypal NoSQL product; it embodies all theinnovations of the NoSQL camp. The Dynamo requirements andassumptions are documented in the paper Dynamo: AmazonsHighly Available Key-value Store (http://s3.amazonaws.com/AllThingsDistributed/sosp/amazon-dynamo-sosp2007.pdf),published in 2007. Here are some excerpts from that paper:

Customers should be able to view and add items to their shop-ping cart even if disks are failing, network routes are flapping,or data centers are being destroyed by tornados. Therefore, theservice responsible for managing shopping carts requires that itcan always write to and read from its data store, and that its

data needs to be available across multiple data centers.There are many services on Amazons platform that only need

primary-key access to a data store. For many services, such asthose that provide best seller lists, shopping carts, customer pref-erences, session management, sales rank, and product catalog,the common pattern of using a relational database would leadto inefficiencies and limit scale and availability. Dynamo pro-vides a simple primary-key only interface to meet the require-ments of these applications.

Experience at Amazon has shown that data stores that provideACID guarantees tend to have poor availability.

Dynamo targets applications that operate with weaker consis-tency (the C in ACID) if this results in high availability.

. . . since each service uses its distinct instance of Dynamo, itsinitial design targets a scale of up to hundreds of storage hosts.

To paraphrase, Amazons requirements were extremeperfor-mance, extremescalability, andextremeavailability.

Functional Segmentation

Amazonspivotaldesign decision was to break its monolithicenterprise-wide database service into simpler component ser-

vices, such as a best-seller list service, a shopping cart service, acustomer preferences service, a sales rank service, and a product

The Rise and Fall of

the SQL Empireby Iggy Fernandez

Iggy Fernandez
http://www.amazon.com/The-Innovators-Dilemma-Technologies-Management/dp/142219602X/http://www.amazon.com/The-Innovators-Dilemma-Technologies-Management/dp/142219602X/http://www.amazon.com/The-Innovators-Dilemma-Technologies-Management/dp/142219602X/http://www.amazon.com/The-Innovators-Dilemma-Technologies-Management/dp/142219602X/



catalog service. This avoided a single point of failure. In an inter-view for the NoCOUG Journal, Amazons first database adminis-trator, Jeremiah Wilton explains the rationale behind Amazonsapproach:

The best availability in the industry comes from applicationsoftware that is predicated upon a surprising assumption: Thedatabases upon which the software relies will inevitably fail.The better the softwares ability to continue operating in such asituation, the higher the overall services availability will be. Butisnt Oracle unbreakable? At the database level, regardless of themeasures taken to improve availability, outages will occur fromtime to time. An outage may be from a required upgrade or abug. Knowing this, if you engineer application software to handlethis eventuality, then a database outage will have less or noimpact on end users. In summary, there are many ways to im-

prove a single databases availability. But the highest availabilitycomes from thoughtful engineering of the entire application ar-chitecture. (http://www.nocoug.org/Journal/NoCOUG_Journal_200711.pdf#page=4)

As an example, the shopping cart service should not be affectedif the checkout service is unavailable or not performing well.

I said that this was the piv ota ldesign decision made byAmazon. I cannot emphasize this enough. If you resist func-tional segmentation, you are notready for NoSQL. If you missthe point, you will not understand NoSQL.

Note that functional segmentation results in simple hierarchi-cal schemas. Here is an example of a simple hierarchical schemafrom Ted Codds 1970 paper on the relational model (http://www.nocoug.org/Journal/NoCOUG_Journal_201111.pdf#page=10). This simple schema stores information aboutemployees, their children, their job histories, and their salaryhistories.

employee (man#, name, birthdate)

children (man#, childname, birthyear)

jobhistory (man#, jobdate, title)

salaryhistory (man#, jobdate, salarydate, salary)

Functional segmentation is the underpinning of NoSQL tech-nology, but it does not present a conflict with the relationalmodel; it is simply a physical database design decision. Eachfunctional segment is usually assigned its own standalone data-base. The collection of functional segments could be regarded asa single distributed database. However, distributed transactionsare forbidden in the NoSQL world. Functional segmentation cantherefore result in temporary inconsistencies if, for example, theshopping cart data is not in the same database as the productcatalog, and occasional inconsistencies result. As an Amazon

customer, I occasionally leave items in my shopping cart butdont complete a purchase. When I resume shopping, I some-times get a notification that an item in my shopping chart is nolonger in stock or has been repriced. This technique is calledeventual consistency. Randy Shoup, one of the architects ofeBays ecommerce platform, explains:

At eBay, we allow absolutely no client-side or distributed trans-actions of any kind no two-phase commit. In certain well-definedsituations, we will combine multiple statements on a single databaseinto a single transactional operation. For the most part, however,individual statements are auto-committed. While this intentionalrelaxation of orthodox ACID properties does not guarantee imme-

diate consistency everywhere, the reality is that most systems areavailable the vast majority of the time. Of course, we do employvarious techniques to help the system reach eventual consistency:careful ordering of database operations, asynchronous recoveryevents, and reconciliation or settlement batches. We choose thetechnique according to the consistency demands of the particularuse case.(http://www.infoq.com/articles/ebay-scalability-best-practices)

ShardingAmazons next design decision was sharding or horizontal

partitioning of all the tables in a hierarchical schema. Hash-partitioning is typically used. Each table is partitioned in thesame way as the other tables in the schema, and each set of parti-tions is placed in a separate database referred to as a shard. Theshards are independent of each other; that is, there is no cluster-ing as in Oracle RAC.

Note that the hierarchical schemas that result from functionalsegmentation are always shardable; that is, hierarchical schemasare shardable by definition.

Returning to the example from Ted Codds 1970 paper on therelational model:

employee (man#, name, birthdate) with primary key (man#)

children (man#, childname, birthyear) with primary key (man#, childname)

jobhistory (man#, jobdate, title) with primary key (man#, jobdate)

salaryhistory (man#, jobdate, salarydate, salary) with primary key (man#, jobdate,

salarydate)

Note that the jobhistory, salaryhistory, and children tableshave composite keys. In each case, the leading column of thecomposite key is the man#. Therefore, all four tables can be par-titioned using the man#.

Sharding is an essential component of NoSQL designs but itdoes not present a conflict with the relational model; it too issimply a physical database design decision. In the relational

model, the collection of standalone databases or shards can belogically viewed as a single distributed database.

Replication and Eventual Consistency

The Dynamo developers saw that one of the keys to extremeavailability was data replication. Multiple copies of the shoppingcart are allowed to exist and, if one of the replicas becomes unre-sponsive, the data can be ser ved by one of the other replicas.However, because of network latencies, the copies may occasion-ally get out of sync, and the customer may occasionally encoun-ter a stale version of the shopping cart. Once again, this can behandled appropriately by the application tier; the node that fallsbehind can catch up eventually, or inconsistencies can be de-

tected and resolved at an opportune time, such as at checkout.This technique is called eventual consistency.

The inventor of relational theory, Dr. Codd, was acutely awareof the potential overhead of consistency checking. In his 1970paper, he said:

There are, of course, several possible ways in which a systemcan detect inconsistencies and respond to them. In one approachthe system checks for possible inconsistency whenever an insertion,deletion, or key update occurs.Naturally, such checking will slowthese operations down[emphasis added]. If an inconsistency hasbeen generated, details are logged internally, and if it is not reme-died within some reasonable time interval, either the user or some-
http://www.nocoug.org/Journal/NoCOUG_Journal_200711.pdf#page=4http://www.nocoug.org/Journal/NoCOUG_Journal_200711.pdf#page=4http://www.nocoug.org/Journal/NoCOUG_Journal_201111.pdf#page=10http://www.nocoug.org/Journal/NoCOUG_Journal_201111.pdf#page=10http://www.nocoug.org/Journal/NoCOUG_Journal_201111.pdf#page=10http://www.nocoug.org/Journal/NoCOUG_Journal_201111.pdf#page=10http://www.nocoug.org/Journal/NoCOUG_Journal_201111.pdf#page=10http://www.nocoug.org/Journal/NoCOUG_Journal_201111.pdf#page=10http://www.nocoug.org/Journal/NoCOUG_Journal_200711.pdf#page=4http://www.nocoug.org/Journal/NoCOUG_Journal_200711.pdf#page=4


12/28

2February 2015

one responsible for the security and integrity of the data is notified.Another approach is to conduct consistency checking as a batchoperation once a day or less frequently.In other words, the inven-tor of relational theory would not have found a conflict betweenhis relational model and the eventual consistency that is oneof the hallmarks of the NoSQL products of today. However, theDynamo developers imagined a conflict because it quite under-standably conflated the relational model with the ACID guar-antees of database management systems. However, ACID has

nothingto do with the relational model per se (although rela-tional theory does come in very handy in defining consistencyconstraints); pre-relational database management systems suchas IMS provided ACID guarantees and so did post-relationalobject-oriented database management systems.

The tradeoff between consistency and performance is as im-portant in the wired world of today as it was in Dr. Codds world.Synchronous replication is rarely used in the relational camp, sowe cannot frown at Dynamo for not using it. Application devel-opers in the relational camp are warned about the negative im-pact of consistency checking, so we cannot frown on Dynamosdecision to permit temporary inconsistencies between func-tional segments.

Using primary and foreign keys can impact performance.Avoid using them when possible. (http://docs.oracle.com/cd/E17904_01/core.1111/e10108/adapters.htm#BABCCCIH)

For performanc e reasons, the Oracle BPEL ProcessManager, Oracle Mediator, human workflow, Oracle B2B,SOA Infrastructure, and Oracle BPM Suite schemas haveno foreign key constraints to enforce integrity. (http://docs.oracle.com/cd/E23943_01/admin.1111/e10226/soaadmin_partition.htm#CJHCJIJI)

For database independence, applications typically do not

store the primary key-foreign key relationships in the da-tabase itself; rather, the relationships are enforced in theapplication. (http://docs.oracle.com/cd/E25178_01/fusionapps.1111/e14496/securing.htm#CHDDGFHH)

The ETL process commonly verifies that certain con-straints are true. For example, it can validate all of theforeign keys in the data coming into the fact table. Thismeans that you can trust it to provide clean data, insteadof implementing constraints in the data warehouse.(http://docs.oracle.com/cd/E24693_01/server.11203/e16579/constra.htm#i1006300)

The False Premise of NoSQL

The final hurdle was extreme performance, and thats wherethe Dynamo developers went astray. The Dynamo developersbelieved that the relational model imposes a join penalty andtherefore chose to store data as blobs. This objection to the re-lational model is colorfully summarized by the following state-ment attributed to Esther Dyson, the editor of the Release 1.0newsletter, Using tables to store objects is like driving your carhome and then disassembling it to put it in the garage. It can beassembled again in the morning, but one eventually asks whetherthis is the most efficient way to park a car.The statement datesback to 1988 and was much quoted when object-oriented data-bases were in vogue.

Since the shopping cart is an object, doesnt disassembling itfor storage make subsequent data retrieval and updates ineffi-cient? The belief stems from an unfounded assumption that hasfound its way into every relational DBMSthat ever y tableshould map to physical storage. In reality, the relational model isa logical model and, therefore, it does not concern itself withstorage details at all. It would be perfectly legitimate to store theshopping cart in a physical form that resembled a shopping cartwhile still offering a relational model of the data complete with

SQL. In other words, the physical representation could be opti-mized for the most important use caseretrieving the entireshopping-cart object using its keywithout affecting the rela-tional model of the data. It would also be perfectly legitimate toprovide a non-relational API for the important use cases. Dr.Codd himself gave conditional blessing to such non-relationalAPIs in his 1985 Computerworldarticle, Is Your DBMS ReallyRelational?, in which he says, If a relational system has a low-level (single-record-at-a-time) language, that low level [should not]be used to subvert or bypass the integrity rules and constraintsexpressed in the higher level relational language (multiple-records-at-a-time).

The key-blob or key-value approach used by Dyna moand successor products would be cal led zeroth normal formin relational terminology. In his 1970 paper, A RelationalModel of Data for Large Shared Data Banks, Dr. Codd says: Nonatomic values can be discussed within the relational frame-work. Thus, some domains may have relations as elements.These relations may, in turn, be defined on nonsimple do-mains, and so on. For example, one of the domains on whichthe relation employee is defined might be salary history. Anelement of the salary history domain is a binary relation de-

fined on the domain date and the domain salary. The salaryhistory domain is the set of all such binary relations. At anyinstant of time there are as many instances of the salary history

relation in the data bank as there are employees. In contrast,there is only one instance of the employee relation.

In common parlance, a relation with non-simple domains issaid to be in zeroth normal form or unnormalized. Dr. Coddsuggested that unnormalized relations should be normalized forease of use. Here again is the unnormalized employee relationfrom Dr. Codds paper:

employee (

employee#,

name,

birthdate,

jobhistory (jobdate, title, salaryhistory (salarydate, salary)),

children (childname, birthyear)

)

The above unnormalized relation can be decomposed intofour normalized relations as follows.

employee' (employee#, name, birthdate)

jobhistory' (employee#, jobdate, title)

salaryhistory' (employee#, jobdate, salarydate, salary)

children' (employee#, childname, birthyear)

However, this is notto suggest that these normalized relationsmust necessarily be mapped to individual buckets of physicalstorage. Dr. Codd differentiated between the stored set, thenamed set, and the expressible set. In the above example, we have
http://docs.oracle.com/cd/E17904_01/core.1111/e10108/adapters.htm#BABCCCIHhttp://docs.oracle.com/cd/E17904_01/core.1111/e10108/adapters.htm#BABCCCIHhttp://docs.oracle.com/cd/E17904_01/core.1111/e10108/adapters.htm#BABCCCIHhttp://docs.oracle.com/cd/E23943_01/admin.1111/e10226/soaadmin_partition.htm#CJHCJIJIhttp://docs.oracle.com/cd/E23943_01/admin.1111/e10226/soaadmin_partition.htm#CJHCJIJIhttp://docs.oracle.com/cd/E23943_01/admin.1111/e10226/soaadmin_partition.htm#CJHCJIJIhttp://docs.oracle.com/cd/E24693_01/server.11203/e16579/constra.htm#i1006300http://docs.oracle.com/cd/E24693_01/server.11203/e16579/constra.htm#i1006300http://docs.oracle.com/cd/E24693_01/server.11203/e16579/constra.htm#i1006300http://docs.oracle.com/cd/E24693_01/server.11203/e16579/constra.htm#i1006300http://docs.oracle.com/cd/E23943_01/admin.1111/e10226/soaadmin_partition.htm#CJHCJIJIhttp://docs.oracle.com/cd/E23943_01/admin.1111/e10226/soaadmin_partition.htm#CJHCJIJIhttp://docs.oracle.com/cd/E23943_01/admin.1111/e10226/soaadmin_partition.htm#CJHCJIJIhttp://docs.oracle.com/cd/E17904_01/core.1111/e10108/adapters.htm#BABCCCIHhttp://docs.oracle.com/cd/E17904_01/core.1111/e10108/adapters.htm#BABCCCIHhttp://docs.oracle.com/cd/E17904_01/core.1111/e10108/adapters.htm#BABCCCIH



one unnormalized relation and four normalized relations. If wepreferred, the unnormalized employee relation could be the onlymember of the stored set. Alternatively, if we preferred, allfiverelations could be part of the stored set; that is, we could legiti-mately store redundant representations of the data. However, thecommon belief blessed by current practice is that the normalizedrelations should be the only members of the stored set.

Even if the stored set contains only normalized relations,they need not map to different buckets of physical storage.

Oracle Database is unique among mainstream databasemanagement systems in providing a convenient constructcalled the table cluster that is suitable for hierarchicalschemas. In Dr. Codds example, employee# would be thecluster key, and rows corresponding to the same clusterkey from a ll four tab les could be stored in the same physicalblock on disk, thus avoiding the join penalty. If the clusterwas a hash cluster, no indexes wou ld b e r equ ire d fo r t heus e c as e of ret ri ev in g re cor ds belonging to a single clusterkey. A demonstration is available at http://iggyfernandez.wordpress.com/2013/12/30/the-twelve-days-of-nosql-day-six-the-false-premise-of-nosql/.

Schemaless DesignThe final innovation of the NoSQL camp is schemaless de-

sign. In database management systems of the NoSQL kind, datais stored in blobs or documents; t he database managementsystem does not police their structure. In mainstream databasemanagement systems on the other hand, doctrinal purity re-quires that the schema be designed beforedata is inserted. Letsdo a thought experiment.

Lets suppose that we dont have a schema and that the follow-ing facts are known:

Iggy Fernandez is an employee with EMPLOYEE_ID=1and SALARY=$1000.

Mogens Norgaard is a commissioned employee withEMPLOYEE_ID=2, SALARY=1000, and COMMISSION_PCT=25.

Morten Egan is a commissioned employee withEMPLOYEE_ID=3, SALARY=1000, and unknownCOMMISSION_PCT.

Could we ask the following questions and expect to re-ceive correct answers?

Question:What is the salary of Iggy Fernandez?

Expected answer: $1000.

Question:What is the commission percentage of Iggy

Fernandez? Expected answer: Invalid question.

Question:What is the commission percentage of MogensNorgaard?

Expected answer: 25%

Question:What is the commission percentage of MortenEgan?

Expected answer: Unknown.

If we humans can process the above data and correctly answerthe above questions, then surely we can program computers todo so.

The above data could be modeled with the following threerelations. It is certainly disruptive to suggest that this be done onthe fly by the database management system, but it is not outsidethe realm of possibility.

EMPLOYEES

EMPLOYEE_ID NOT NULL NUMBER(6)

EMPLOYEE_NAME VARCHAR2(128)

UNCOMMISSIONED_EMPLOYEES


SALARY NUMBER(8,2)

COMMISSIONED_EMPLOYEES


SALARY NUMBER(8,2)

COMMISSION_PCT NUMBER(2,2)

A NoSQL company called Hadapt has already stepped for-ward with such a feature:

While it is true that SQL requires a schema, it is entirely untruethat the user has to define this schema in advance before query

processing. There are many data sets out there, includingJSON, XML, and generic key-value data sets that are self-

describing each value is associated with some key thatdescribes what entity attribute this value is associatedwith[emphasis added]. If these data sets are stored in Hadoop,there is no reason why Hadoop cannot automatically generate avirtual schema against which SQL queries can be issued. And ifthis is true, users should not be forced to define a schema beforeusing a SQL-on-Hadoop solution they should be able to ef-

fortlessly issue SQL against a schema that was automaticallygenerated for them when data was loaded into Hadoop.(http://hadapt.com/blog/2013/10/28/all-sql-on-hadoop-solutions-are-missing-the-point-of-hadoop/)

This is not really new ground. Oracle Database provides theability to convert XML documents into relational tables (http://docs.oracle.com/cd/E11882_01/appdev.112/e23094/xdb01inthtm#ADXDB0120), though it ought to be possible to view XMLdata as tables while physically storing it in XML format in orderto benefit certain use cases. It should also be possible to redun-dantly store data in both XML and relational formats in order tobenefit other use cases.

In Extending the Database Relational Model to CaptureMore Meaning, Dr. Codd explains how a formatted database iscreated from a collection of facts:

Suppose we think of a database initially as a set of formulas infirst-order predicate logic. Further, each formula has no freevariables and is in as atomic a form as possible (e.g, A & B

would be replaced by the component formulas A, B). Now sup-pose that most of the formulas are simple assertions of the formPabz (where P is a predicate and a, b, , z are constants),and that the number of distinct predicates in the database is fewcompared with the number of simple assertions. Such a data-base is usually called formatted, because the major part of itlends itself to rather regular structuring. One obvious way is to

factor out the predicate common to a set of simple assertionsand then treat the set as an instance of an n-ary relation and the

predicate as the name of the relation.

In other words, a collection of facts can always be organizedinto a collection of relations.
http://iggyfernandez.wordpress.com/2013/12/30/the-twelve-days-of-nosql-day-six-the-false-premise-of-nosql/http://iggyfernandez.wordpress.com/2013/12/30/the-twelve-days-of-nosql-day-six-the-false-premise-of-nosql/http://iggyfernandez.wordpress.com/2013/12/30/the-twelve-days-of-nosql-day-six-the-false-premise-of-nosql/http://hadapt.com/blog/2013/10/28/all-sql-on-hadoop-solutions-are-missing-the-point-of-hadoop/http://hadapt.com/blog/2013/10/28/all-sql-on-hadoop-solutions-are-missing-the-point-of-hadoop/http://docs.oracle.com/cd/E11882_01/appdev.112/e23094/xdb01int.htmhttp://docs.oracle.com/cd/E11882_01/appdev.112/e23094/xdb01int.htmhttp://docs.oracle.com/cd/E11882_01/appdev.112/e23094/xdb01int.htmhttp://docs.oracle.com/cd/E11882_01/appdev.112/e23094/xdb01int.htmhttp://docs.oracle.com/cd/E11882_01/appdev.112/e23094/xdb01int.htmhttp://docs.oracle.com/cd/E11882_01/appdev.112/e23094/xdb01int.htmhttp://hadapt.com/blog/2013/10/28/all-sql-on-hadoop-solutions-are-missing-the-point-of-hadoop/http://hadapt.com/blog/2013/10/28/all-sql-on-hadoop-solutions-are-missing-the-point-of-hadoop/http://iggyfernandez.wordpress.com/2013/12/30/the-twelve-days-of-nosql-day-six-the-false-premise-of-nosql/http://iggyfernandez.wordpress.com/2013/12/30/the-twelve-days-of-nosql-day-six-the-false-premise-of-nosql/http://iggyfernandez.wordpress.com/2013/12/30/the-twelve-days-of-nosql-day-six-the-false-premise-of-nosql/


14/28

4February 2015

Oracle NoSQL Database

In May 2011, Oracle Corporation published a white paper ti-tled Debunking the NoSQL Hype, the final advice being Go forthe tried and true path. Dont be risking your data on NoSQL da-tabases.(https://www.google.com/#q=%2B%22Debunking+the+NoSQL+Hype%22) However, in September of the same year,Oracle Corporation released Oracle NoSQL Database. Oraclesuggested that the NoSQL approach was well-suited for certainuse-cases:

The Oracle NoSQL Database, with its No Single Point ofFailure architecture, is the right solution when data access issimple in nature and application demands exceed the volumeor latency capability of traditional data management solutions.For example, click-stream data from high volume web sites,high-throughput event processing and social networkingcommunications all represent application domains that

produce ex traordinar y volumes of simple keyed data .Monitoring online retail behavior, accessing customer profiles,pu ll in g up ap propria te cus tome r ad s an d st or in g an dforwarding real-time communication are examples of domainsrequiring the ultimate in low-latency access. Highly distributedapplications such as real-time sensor aggregation and scalableauthentication also represent domains well-suited to OracleNoSQL Database. (http://www.oracle.com/technetwork/products/nosqldb/learnmore/nosql-wp-1436762.pdf)

Oracle NoSQL Database has two features that distinguish itfrom other key-value stores: A key is the concatenation of amajor key path and a minor key path. All records with thesame major key path will be colocated on the same storagenode. In addition, Oracle NoSQL provides transactional supportfor modifying multiple records with the same major key path.

Challeges to NoSQL

There are already proofs that performance, scalability, and

reliability can be achieved without abandoning the relationalmodel. For example, ScaleBase provides sharding and replicationon top of MySQL storage nodes. Another good example to studyis VoltDB, which claims to be the worlds fastest OLTP database(though it has never published an audited TPC benchmark). Acounter-example to Amazon is eBay, which arguably has equalscale and equally high-performance, scalability, and reliabilityrequirements. eBay uses performance segmentation, sharding,replication, and eventual consistency but continues to use Oracle(and SQL) to manage local databases. I asked Randy Shoup, oneof the architects of the eBay e-commerce platform, why eBay didnot abandon Oracle Database, and he answered in one word:comfort. Here are links to some of his presentations and arti-

cles on the eBay architecture:

eBays Scaling Odyssey: Growing and Evolving a LargeeCommerce Site (http://francotravostino.name/papers/eBayScalingOdysseyShoupTravostino.pdf)

The eBay Architecture: Striking a balance between sitestability, feature velocity, performance, and cost (http://w w w . a d d s i m p l i c i t y . c o m / d o w n l o a d s /eBaySDForum2006-11-29.pdf)

Randy Shoup Discusses the eBay Architecture (http://www.infoq.com/interviews/shoup-ebay-architecture)

NoSQL Taxonomy

NoSQL databases can be classified into the following catego-ries:

Key-value stores:The archetype is Amazon Dynamo, ofwhich DynamoDB is the commercial successor. Key-valuestores basically allow applications to put and get val-ues, but each product has differentiators. For example,DynamoDB supports tables (namespaces) while OracleNoSQL Database offers major and minor key paths.

Document stores:While key-value stores treat values asuninterpreted strings, document stores al low values tobe managed using formats such as JSON (JavaScriptObject Notation) that are conceptually similar to XML.This allows key-value pairs to be indexed by any compo-nent of the value just as XML data can be indexed inmainstream database management systems.

Column-family stores: Column-family stores allow dataassociated with a single key to be spread over multiplestorage nodes. Each storage node only stores a subset ofthe data associated with the key, hence the name column-family. A key is therefore composed of a row key and a

column key. Graph databases:Graph databases are non-relational da-

tabases that use graph concepts such as nodes and edgesto solve certain classes of problems, such as determiningthe shortest route between two towns on a map. The con-cepts of functional segmentation, sharding, replication,eventual consistency, and schemaless design do not applyto graph databases, so I will not discuss graph databases.

NoSQL Buyers Guide

NoSQL products are numerous and rapidly evolving. There isa crying need for a continuously updated encyclopedia ofNoSQL products, but none exists. There is a crying need for an

independent benchmarking organization, but none exists. Mybest advice is to do a proof of concept (POC) as well as a PSR(Performance, Scalability, and Reliability) test before committingto using a NoSQL product. Back in 1985, Dr. Codd had words ofadvice for those who were debating between the new relationalproducts and the established pre-relational products of the time.

Any buyer confronted with the decision of which DBMS to ac-quire should weigh three factors heavily. The first factor is thebuyers performance requirements, often expressed in terms ofthe number of transactions that must be executed per second.The average complexity of each transaction is also an importantconsideration. Only if the performance requirements are ex-tremely severe should buyers rule out present relational DBMS

products on this basis. Even then buyers should design perfor-mance tests of their own, rather than rely on vendor-designedtests or vendor-declared strategies[emphasis added]. Thesecond factor is reduced costs for developing new databases andnew application programs . . . The third factor is protecting fu-ture investments in application programs by acquiring a DBMSwith a solid theoretical foundation . . . In every case, a relationalDBMS wins on factors two and three. In many cases, it can winon factor one alsoin spite of all the myths about perfor-mance.An Evaluation Scheme for Database ManagementSystems that are claimed to be Relational

The above advice is as solid today as it was in Dr. Codds day. (continued on page 26)
http://www.oracle.com/technetwork/products/nosqldb/learnmore/nosql-wp-1436762.pdfhttp://www.oracle.com/technetwork/products/nosqldb/learnmore/nosql-wp-1436762.pdfhttp://www.addsimplicity.com/downloads/eBaySDForum2006-11-29.pdfhttp://www.addsimplicity.com/downloads/eBaySDForum2006-11-29.pdfhttp://www.addsimplicity.com/downloads/eBaySDForum2006-11-29.pdfhttp://www.infoq.com/interviews/shoup-ebay-architecturehttp://www.infoq.com/interviews/shoup-ebay-architecturehttp://www.infoq.com/interviews/shoup-ebay-architecturehttp://www.infoq.com/interviews/shoup-ebay-architecturehttp://www.addsimplicity.com/downloads/eBaySDForum2006-11-29.pdfhttp://www.addsimplicity.com/downloads/eBaySDForum2006-11-29.pdfhttp://www.addsimplicity.com/downloads/eBaySDForum2006-11-29.pdfhttp://www.oracle.com/technetwork/products/nosqldb/learnmore/nosql-wp-1436762.pdfhttp://www.oracle.com/technetwork/products/nosqldb/learnmore/nosql-wp-1436762.pdf



PERFORMANCEC O RN ER

Adapt or Perishby Janis GriffinJanis Griffin

In Oracle Database 12c, the optimizer can make runtimeadjustments during the execution of the plan. It does thisby instrumenting the default plan with statistic collectors.At runtime, statistic collectors read and buffer a small

amount of rows in each step and compare the actual statisticswith the estimates. If they are different, the step is adapted to use

the better plan in order to avoid poor performance. Currently,the only steps that can change are join method operations, suchas changing a nested loop join to a hash join. Also, when theoptimizer is working on a parallel query, it can change the paral-lel distribution method.

To identify whether a SQL statement is using an adaptiveplan, we need to look at a new column in the V$SQL table calledIS_RESOLVED_ADAPTIVE_PLAN. If the value of this columnis Y, then the plan has been adapted and is the final plan. If the

value is N, then the plan is adaptable, but the final plan has notyet been chosen. This value will probably never be seen, as thefinal plan is usually chosen on the first run. If the column is null,the plan is not adaptable. Also, there is another new column in

V$SQL called IS_REOPTIMIZABLE. This column is used to tellthe optimizer to adapt the statistics. The values for this columncan be R, for reporting-only mode; Y, to signal a hard parse soit can gather better statistics; and N, when no new statistics areneeded.

The parameters that control the behavior of the adaptive queryoptimizer are optimizer_adaptive_features and optimizer_adap-tive_reporting_only. By default, the optimizer_adaptive_featuresoption is set to TRUE and the optimizer_adaptive_reporting_onlyoption is set to FALSE. The reporting only parameter can be setto TRUE to first test and see which queries are going to be adaptedbefore the optimizer actually uses them. This might be helpfulwhen first upgrading from an earlier version of Oracle to 12c, inorder to avoid any unexpected behavior. You can also set the opti-mizer_feature_enable parameter to something less than 12c toturn this feature off.

To view adaptive plans, there are two new format parametervalues for the DBMS_XPLAN.DISPLAY_CURSOR function.The +adaptive format parameter will show all active and inac-tive steps of a plan. The +report format parameter can be usedwhen the optimizer_adaptive_reporting_only parameter is setto TRUE in order to see how the plan would have been adaptedif the adaptive optimizer were turned on. Figure 1 shows an ex-ample of setting the reporting_only parameter and then usingthe report format parameter to view what it would be.

Figure 1

Figure 2 shows an example of using the adaptive formatparameter. Notice the inactive steps are the hash join, statisticscollector, and full table access of product. The plan adapted fromusing a hash join to using nested loops.

Figure 2

Figure 3 shows an example of an adapted and reoptimizedplan for the SQL_ID beginning with 8qpak. Notice that all ofthe children have adaptive plansidentified by the Y in theIS_RESOLVED_ADAPTIVE column. Also, notice that the IS_REOPTIMIZABLE column has a value of R for child_number0 because the reporting_only parameter was set to TRUE on thefirst execution. In other words, the optimizer didnt actually usethe adaptive plan. On second execution, the reporting_only pa-rameter was set to FALSE, so the optimizer not only adapted theplan but also sampled new statistics and found the default statis-tics to be incorrect. Child_number 1 was marked for a reparse on


16/28

6February 2015

the next execution, and the value of the IS_REOPTIMIZABLEcolumn was set to Y. Finally, child_number 2 has been success-fully reoptimized, and no more changes are currently required(IS_REOPTIMIZABLE is set to N).

Figure 3

The other way for a plan to adapt is when the SQL statement

is resolved by using a parallel query. The optimizer tries to dis-tribute the number of rows evenly across the number of pro-cesses that it has to work with. Historically, the optimizer wouldlook at the number of rows and Degree of Parallelism (DOP),and then decide how to distribute the data. This often causedproblems when there were only a few parallel processes tryingto distribute a large number of rows. Also, any data skew couldcause unequal distribution and performance issues. In 12c, anew Hybrid Hash distribution technique tries to fix those issues.Basically, the optimizer chooses the distribution method duringexecution time by inserting statistic collectors in front of theparallel processes on the producer side of the operation. Theoptimizer will choose the Hybrid Hash distribution method if

rows are greater than the threshold, or it will choose the Broad-cast distribution method if rows are less than the threshold. Thethreshold is defined as 2 * DOP.

In Figure 4, there is an example of a Hybrid Hash distribution.Notice that I gave it a hint to use parallel 20, and with that mythreshold is 2 times that, or 40. Notice the number of rows ismuch greater than that, at 75329.

Figure 4

We briefly discussed automatic reoptimization when we werelooking at adaptive plans. However, it differs from adaptive plansin that reoptimization changes the plan after the first executionand not during it. It uses information gathered during the exe-cution and compares that with the default plan. If they differsignificantly, the plan will be reoptimized on the next run. Some-times the optimizer will store the new information gathered as aSQL PLAN DIRECTIVE, so other similar queries can benefitfrom the information. The optimizer can reoptimize a queryseveral times by using statistics or performance feedback tochange the plan.

Statistics feedback is used when estimated cardinalities arewrong. This can be due to missing or inaccurate statistics, orcomplex predicates. The optimizer will turn on statistics feed-back if a table has no statistics or if there are multiple (and/or)filter predicates on a table, or if the predicates contain complexoperations for which it cant compute estimates.

Figure 4 shows an example of how statistics feedback works.For this example, the format parameter, ALLSTATS LAST, aswell as the hint of gather_plan_statistics are used to show theestimated rows versus the actual rows returned in each step.

Figure 5

In Figure 5, notice how far off the default values were. Theoptimizer realized this discrepancy, so it used statistics feedbackto further optimize it.

Does the adaptive query optimizer work as advertised? Here isa SolarWinds Database Performance Analyzer (formerly ConfioIgnite) for Oracle screenshot of one query that ran for severaldays with the same work load. Basically, I had eight users runningthe same order entry load over several days. This one query,which looked up product inventory, was running every hour ofthe day. You can see that it wasnt performing very well prior to 9a.m. In fact, the throughput or number of executions was about10,000 per hour. During that time I had been running the adap-tive optimizer in report only mode, so it was only using thedefault plan. After I turned the mode off, you can see that theplan changed by adapting, and then it was reoptimized severaltimes. Notice the time spent and the increase in throughput.

Figure 6

In summary, it looks like the adaptive query optimizer canimprove performance over time. s

Janis Griffin is an Oracle ACE and a database performance evangelistat SolarWinds, the maker of Database Performance Analyzer (for-merly Confio Ignite) for Oracle. Information on Database Performance

Analyzer can be found at www.solarwinds.com/dpa-oracle .Copyright 2015, Janis Griffin
http://www.solarwinds.com/dpa-oraclehttp://www.solarwinds.com/dpa-oracle



S Q L C O R N E R

Two men sit together in a train. One of them takes outa banana, peels it, salts it, and throws it out of the win-dow. Then he takes another banana and repeats theprocedure.

After the third banana, the second man cant contain his curi-osity. He asks, Why are you throwing all of these bananas out ofthe window?

Well, the other man replies, do youlike salted bananas?

Throw-Away Rows in Performance Tuning

Lets take a moment to examine why this joke is funny. Itsfunny because its ridiculous: if you want to throw a banana away,why would you go through the process of peeling and salting itfirst? But now we move on to more serious stuff: Is it possiblethat we do something similar in real life, for example, in databaseperformance tuning?

Throw-away rows are rows that were retrieved by the databasebut not used in any way to compute the result of a query.

Minimizing throw-away rows is a common theme in perfor-mance tuning ([1], paragraph 12.1.2).

The methods are intuitively plausible, but a row that is re-trieved but not used certainly represents lost effort.

One common application of throw-away is index optimiza-tion, as shown in Figure 1.

Figure 1: Sub-optimal index access indicated by throw-away

Here we see the essential part of a SQL monitor output. Wesee an index access retrieving 175k rows. When the table is ac-cessed, only one row remains.

There must be an additional filter on the table that reduces therows to that level. Certainly that filter is missing in the index. Allwe need to do is look up this filter in the execution plan and addit to the index. (Note: Of course, we must also make sure that theadditional field in the index will be commonly used and not usedby just one query.)

This way we would have the throw-away in the index accessas opposed to having it in the table access.

So why would it be better to have the throw-away in the

index? Lets have a look on the next figure.Here we see the exact same execution plan as the one above,

but this time its focused on physical I/O.As we can see, we have a lot more I/O requests on the table

access than on the index access. To understand this, we have tothink about how the buffer cache works. The index has 3,075 leafblocks, and the table has 78,954 blocks.

Each segment is accessed 175,000 times (see Figure 1). For theindex, that means 57 hits per leaf block. For the table, that wouldmean roughly two hits per block.

In addition, the index is sorted based on the search criteria;therefore, we can expect that not the whole index is retrieved butonly the part that is relevant for the search.

Thus, under normal circumstances, most of the 175,000 buf-fer gets of the index access will be found in the buffer cache. Incomparison, a higher proportion of the buffer gets of the tableaccess will translate into physical reads.

Figure 2: I/O wait time compared: index access vs. table access

Adding an additional filter column in the index will thereforereduce the overall number of physical reads.

A Long-Running Query

Some years ago I received a query to tune that kept me think-ing for years. Of course I did not work on it all the time, but oc-casionally it popped up and I started contemplating again how it

could be solved.At first glance, it seemed very simple. In essence it was findall people with the name Miller in the city of Bern.

How to Avoid a

Salted Bananaby Lothar Flatz

Lothar Flatz

Figure 3: Set diagram for the Miller-in-Bern search


18/28

8February 2015

That seems to be a common, simple query. On second glance,it is not that simple. Miller (actually Mller in German) is a com-mon name in Switzerland (as well as in Germany). Bern, theSwiss capital, has about 350,000 inhabitants.

No matter where you start searching, your first result tends tobe rather big, and the overlapping final result is comparativelysmall.

Actually, the best way would be to combine the two searchcriteria. That is hard to do, however, since the two search criteria

are against different tables: one on table person and the other ontable address.

Lets explore the different options to combine two search cri-teria in one index using features of the Oracle database.

Our underlying assumption is that we work in an OLTP envi-ronment, like a call center application. Actually, that is the envi-ronment that typically generates such queries.

Most of our ideas would also work in a data warehouse, buthere the alternative solutions (see below) could be consideredmore seriously.

Materialized View

One way to deal with this situation is to join table address and

table person by building a materialized view, and then building acomposite index on top of it.

In order for this to work there are a few prerequisites:Creating a materialized view must be possible in the sense

that we have the rights and the space, etc.In an OLTP environment we must create a fast refresh on

commit materialized view, which can be very ambitious to man-age. We are violating normal forms, which will lead into updateanomalies that can make the refresh very resource consuming.

This solution can work out in some carefully selected sce-narios, but it is certainly not a generic solution.

Bitmap Join Index

In principle this solution has about the same drawbacks as thematerialized view. It also holds a more serious drawback in thelock that will be applied due to the bitmap index.

In an OLTP environment this is not an option.

Text Index

It is possible to have a multi-column, multi-table text index.However, the refresh per commit would be challenging in anOTLP environment.

In addition, we would have to change the queries to adopt textindex search operators.

Searching for a Solution

I came across this type of query occasionally in my profes-sional career. After all, it is a typical query used in call centers.Every time somebody told me, Our call center software is tunedto the max and will respond instantly, I told them to search forMller in Bern, or for Meier in Zurich.

The result was always a jaw-droppingly long wait. After con-templating awhile over the issue, I came up with the idea that thequery was indeed dominated by the problem of throw-awayrows, or throw-away dominated, as Ill refer to them going for-ward.

Lets look at Figure 3 again. Only the orange part is the result.The rest is throw-away.

Actually, the throw-away is far bigger than the result. So thatmeans the query is throw-away dominated.

The question I was asking myself was, Why would you gothrough the extra effort of reading a row from the table whenyou know with high probability that you are going to throw itaway anyway?

The idea of the salted banana arrived in my head as a kind ofmetaphor for the table access that might now be required. Again,if you want to throw away your banana, why would you go

through the process of peeling and salting it first?

Our Example

To redo and test the issues with the Mller-in-Bern query, Ihave created an example that roughly resembles the real data set.

I created a schema containing two tables.

Create table address (city_name varchar2(30) not null,

person_id number(7) not null,

data varchar2(200))

/

create Table person (person_id number(7) not null,

last_name varchar2(15) not null,

data varchar2(200))

As you can see the schema is simplified. It does not mean tobe realistic, but be enough to aid our discussion.

In addition these indexes exist to allow various type of joinsequences and to give the optimizer a choice.

create Index city_idx1 on address (city_name, person_id) compress;

create Index city_idx2 on address (person_id, city_name) compress;

create Index person_idx1 on person(last_name, person_id) compress;

create Index person_idx2 on person(person_id, last_name) compress;

I also created statistics

Documents

NoCOUG Journal Latest (1)