Embed Size (px)
Citation preview
1
Hello DBAs!We Need an Index! This Index!
UKOUG
4th December 2019John Lancashire
Who Are We?
• Direct Wines is a Family Owned Company Founded in 1969
• The World’s No 1 Home Delivery Wine Merchant
• Operating in UK, USA, Australia and New Zealand
• Deliver Over 4.5 Million Cases of Wine a Year
UK
USA
AU/NZ
The Beginning
The First Arch
Bricks
Logistics
Clicks
Celebrating 50 Years
Our Technologies
Plus:
• Telephony
• Diallers
• Email Servers
• ESB
Our Technologies
• Oracle Database
– Editions: Standard and Enterprise
– Versions: 10.2.0.4, 11.2.0.2, 11.2.0.4, 12.2.0.2
– Key Options: OLAP, Partitioning
– Oracle18c ASM
• Microsoft SQL Server
– Editions: Standard and Enterprise
– Versions: 2008R2/2014/2016
• Linux/UNIX
– Red Hat Enterprise Linux (RHEL 5/6/7)
– HP-UX 10.23
• Microsoft Windows Server
– 2008, 2014
About Me
• Potted Career History– Bailed out of accountancy and teaching before settling on a career in IT
– First IT job: COBOL Programmer in Civil Service1988
– Worked with Oracle Database since 1995 (Oracle 7.1.4)
– Direct Wines since 1999
– Dabbling with SQL Server since 2010
– Current Role: Database Architect and DBA Manager
• Skill Set– Database
• Oracle 7-12c (OCP 7,8,8i)
• SQL Server 2005/2008R2/2014
– Operating Systems• HP-UX
• Red Hat Linux RHEL 5/6/7
• Windows I only do databases on Windows.
Agenda
• What Got Me Standing Here?
– Four Real World Examples
– One General Observation
• Indexes v Constraints
• A Quick Overview of Indexing Options
– B-Tree
– Bitmap
• Index States
– Visible/Invisible
– Unusable
• Back to the Examples
The Inspiration That Got Me Standing
Here
The Inspiration That Got Me …The Watercooler Conversation
“Aren’t we using bitmap indexes on our website?”
The Inspiration That Got Me …A Request for Many Normal Indexes in DSS System
• Hey. Can you create these indexes for me?
• To be helpful, the developer even sent me the script
so I wouldn’t even have to think about it.
• 15 years of business analysis and report writing.
• 100+ million rows in the table
The Inspiration That Got Me …A Request for Many Normal Indexes in DSS System
• A Typical Query Against the Table:
The Inspiration That Got Me …Several Instances of Redundant Indexes
• Duplicate Leading Columns
• Not as crazy as it looks when you know the sequence of events
The Inspiration That Got Me …Some Things I Just Can’t Explain
• An Honest Mistake or Lack of Knowledge?
• Can’t have had any positive effect during testing
The Inspiration That Got Me …Not Quite Getting NULLs
• So Many Queries Where a Flag Column is NULL
• So many occurrences of NULL used as a 0 or ‘N’
equivalent
Off We Go
Indexes v Constraints
Indexes v Constraints
Constraints
• Logical Design
• Protect Data
• Define Relationships
• Types
– Primary Keys
– Unique Constraints
– Foreign Keys
– Check Constraints
Indexes
• Physical Design
• Primarily for Performance
• However
– Primary Keys/
Unique Constraints
• Both enforced by indexes
• Indexes will be created if
none exist
Indexes and Small Tables
• Often stated that small tables do not need indexes. BUT
• A truly relational table should have a primary key ( => index)
• Primary key prevents accidental duplication in a single row table
• Replication is easier if a primary key exists and may be essential
A Quick Overview ofIndexing Options
Main Index Types
B*-Tree Indexes• The ‘Conventional’ Index
– The default of the CREATE INDEX statement
– Implementation Similar to Binary Search Tree
– Goal is Minimise Amount of Time Oracle Spends Searching
– Can be UNIQUE or Non-Unique
– Can be on One or Many Columns• Either to Match multi-column predicates
• OR to implement ‘covering index’ (poor man’s IOT)
– NULL Value Excluded From Single Column Indexes
• When to Use
– Selecting a Small Portion of a Table (say 10% or Less)
– When the Index Can Satisfy the Query Instead of Table.
– OLTP systems with multi-user DML
• All Database Editions
Bitmap Indexes• Ad-hoc Queries in Reporting/DSS/Data Warehouses
– CREATE BITMAP INDEX …
– Still stored in B-Tree format
– Different leaf block structure• One index entry points to several rows
• Column value, starting and ending ROWIDs
• Plus a series of 1/0 bits identifying if the row contains the values
– Stores NULLs
– Excellent in aggregating operations
– Goal is Minimise Amount of Time Oracle Spends Searching
– Can be on One or Many Columns (Why?)
– Small (compared to B-Tree indexes)
– Fast to build (compared to B-Tree indexes)
• When to Use
– Great for multiple low cardinality columns on the same fact or dimension tables.
– Especially effective in Star Schema designs
– If Indexing NULLs is required
• Enterprise Edition Only
Bitmap Index - Caveat
• Just Don’t … (Trust Me)
– Avoid use in an OLTP system
– Potential for multi-user DML contention is immense
– Simplistic view:
• The values get locked
– A single row update:
» UPDATE row set saleable_flag = ‘N’ where id = 7;
– Has the potential to block all other DML (IUD) on the table
– (The from/to values being changed covered by the
particularly bitmap are locked)
Index Sub-typesFocusing on B-Tree (some apply to Bitmap Indexes too)
Type Syntax Usage
Basic create [unique] index ix1
on t1(col1);
High cardinality columns. Use unless have a
concrete reason to use an alternative.
Composite/
Concatenated
create [unique] index ix1
on t1(col1,col2,...,colN);
As above, but use where may be beneficial to index
columns satisfying a query avoiding access to the
table OR where more than one column defines
uniqueness in PKs and UCs
Descending create [unique] index ix1
on t1(col1,col2 desc);
Useful for reports or specific result sets requiring a
specific sort order.
Different columns will be returned in ascending and
descending order. Avoids having to perform a sort.
Reverse Key create [unique] index ix1
on t1(col1) reverse;
Useful in high performance parallel logging.
123456 stored as 654321
123457 stored as 754321
Avoids contention on the insert as sequential keys
should end up in different leaf blocks.
Can only use for lookups; no range-scans
Index Sub-typesFocusing on B-Tree (some apply to Bitmap Indexes too)
Type Syntax Usage
Compressed create [unique] index ix1 on
t1(col1,col2,col3) compress 2;
Consider for concatenated indexes
with often repeated values in the
leading columns.
(e.g. in ERP systems where several
companies or regions keep their
data separated by a leading region
code – PK (Region, OrderNo) etc)
Index Organised
Table
create table t1
(col1 number
,col2 varchar2 (40),
constraint pk_t1
primary key (col1)
) organization index;
Useful for tall narrow tables.
If all the column make up the PK,
why have a heap and an index.
Intersection tables also an ideal
case.
Secondary indexes include the PK
columns
Index Sub-typesFunctions or Derived Values
Type Syntax Usage
Function Based
Index
create [unique] index ix1 on
t1(upper(email));
When you need to apply a function
to a column to get value used in the
filter of predicate.
(Creates a hidden column on the
table which is then indexed)
Indexed Virtual
Column
create table t1
(col1 number
,email varchar2 (50),
,upper_email generated always
as (upper(email)) virtual
);
create [unique] index ix1 on
t1(upper_email);
This is pretty much a choice as to
which works best for you.
You create a derived column on the
table which you then index.
Very similar. Easier to deal with
more complex derivations than an
FBI.
Expression could be a CASE
statement, or a complex calculation.
Index Sub-typesNote on Implementation of Descending Indexes
INDEX_NAME UNIQUENESS INDEX_TYPE FUNCIDX_
STATUS
COLUMS COLUMN_
EXPRESSION
IX1_DESC NONUNIQUE FUNCTION-BASED
NORMAL
ENABLED SYS_NC00005$ "CUST_NO"
IX2_DESC NONUNIQUE FUNCTION-BASED
NORMAL
ENABLED CUST_NO,
SYS_NC00006$
"FIRST_NAME"
IX3_BAD_SPEC NONUNIQUE FUNCTION-BASED
NORMAL
ENABLED SYS_NC00007$ 'LAST_NAME'
IX14_REVERSE NONUNIQUE NORMAL/REV FIRST_NAME
PK_CUSTOMER UNIQUE NORMAL CUST_NO
Indexing ConstraintsA bit more detail (PK shown, UC similar)
• By default an index matching the constraint columns is created.
• Avoid System Generated Names
• No clue as to what the PK or index relates to.
• Not my preferred method to add PK or UC as in this case dropping the
constraint results in index being dropped too.
Indexing ConstraintsA bit more detail
• Named constraint, named index. Some clue as to what it relates to.
• Because an existing index is used to enforce the constraint, the index will
not get dropped when the constraint is dropped.
Indexing ConstraintsIndex Can Have More Columns Than Constraint, But…
SQL> CREATE UNIQUE INDEX UI_T1_REF_ID_CLU ON T1(REF_ID, REF_CLUSTERED);
INDEX UI_T1_REF_ID_CLU created.
SQL> ALTER TABLE T1 ADD CONSTRAINT PK_T1 PRIMARY KEY(REF_ID) USING INDEX UI_T1_REF_ID_CLU;
Error starting at line : 69 in command -
ALTER TABLE T1 ADD CONSTRAINT PK_T1 PRIMARY KEY(REF_ID) USING INDEX UI_T1_REF_ID_CLU
Error report -
ORA-14196: Specified index cannot be used to enforce the constraint.
14196. 00000 - "Specified index cannot be used to enforce the constraint."
*Cause: The index specified to enforce the constraint is unsuitable
for the purpose.
*Action: Specify a suitable index or allow one to be built automatically.
SQL> DROP INDEX UI_T1_REF_ID_CLU;
Index UI_T1_REF_ID_CLU dropped.
SQL> CREATE INDEX IX_T1_REF_ID_CLU ON T1(REF_ID,REF_CLUSTERED);
Index IX_T1_REF_ID_CLU created.
SQL> ALTER TABLE T1 ADD CONSTRAINT PK_T1 PRIMARY KEY(REF_ID) USING INDEX IX_T1_REF_ID_CLU;
Table T1 altered.
Indexing ConstraintsForeign Key Columns
• Parent Table
– Foreign Keys must reference either a PK or a UC.
– Therefore parent table columns are indexed
• Child Table
– Indexes not automatically created
– Highly Recommended
• Especially important for ON DELETE CASCADE
– Performance benefits noticeable even in batch/single user
systems due to avoiding table scans on child table
• Locking and Contention in OLTP systems
– When indexes not present, updates to child table place row
exclusive locks on parent table.
– Latest Info: REFERENTIAL INTEGRITY AND LOCKING [ID 33453.1]
Indexing ConstraintsCheck Constraints
• Not Relevant
– Of course you can add indexes on columns with check
constraints if they will improve the performance of your
application.
– But and index has no benefit or necessity in the function
or enforcement of the constraint.
Practical Stuff
Back to the Examples
Case 1OLTP Tuning Example
• Solarwinds DPA – Top SQL Statements
– Something new appeared after a recent release
– Occurs during call-centre hours
– Let’s investigate
(the names have been changed to protect the guilty)
OLTP Tuning ExampleModerate Sized Tables/Very Frequent Executions
OLTP Tuning ExampleTwo Indexes, Same Leading Column
T3 has two indexes:
• One associated with the two column PK
• One on the PK’s leading column
• Seems a bit pointless but
IX_T3_REF_ID came first.
• Discovered the missing PK later
during a replication project
• IX_T3_REF_ID became pointless
• No index leading with filter column on the
new problem query
OLTP Tuning ExampleDrop Redundant Index; Index Filter Column
OLTP Tuning Example271 Fold Reduction in Blocks Accessed
OLTP Tuning ExampleWhat if we Make it a Covering Index?
• Leading column on filter
• Second column on join column
OLTP Tuning ExampleNo Table Access and Shaves off Another 15%
• Query satisfied by both indexes
• Table access avoided
OLTP Tuning ExampleWow! It’s a Two Column Table and I’m Indexing Both!
• Surely a perfect candidate for an index organised table!
OLTP Tuning ExampleTry With IOT Version (No Secondary Indexes)
OLTP Tuning ExampleAdd Secondary Index on Filter Column
OLTP Tuning ExampleBest Yet! Plus we’ve lost one object (no heap).
• Secondary index on IOT
includes PK columns.
• No need to access table T3.
OLTP Tuning ExampleIn Reality What Does It Mean?
Measure (For 500 Loops) Original Index Mods IOT with
Secondary
Index
Elapsed Time (uncached) 47.99s 2.63s 2.80s
Elapsed Time (repeated) 45.33s 0.02s 0.02s
Physical Reads 18,887 184 197
Logical Reads 9,435,516 3,016 3,019
With a bit of help from Tom Kytes’ RunStats
OLTP Tuning ExampleResult: No Longer Registering in Solarwinds
Case 2 - DSS System ExampleStarting Point: No Indexes, Full Scan
DSS System ExampleAs Requested – Multiple Single Column Normal Indexes
Slow to Build
DSS System ExampleI Was Surprised by the Plan in the Demo – Have Seen Individual Range Scans in Practice
Repeat execution (cached)
DSS System ExampleBITMAP CONVERION (FROM ROWIDS)
• Enterprise Edition only Optimiser Operation
– Creates a bitmap dynamically when the optimiser calculates it to be faster to use logical operations on bitmaps than any other data access method.
– In most cases this kind of conversion is only beneficial if the WHERE clause contains OR operator, which is typically very slow with other access methods.
• Depending on Circumstances, Have seen just one of the single column indexes being used in this situation
– Standard Edition
– Data Distribution/Clustering Factor
– _b_tree_bitmap_plans = false
DSS System ExampleWhat I Thought Would be Requested – Composite Index
Slow to Build
DSS System ExampleEffective But Inflexible – An Index for Every Scenario?
DSS System ExampleBut (as it’s Enterprise Edition) What We Could Do
Quicker to Build
DSS System ExampleElimination Done By Combining Bitmaps
DSS System ExampleWe Can Easily Add Additional Bitmap Indexes …
DSS System Example… Which Adds an Additional Layer of Elimination
Case 3 - Rogue FBIsCaused by Cutting and Pasting and Incorrect Editing
Result
• Hidden Column containing same value
• Corresponding index
• Pointless overhead
• Not picked up in testing
Case 4 – Indexing NULLs
• NULLs Aren’t Indexed, Are They?
Indexing NULLsBasic Single (Flag) Column Index
Indexing NULLsBasic Single Column Index - Ignored
Indexing NULLsFunction Based Index – Effective but Code Change
Indexing NULLsFunction Based Index – Effective but Code Change
Indexing NULLsVirtual Columns – Good, but Again, Code Change
Buffer Cache Flushed
Repeated Execution
Indexing NULLsBitmap Index – Good, but Contention in Multiuser Sys
Buffer Cache Flushed
Repeated Execution
Indexing NULLsBitmap Index – Good, but Contention in Multiuser Sys
Indexing NULLsMulti Column Index – Add Trailing NOT NULL Column
No Code Changes
Buffer Cache Flushed
Repeated Execution
Indexing NULLsMulti Column Index – Add Trailing NOT NULL Column
Unindexed Foreign Key ColumnsPoor Performance Leading to Contention and Blocking
• Referential Integrity
– Child Table REF_ID
column Foreign Key
References PK of
Parent
– 1 to many
relationship
Unindexed Foreign Key ColumnsPoor Delete Performance Leading to Contention
• No Index on Child Table– Not automatically created when FKs
are added
Full Scan on Child Table
Unindexed Foreign Key ColumnsFixed by Adding Index to Child Table
My Point!
• Work Together, Learn Together– Developers
• Know what they want to get
• It’s in their domain to know the application design
• It’s in their domain to be experts in the language they use
– DBAs• Know what’s in the armoury of the database
• Familiar with the way indexes work
• Familiar with the options available in the edition being used
• Dos– Ask for a solution
– Ask about options that might be available
• Don’ts– Be afraid of composite indexes
– Tell the DBA what indexes to add and how
• You Might Get What You Ask For, But Not Get What You Need
References
Oracle Concepts Manual: Ch 3 Index and Index-Organized Tables
Expert Indexing in Oracle Database 11g
Apress 2012, Darl Kuhn, Sam R Alapati, Bill Padfield
Foreign Keys, Indexes and Locking (Doc ID 1945574.1)
Optimizing the Indexes for an Oracle Schema: Dave Ensor (Paper circa 2000)
Made It - Cheers
