Copyright © 2012, Oracle and/or its affiliates. All rights ... · Tom Kyte Been with Oracle ... Indexing is an OLTP technique for operations on a small number of rows ... No change

Copyright © 2012, Oracle and/or its affiliates. All rights reserved. 1


A Day of Real World Performance

The Real-World Performance Group

Andrew Holdsworth, Tom Kyte, Graham Wood


Program Agenda

Introductions

Think Different

Coffee

Analyzing SQL

Lunch

OLTP Basics

Coffee

Set Based Programming Concepts

Running out of Memory


Introductions


Introductions

25 Years at Oracle

Vice President Real World Performance

– Good performance is rarely an accident

– Most people get the systems they deserve

– Good enough rarely is, aspire for excellence not good enough.

Andrew Holdsworth


Tom Kyte

Been with Oracle since 1993

User of Oracle since 1987

The “Tom” behind AskTom in Oracle

Magazine

www.oracle.com/oramag

Expert Oracle Database

Architecture

Effective Oracle by Design

Expert One on One Oracle

Beginning Oracle


Graham Wood

Architect

Server Technologies


Think Different ( About Performance !


Already have a service that

processes a single message

Process more messages

concurrently by using multiple

threads or change the service

to process a set of messages.

Which is better?

Threads vs Arrays

Scenario


Not much of a contest

Array takes 1minute

Threads takes 22 minutes

With higher load on server

But still resource available

Larger pool size for threads?

Threads vs Arrays

Analysis


4x threads improved

performance

Array takes 1minute

Threads takes 10 minutes

Much higher load on server

This is ONE dedicated process

vs 64 parallel threads

Threads vs Arrays

Analysis


What happens when we add an

index to both systems?

Threads vs Arrays

Scenario


Both systems slowed by index

Multiple threads suffers from

contention, buffer busy wait, TX

index contention

Threads vs Arrays

Analysis


Index Contention Performance Data

Transaction rate: 4,500 TPS

Response time: 13 ms

Dominant wait events:

buffer busy waits and

enq:TX–index contention


Index Contention Reverse Key Index

Transaction rate: 10,000

TPS Response time: 5 ms

Contention eliminated while the

history table remains small.

CPU increases with workload.


Index Contention Reverse Key Index



Contention shifts to I/O

subsystem when the history

table grows larger


Index Contention Hash Partition Index


Response time: 5 ms

Contention eliminated by hash

partitioned history table index


Index Contention Add RAC node



RAC appears not to scale well

with contention in cluster waits


Index Contention Composite Key Index


Response time: 5 ms

RAC configuration scales well

with no contention or gc waits


Devices ship files.

Files read and processed by

multiple application servers

Each application server uses

multiple threads that connect to

database through a connection

pool which is distributed by a

scan listener over two

instances.

Is Load Balancing Slowing Your System? Scenario


It’s too slow

It’s a problem with the database

– Look at all those waits

Need to be able to process an

order of magnitude more data

Obviously need to move to

Hadoop

Is Load Balancing Slowing Your System? Problem


Only small amount of data

being processed.

Both instances essentially idle

with most processes waiting in

RAC and concurrency waits.

Is Load Balancing Slowing Your System? Analysis


Remove all of those RAC waits

by running against a single

database instance.

Is Load Balancing Slowing Your System?

Solution


Throughput up by factor of 10x

RAC waits gone

CPU time actually visible

High concurrency waits

– Buffer busy

– Tx index contention


Analysis


Reduce contention waits by

processing a file entirely within

a single application server


Solution


More throughput

Log file sync predominant event

CPU usage close to core count


Analysis


Reintroduce RAC to add more

CPU resource

Implement separate service for

each instance

Connect application server to

one instance


Solution


Analyzing SQL


Why is My SQL Slow ?


Problem Query

Table has 1.2B rows

and is 64 GB.


Problem Query

Query 1 consists of two

subqueries. The first

subquery finds all of the

Ferraris.


Problem Query

The second subquery finds all

of the Ferrari 458s.


Problem Query

Outer query joins the results

of the subqueries.

Outer query performs

aggregations.


Problem Query

Query 2 is the same but has

different predicate values.


Default Statistics


1. Default Statistics

Baseline Performance for Query 1

Query 1 exceeds target

Initial Optimization Steps



Query 1 takes 49

seconds with default

statistics




Add more predicate values


2. Development Findings

Query runs faster just by changing the list of values

in the select list

Plan changed from a broadcast to a hash distribution

due to the higher but inaccurate cardinality estimate

Get correct plan with wrong cardinality estimate—

can lead to inconsistent plans and performance




Increase the list of

predicate values

Now query takes 3

seconds




Change Degree of Parallelism


3. Development Findings

Just changing DoP from 32 to 128 improves

performance and meets the target; 4X more

resources yields a 25X performance improvement

Plan has changed from a broadcast distribution to a

hash distribution due to DoP change

DoP is a resource management technique, not a

query tuning tool




Change DoP

from 32 to 128 Now query takes 2

seconds




Indexes


4. Indexes

Indexes on columns:

owner_id

country

make

model

country, make, model



4. Indexes

Not understanding the big/little data challenge

Indexes are not efficient for operations on a large

numbers of rows

Full table scan is faster with predictable performance



To Index or Not

Indexing is an OLTP technique for operations on a small number of

rows

A table scan may consume more resources but it will be predictable

no matter how many rows are returned

Indexes impact DML operations

If I/O bandwidth went from 70MB/sec to 70GB/sec would you

change your optimization/execution strategy?


To Index or Not

Index driven query retrieving 1,000,000 rows

– Assume the index is cached and the data is not.

1,000,000 random IOPS @ 5ms per I/O

This would require 5000 Seconds ( or over 1 hour ) to Execute

– How much data could you scan in 5000 Seconds with a fully sized I/O

system able to scan 25 GB/Sec ?

Over 100 TB !



Add Indexes and

query takes longer—

160 seconds!



Index lookups on

millions of rows is

slow


Histograms


5. Histograms

Re-gathered stats to automatically create histograms

Frequency histograms on country, make and

model columns

No change in plan—query still exceeds target




Rerun stats to get

histograms—no change in

plan or run time



Lots of wait time on temp

IO


Flash Temp


6. Flash Temp

Most of the wait time was spent performing IO on

temp, so move temp to flash disks

Improved performance but still does not meet target

Not a good use of flash

Incorrect use of tools/products






Move temp to flash and

now IO accounts for a

smaller percentage of

database time


Manual Memory Parameters


7. Manual Memory Parameters

Set sort_area_size and hash_area_size to 2G

Eliminated temp usage but still did not meet target

Memory is allocated per parallel server process,

which can quickly exceed resources

Moving to a solution before understanding the

problem





8. Cardinality Hint

SQL Monitor showed poor cardinality estimates

Cardinality hint gives optimizer the correct number of

rows for the table scan

Plan changed from a broadcast to hash distribution

Query time now meets target

Now temp is not an issue

Optimization Steps


Optimization Steps


Optimization Steps

Use cardinality hint to specify

correct number of rows

Plan switches from a broadcast

to a hash distribution


Disable Broadcast Distribution


9. Disable Broadcast Distribution

Googling reveals a hidden parameter to disable

broadcast distribution

Plan and run times are similar to cardinality hint,

meeting target

Moving to a solution before understanding the

problem

Optimization Steps


Optimization Steps


Histogram on Column Group


13.Histogram on Extended Stats

Re-gathered stats after running the query with the

column groups

Frequency Histogram on the column group

Accurate cardinality estimates

Optimizer now uses a hash distribution

Optimization Steps


Optimization Steps


Optimization Steps

With a histogram on

the column group we

now have a good

cardinality estimate

Now we get a hash distribution

and meet the target


Second Query with Histogram Column Group


14.Query 2 with Extended Stats

Accurate cardinality estimates

Optimizer uses a broadcast distribution on second

query

Optimization Steps


16.Auto Column Groups

dbms_stats.report_col_usage shows column groups

identified during Seed Column Usage

dbms_stats.create_extended_stats creates column

groups identified

Automatically identifies usage of Country, Make and

Model columns together and creates column group

Optimization Steps


16.Auto Column Groups

Regather stats

Automatically creates Histogram on the column

group

Query meets target

Optimization Steps


Optimization Steps


Optimization Steps

Now with the column group identified and created

we have a good cardinality estimate

And we get a hash

distribution


What Did We Learn ?


What Did We Learn

Real World Mistake Example

Incorrect use of tools/products Using DoP for Query Tuning

Indexes for Large Rows

Temp on Flash

Forcing Use of More Memory

Not understanding the big/little data

challenge

Indexes for Large Rows

Moving to a solution before

understanding the problem

Indexes

Temp on Flash

Forcing Use of More Memory

Disable Broadcast Distribution


OLTP Basics


Connection Pools Performance Data

The workload is increased by

doubling the load. System

appears scalable up to 60%

CPU on the DB server.



A checkpoint is initiated,

creating a CPU spike that

results in unpredictable

response time



A slight increase to the

workload results in a

disproportionate CPU

increase and response time

degrades. System

monitoring tools become

unreliable



Reducing the connection pool by 50% results

in more application server queuing and less

DB processes in a wait state. No observable

improvement in response time or transaction

rate (value or consistency)



Connection pool reduced to 96.

Note improvement in response

time and transaction rate.

CPU utilization is reduced.


Resource Management Performance Data

By reducing the CPU_COUNT in the

resource manager, the database

can be throttled back. Note the

increase in response time and wait

event resmgr: cpu quantum


SQL Programming Basics Performance Data

Transaction rate: 300 TPS


CPU profile shows equal

system and user

utilization





Hard parsing and shared

pool contention. Do we

have a security issue with

literals?




Response time: 1 ms

No contention and no

parsing. CPU is well

utilized.



Performance

significantly reduced on

production as compared

to development and test

Contention in the

shared pool


Programming Errors Performance Data

Error message:

ORA-01000 Maximum open

cursors exceeded

“SQL*Net

break/reset to

client”


Programming Errors Session Details

SQL*Net

break/reset to

client


Programming Errors Cursor Data

Cursor list with Count > 1

implies “leaked” cursors



Load diminishes to zero


Bad Performance #4

Potential indicators of session leaking:

– Frequent application server resets

– init.ora parameters process and sessions set very high

– Configuration of large and dynamic connection pools

– Large number of idle connections connected to the database

– Free memory on database server continually reduced

– Presence of idle connection kill scripts or middleware configured to kill idle

sessions

Session Leaking


Bad Performance #5

Without warning, the database appears to hang and the application

servers time out simultaneously

The DBA sees that all connections are waiting on a single lock held by

a process that has not been active for a while.

Each time the problem occurs, the DBA responds by running a script to

kill sessions held by long time lock holders and allowing the system to

restart.

Observations



Leaked lock holder

causes entire

system to stall


Programming Errors Blocking Tree

Block tree reveals

lock holder which

can be killed



enq: TX – row lock contention



System returns to normal

when lock is released.

There could have been

logical data corruption

and it may happen again


Set Based Processing Concepts


Loading


Loading


De-duplicating


De-duplicating


Transforming


Transforming


Aggregating using Set-based Method


Aggregating using Pipelined Table Function Method


Running out Memory


Huge Pages

To fill a shape with an image.

1. Use existing picture box, DO NOT delete and

create new picture box.

2. Right click on the shape.

3. At the bottom of the submenu select

“Format Shape”

4. Select “Fill” at the top of the “Format Shape”

dialog box.

5. Select “Picture or Texture fill” from the options.

6. And select “File” under the “Insert from” option.

7. Navigate to the file you want to use and

select “Insert”

8. On the “Format” tab, in the Size group, click on

“Crop to Fill” in the Crop tool and drag the image

bounding box to the desired size

9. DELETE THIS INSTRUCTION NOTE WHEN

NOT IN USE


Two system identical other than one (top) uses huge pages for SGA and other does not

Both systems have huge pages configured

At small number of connections there is little difference in memory utilization

Scenario

Huge Pages


Increase connection by 100x but keep workload fixed

Page table grows much faster if not using huge pages

Causes system to swap

– High system CPU

– Poor performance

Scenario

Huge Pages


Reduce connection by a few

hundred relieves memory

pressure and system starts

working again

BUT…

Scenario

Huge Pages


Page table usage is on demand

so over time continues to

increase until swapping

resumes

Scenario

Huge Pages



Documents

Copyright © 2012, Oracle and/or its affiliates. All rights ... · Tom Kyte Been with Oracle ... Indexing is an OLTP technique for operations on a small number of rows ... No change