#.1 Average Active Sessions (AAS) The Golden Metric ? Kyle Hailey

#.1

Average Active Sessions (AAS)The Golden Metric ?

Kyle Haileyhttp://perfvision.com

#.2205/03/23 Copyright 2006 Kyle Hailey

In this SessionIn this Session1. AAS

Single Metric Shows DB Performance Two methods to calculate

Sampling Time statistics

2. Yardstick Max CPU CPU Count To measure AAS against

3. Subcomponents CPU Waits Time series

#.3305/03/23

Database Performance Database Performance How quick can you find

Bottleneck in DBIf DB is idleCurrent DB Load

what is DB Load ?

What do you use?Statspack/AWRV$active_session_historyAlerts

what do you alert on ?

what the *!####!*!*? is the database doing ?!

#.4

Active SessionActive Session

Copyright 2006 Kyle Hailey

Database Machine

SQL*Plus

Shadow

Process

SQL*Plus

Shadow

Process

Application Server Machine

Shadow

Process

Shadow

Process Shadow

Process

Application

SGA

#.5

idle



SQL*Plus (ie application)

idle idle idle

Have a coffeewaiting waitingTyping waitingtyping typing

Query 1 Query 2 Query 3

Database (shadow process)

TimeActive Active Active

Shadow

Process

#.6


Database

User 1User 2User 3User 4

Active Sessions

=

1234

Graph represents # of sessions active, but also represents amount of time active in the database

#.7



Active Sessions

Users Waiting

For Every Active Session there is a user (or application) waiting

1234

1234

#.8

Measuring Active Sessions


Fast query run often

Fast query run rarely

Slow query

If happens a lot or for long … we’ll catch it, guaranteed

User 1

User 2

User 3

Sampling Every Second

. . .

#.9Copyright 2006 Kyle Hailey

Sampling is like taking PicturesSampling is like taking Pictures

#.10

idleidle



idleidle idleidle idleidleQuery 1 Query 2 Query 3Database

To execute a query , some CPU will be used but we also might spend time waiting for IO or on waiting for concurrency resources like latches

CPU IO Wait

idleidleidleidle idleidle idleidleDatabaseQuery 1 Query 2 Query 3

Activity can thus further be broken down into the type of activity: CPU, IO or WAIT

Work Work LatencyLatencyContentionContention

#.11

Session 1

Session 2

Session 3

Session 4

TIME

Graphical ASHGraphical ASH


Graph of User StatesGraph of User States


One Second GraphOne Second Graph


15 Second Averages15 Second Averages


Maximum CPU LineMaximum CPU Line


Idle UsersIdle Users


OEM Perf Page OEM Perf Page

#.18

AAS in OEMAAS in OEM

LOAD

AAS


The Power ASH gives AASThe Power ASH gives AASDB Home DB Home

PerformancePerformance

Top ActivityTop Activity

Based on Based on ASHASH

Based on TIMEBased on TIME(events Statistics)(events Statistics)

AAS=db AAS=db time/elapsed timetime/elapsed time

AAS=count active AAS=count active users /samplesusers /samples


DB TIME = area under the curveDB TIME = area under the curve

DB Time = DB Time = active sessions(ti) * ΔtΔt nn

ΣΣ00

Height = # of SessionsHeight = # of SessionsWidth = secondsWidth = secondsArea under curve = DB TimeDB Time

DB Time = sum of active time in databaseDB Time = sum of active time in database


AAS SourcesAAS Sources

1. Manually from v$sysstat (9i : v$system_event )

2. Statspack Need several calculations

3. AWR One calculation

4. OEM 10g Directly displayed

AAS = DB Time/Elapsed Time


1. Manually1. Manually

DB Time (DBT) = Time Spent in Database

DB TIME (10g) =

DB TIME (9i) = Select sum(time_waited) from v$system_event where event not in ( ... idle events …);+Select value from v$sysstat where name = ‘CPU used by this session’;

select value from v$sysstat where name = ‘DB time’;‘DB time’

still need to take delta valuesNote: stats$idle_event : 70 v$event_name.wait_class=‘Idle’ :62

AAS = DB TIME / Elapsed Time


2. Statspack AAS 2. Statspack AAS Look for

Elapsed TimeTop 5 Timed Events

Start at line 52 of about 1300Start at line 52 of about 1300


2. Statspack AAS2. Statspack AAS

Elapsed Time

Look at Top 5 Timed EventsTop 5 Timed EventsTop 5 Timed Events~~~~~~~~~~~~~~~~~~ % Total~~~~~~~~~~~~~~~~~~ % TotalEvent Waits Time (s) Call TimeEvent Waits Time (s) Call Time

--------------------- --------- --------- ---------------------------- --------- --------- -------

buffer busy waits 2,748 250 78.72buffer busy waits 2,748 250 78.72CPU time 32 10.16CPU time 32 10.16free buffer waits 1,588 15 4.63free buffer waits 1,588 15 4.63write complete waits 10 8 2.51write complete waits 10 8 2.51log buffer space 306 5 1.51log buffer space 306 5 1.51

STATSPACK report for

DB Name DB Id Instance Inst Num Release RAC Host------- ----------- -------- -------- ---------- --- -------

LABSF03 1420044432 labsf03 1 10.1.0.2.0 NO labsfr

Snap Id Snap Time Sessions Curs/Sess

--------- ------------------ -------- ---------

Begin Snap: 1 03-Apr-06 12:34:06 18 5.6 End Snap: 2 03-Apr-06 12:34:36 18 4.8 Elapsed: 1.00 (mins)


2. Statspack AAS2. Statspack AAS

DBTIME= CPU + WAITSCPU = 32 WAITS = 250+15+8+5 = 278 secs

DBTIME=320 Elapsed Time = 60 secs 320 secs / 60 secs

AAS = 5.1

Top 5 Timed EventsTop 5 Timed Events Event Time (s) Event Time (s) ----------------- ---------------------- -----buffer busy waits 250buffer busy waits 250CPU time 32CPU time 32free buffer waits 15free buffer waits 15write complete waits 8write complete waits 8log buffer spacelog buffer space 55

#.262605/03/23

3. AWR Report3. AWR ReportAAS = DB Time/Elapsed Time

23.56/59.66 = 0.39

AAS= 0.39

#.272705/03/23

4. OEM 10g4. OEM 10g

AAS = ~0.75

•OEM 10g


Got AAS, Now What ?Got AAS, Now What ?

We Need one more item:

CPU Count

# of CPUs available on SystemShared with other applications

Need to track CPU used on the system as wellOn dual & quad cores, lower the CPU countRepresents max active sessions that can do work


CPU CountCPU Count # of CPUs available in

Statspack 10g AWR report OEM 10g

Statspack 9i # of CPUs missing # of CPUs

SQLPLUS> show parameters cpu_count

NAME VALUE------------------ ----------cpu_count 2


What’s the DB Doing?!What’s the DB Doing?!

It’s 2am … your manager calls

Whip out the stethoscope:

AASwhat the *!####!*!*? is the database doing ?!


AAS FormulasAAS FormulasUse CPU count as yardstick:

AAS < 1 Database is not blocked

AAS ~= 0 Database basically idleProblems are in the APP not DB

AAS < # of CPUsCPU availableAre any single sessions 100% active?

AAS > # of CPUsCould have performance problems

AAS >> # of CPUSThere is a bottleneck

Ideal world – one databasesolution track CPU at OS

AAS > 1 still want to know if a single user is 100% active


Available CPU vs AASAvailable CPU vs AASAAS far above available CPU => problem

AAS = 5.1

# of CPU = 2

AAS = 0.39

# of CPU = 2

AAS < 1 , database is fine

Statspack DBTIME= CPU + WAITS

CPU = 32 WAITS = 250+15+8+5 = 278 secs

DBTIME=320 Elapsed Time = 60 secs 320 secs / 60 secs

AAS = 5.1

AWR Report

AAS = 0.75

# of CPU = 2

AAS < 1 , database is fine


Going Farther with AASGoing Farther with AASAAS can tell you a lotBut it’s components tell you much moreTo go farther need the components of AAS

1.CPU 2.Wait3.Value over time

Only OEM 10g shows the value over time, Statspack and AWR are aggregated over the snapshot period


EM DB Home PageEM DB Home Page


OEM 10g Perf PagesOEM 10g Perf PagesDB Home DB Home


AAS Point in TimeAAS Point in Time

AAS over TimeAAS over Time

#.363605/03/23

AAS Components : OEM 10gAAS Components : OEM 10g

OEM 10gPerformance Page

Available CPU AAS:CPU + WAIT

Real CPU available:

Max CPU - non instance CPU

#.373705/03/23

OEM 10gOEM 10g

Relax Relax Get to Work!Get to Work!Looks Looks OK OK

But …But …


Idle Database – Perf PageIdle Database – Perf Page Value of proving the database is

Idle It’s the Databases Fault

How many times do you hear that? Database Idle

No load on database Database “performance” is fine

Under utilized Problem lies elsewhere Saved me time and stress many

times And weeks of debate about where the

problem is coming from


More than AAS and #CPUMore than AAS and #CPUKnowing your DB Profile


When to TuneWhen to Tune1. Machine

a) CPU Response times skewed 100% CPU might be fine Users wait in queue (run queue) => machine

underpowered

b) Memory Paging Wait times skewed (ex : latch free) Erratic response times ( ex : ls )

2. Oracle1) Waits > CPU ?

tune waits2) CPU > 100% ?

tune top CPU SQL3) Else

It’s the application

Oracle Load (AAS)

Top Session Top Wait Top SQL

SQL Detail Session Detail File DetailObject Detail Wait Detail

Host

AAS > #CPU

AAS > 1Waits > CPU

CPU > Waits

CPU Memory

#.414105/03/23

Limited Analysis Limited Analysis What if you find a problem ?Of the 800 waits which in order to solve most need to know

What SQLWhich sessionsValues of P1, P2 and P3Statspack and AWR fail

AAS = DB TIME / Elapsed Time

But there is another wayBut there is another way … …


AAS based on ASHAAS based on ASH

ASH - Active Session History v$active_session_historySamples sessions once a second

AAS = count(*) / elapsed_secondsA statistical approximation, but surprisingly close

ASH data source empowers drilldownsTop Sql Top WaitsDetails p1,p2,p3 and more


ASHASHvsvsStatisticsStatistics

Statistics are more expensivehave lag time lack clear identification of culprits


Statistic Lag TimeStatistic Lag Time

StatisticsStatistics

Samples (ASH)Samples (ASH)

Slight LagsSlight Lags


CPU Lag ProblemCPU Lag Problem ASH is the only way to see CPU usage realtime V$sysstat reports CPU but

is only updated at the end of the call. Long calls look deceiving like no CPU is being used

Time Model also reports CPUUpdated quicker


CPU in ASH vs StatsCPU in ASH vs Stats

#.474705/03/23

2. OEM 10g : Top Activity2. OEM 10g : Top Activity•Top Activity

•Based on ASH•Enables Drilldowns

•Top SQL•Top Session

•Drill into a session• Stats• Raw waits• Open cursors• General info

•Drill into a SQL• Stats and text• Users executing• Explain plan• Tuning options


Top Activity : Based on ASHTop Activity : Based on ASH

missingmissing

Thanks Thanks To To ASHASH


AAS – %Session Time IssueAAS – %Session Time Issue

Whenever AAS > 1I want to know if any one session is 100% activeUnfortunately OEM isn’t set up for this(ASHMON, my free tool, is )


Top Activity: ASH SessionsTop Activity: ASH Sessions

Many Users Active Many Users Active

On Performance Page, no way to tell how many usersOn Performance Page, no way to tell how many users

But Top Activity Page fixes that But Top Activity Page fixes that


Top Activity: ASH SessionsTop Activity: ASH Sessions

Two Users ActiveTwo Users Active

Shown in % DB TimeShown in % DB TimeMissing % Session TimeMissing % Session Time


OEM 10g Perf PagesOEM 10g Perf PagesDB Home DB Home



SQLSQLSessionSession



SQLSQLSessionSession


Session : ASH ActivitySession : ASH Activity


SQL : ASH ActivitySQL : ASH Activity


AAS from ASHAAS from ASH1. ASHRPT

Based entirely on v$active_session_history

@?/rdbms/admin/ashrpt.sql Exec ASH_REPORT_TEXT/HTML

select * from tableselect * from table(dbms_workload_repository.ash_report_text((dbms_workload_repository.ash_report_text( (select dbid from v$database),(select dbid from v$database), 1,1, sysdate – 1/24, sysdate – 1/24, sysdate )) ;sysdate )) ;


1. ASHRPT1. ASHRPTASH Report For TESTDB/testdbASH Report For TESTDB/testdbDB Name DB Id Instance Inst Num Release RAC HostDB Name DB Id Instance Inst Num Release RAC Host

------------ ----------- ------------ -------- ----------- --- ------------------------ ----------- ------------ -------- ----------- --- ------------

TESTDB 2371570538 testdb 1 10.2.0.1.0 NO sdbe604aTESTDB 2371570538 testdb 1 10.2.0.1.0 NO sdbe604aCPUsCPUs SGA Size Buffer Cache Shared Pool ASH Buffer Size SGA Size Buffer Cache Shared Pool ASH Buffer Size

-------- ------------------ ------------------ ------------------ ------------------ ------------------ ------------------ ------------------ ------------------

22 1,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%) 1,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%) Analysis Begin Time: 21-Apr-06 12:00:01Analysis Begin Time: 21-Apr-06 12:00:01 Analysis End Time: 21-Apr-06 12:05:01Analysis End Time: 21-Apr-06 12:05:01 Elapsed Time: 5.0 (mins)Elapsed Time: 5.0 (mins) Sample Count: 3,716Sample Count: 3,716 Average Active Sessions: 12.39Average Active Sessions: 12.39 Avg. Active Session per CPU: 6.19Avg. Active Session per CPU: 6.19 Report Target: None specifiedReport Target: None specifiedTop User Events DB/Inst: TESTDB/testdb (Apr 21 12:00 to 12:05)Top User Events DB/Inst: TESTDB/testdb (Apr 21 12:00 to 12:05) Avg ActiveAvg ActiveEvent Event Class % Activity SessionsEvent Event Class % Activity Sessions

----------------------------------- --------------- ---------- --------------------------------------------- --------------- ---------- ----------

CPU + Wait for CPU CPU 67.98 8.42CPU + Wait for CPU CPU 67.98 8.42enq: TX - row lock contention Application 23.98 2.97enq: TX - row lock contention Application 23.98 2.97buffer busy waits Concurrency 4.66 0.58buffer busy waits Concurrency 4.66 0.58latch: cache buffers chains Concurrency 2.26 0.28latch: cache buffers chains Concurrency 2.26 0.28


1. ASH RPT1. ASH RPT

1) General info2) Top User Events ***3) Top Background Events4) Top Event P1/P2/P3 Values 5) Top Service/Module6) Top Client IDs7) Top SQL Command Types 8) Top SQL Statements ***

9) Top SQL using literals10) Top Sessions ***11) Top Blocking Sessions12) Top Sessions running PQs 13) Top DB Objects14) Top DB Files15) Top Latches16) Activity Over Time ***

#.585805/03/23

1. ASHRPT over Time1. ASHRPT over Time

Waits over TimeNot in AAS

Difficult but better than nothing

Compare to …

#.595905/03/23

3. Custom Scripts3. Custom ScriptsHate Graphics ?

Query v$active_session_history directlyJoin to dba_hist_active_sess_history for week of data

act.sqlLike top 5 timed events

Aveact.sqlCharts with text AAS by hour (15 minute, minute , etc)

Aveactn.sqlDitto, with top 2 wait events per bucket

Following Scripts Available on http://perfvision.com/ashscripts.php


3. Custom Scripts3. Custom Scripts@actAnalysis Begin Time : 2007-07-24 11:04:48Analysis End Time : 2007-07-24 11:19:45Start time, mins ago: 15Request Duration : 15Collections : 528Data Values : 3327Elapsed Time: 15 mins

WAIT_EVENT CNT % Active Ave_Act_Sess

-------------------------------------- ---------- ------------

latch free 10 .3 .02log buffer space 13 .39 .02buffer busy waits 14 .42 .03db file scattered read 15 .45 .03library cache pin 78 2.34 .15log file sync 213 6.40 .40ON CPU 726 21.82 1.38enqueue 855 25.70 1.62db file sequential read 1399 42.05 2.65

------------

sum 6.30


3. Custom Scripts3. Custom Scripts@aveactTM NPTS AVEACT GRAPH CPU WAITS---------------- ------ ------- ---------------------- ---- -----06-AUG 13:00:00 270 .33 +- 2 29 5906-AUG 14:00:00 1040 2.24 ++--------2--- 341 198406-AUG 15:00:00 623 6.67 ++++------2---------- 438 371806-AUG 16:00:00 1088 2.59 ++--------2---- 335 248606-AUG 17:00:00 1104 1.26 ++----- 2 349 104306-AUG 18:00:00 1093 1.38 +++---- 2 663 84206-AUG 19:00:00 1012 1.74 ++------- 2 373 138806-AUG 20:00:00 1131 .99 +---- 2 304 82006-AUG 21:00:00 1111 1.22 ++----- 2 344 101206-AUG 22:00:00 1010 1.66 ++------ 2 414 125906-AUG 23:00:00 1120 1.08 +---- 2 298 91307-AUG 00:00:00 1024 .83 +--- 2 273 57607-AUG 01:00:00 1006 1.74 ++------- 2 319 142807-AUG 02:00:00 1090 2.47 ++--------2---- 347 234507-AUG 03:00:00 687 6.59 +++-------2---------- 382 414207-AUG 04:00:00 1004 1.95 ++++++--- 2 1299 65907-AUG 05:00:00 1104 3.08 +++++-----2------ 1170 222607-AUG 06:00:00 1122 1.91 +++++++-- 2 1582 558


Aveact.sqlAveact.sql

#.63

Aveact.sqlAveact.sql


“-” = WAIT

“+” = CPUwhich waits ? -> aveactn.sql

#.64

Aveactn.sqlAveactn.sql



SummarySummary AAS: Two Sources

1. v$system_event & v$sysstat Performance page in OEM Statspack, awr report Indirect – based on time converted to AAS Accurate Lags (especially CPU) Limits analysis

2. v$active_session_history Top activity in OEM ashrpt Direct measure of AAS Real time Approximation ***Allows drilldowns***

Tuning: Machine first – CPU, Memory Oracle

AAS ~= 0 Idle AAS > 1 : possible sessions blocked AAS > # CPU : bottleneck

Oracle Load (AAS)

Top Session Top Wait Top SQL

SQL Detail Session Detail File DetailObject Detail Wait Detail

Host

AAS > #CPU

AAS > 1

Waits > CPU

CPU > Waits

CPU Memory

ADDM SQL Tuning Advisor

#.66

Q1Q1


What is the easiest way to find the load on the database

a. top 5 timed events

b. Average Active Sessions and CPU count

c. The transaction rate

d. The commit rate

b. Average Active Sessions takes into account all the major criteria in Oracle performance - cpu usage, wait time, elapsed time

others:

a. lacks the elapsed time

c & d - the amount of load created by commits and transactions varies drastically between database to database and even between different loads on a database during the day

#.67

Q2Q2


Average active sessions, the measurement in the performance chart in OEM 10g can be calculated by which formula(s)

a. db time / elapsed time

b. wait time / elapsed time

c. count of rows in v$active_session_history over an interval/ number of samples in interval

d. count of rows in v$active_session_history / seconds of elapsed time

answer

a,c,d

c, d are equivalent because ASH samples once a second

others

b doesn't work because it is missing CPU time

#.68

Q3Q3


If Average Active Session is less than one, we can say

a. the database is using less than 100% CPU on machine

b. the database is using 100% CPU

c. No session is completely blocked

d. Database performance should be acceptable

answer

a, c and d

#.69

Q4 Q4


An Average Active Session value of zero means

a. the database is down

b. the database is hung

c. the database is idle

d. the database is overloaded

answer

c

#.70

Q5Q5


CPU is updated in v$sysstata. when call finishesb. every secondc. every five secondsd. in real time

a - which can be a problem in monitoring tools if the call last a long time showing no CPU usage until the call finishes, then reporting an unrealistic spike of CPU when the call finishes

Documents

#.1 Average Active Sessions (AAS) The Golden Metric ? Kyle Hailey