Towards Self-Tuning Databases Sang-Won Lee ([email protected])[email protected] SKKU VLDB Lab. ( oralab )oralab

Towards Self-Tuning Databases

Sang-Won Lee([email protected])

SKKU VLDB Lab.( http://oralab.skku.ac.kr/~oralab )

Nov. 1, 2002

mailto:[email protected]

http://oralab.skku.ac.kr/~oralab

http://www.vldb.informatik.hu-berlin.de/index.html

http://oralab.skku.ac.kr/sos/default.php

SKKU VLDB Lab.Toward Self-Tuning Databases

2

Contents

• Introduction

- Why DB tuning?

• Current Database Tuning Paradigm

• Toward Self-Tuning Databases


3

What is DB Tuning?

• Car tuning- Goals? - What? - How?


4

DB = IT Core Platform in the 21st Century?

• ERP/CRM/SEM/BSC

• DW/OLAP/Data Mining

• Web Log Analysis

• GIS/XML/Mobile

• Bio-informatics

• ....

• Ubiquitous Databases


5

Why DB Tuning in the 21st Century?

• DB performance influences all aspects of your business- DB performance management is one of the most important

success factors in almost every IT projects- “ 미래 웹경쟁력은 데이터처리능력에” (Scott McNealy, Sun

CEO, 전자신문 )

• Database tuning is not a technical issue!

• Database tuning is a business issue!

DB tuning can improve your productivity

DB tuning can increase your revenue/profit

DB tuning can increase your ROI(Return on Investment)


6

Why DB Tuning in the 21st Century?(2)

• More data- doubling every 9 month

• More users- from CEO to sales representatives

• More complex query- e.g. OLAP/data mining query

• Faster responses- e.g. web response time impacts revenue!- e.g. real time personalization

Without DB tuning, you need double-up investment on DB server and software every 6 month!


7


• Data growth vs. computer speedup

** Moore’s Law -- # of transistors/chip doubles every 18 months (1965)

0

500

1000

1500

2000

2500

3000

3500

Year

Pe

tab

yte

s

DiskSalesMoore'sLaw


8


• Disk sales doubling every 9 months - Greg Papadopoulos, Sun CTO- similar results from Winter VLDB survey (http://www.wintercorp.c

om)

• Time to process all your data doubles every 18 months!- You need DPA, not DBA!

- Database Performance Administrators- BUT! they are extremely expensive!!

• Maximize your DB performance!! (DBMax)Maximize your PRODUCTIVITYMaximize your REVENUE/PROFITMaximize your ROI


9

Tuning Activities for Databases

Systems (H/W, O/S)Systems (H/W, O/S)

Oracle ServerOracle Server DatabaseDatabaseApplicationsApplications

SQLSQL

Oracle Database Architecture

Components for Tuning Check Lists for Tuning Criterial for Tuning

ApplicationsLogic / Server Connection / APPS Config etc Response Time, CPU usage

SQLAccess Path / Join Order / Join Methods / Parallelization / Efficient Parsing etc. SQL Tuning

Elapsed Time, Plan, Trace, Performance Statistics

Oracle Database ServerSGA Size / Server Configuration / Initial Paramaters Setting Performance Statistics, Waiting

Event, CPU usage

Logical & Physical Database Design

Physical DB Design / IO Balancing / De-Fragmentation / Removing Chaining etc.

Performance Statistics, Waiting Event, Disk I/O

Systems (H/W, O/S)Addign CPUs / Kernel Parameter Setting / RAID Config. etc CPU Usage, Memory Usage,

Disk I/O Traffic


10

Current DB Tuning Paradigm

1. (Logical) DB design DB Modeller

2. H/W set-up and configuration HW experts

3. Application development and test Developers

4. System open DPA

5. System operation

6. Data increases

7. Unpredicted performance problems DPA

MonitoringDrill-down

to BottleneckRoot Cause

AnalysisTuning


11

Why DB Monitoring in the 21st Century?

• DB tuning vs. DB monitoring

- DB Monitoring DB Tuning

- E.g.: Tuning 10 hour SQL into 10 minute SQL is not any more special skill

• Why DB monitoring is so difficult in the 21st century?

- e.g. ERP, DW/OLAP, CRM etc

- packaged and tool-based

- DB schema is too complex

- SQL is unknown

- data size is unpredictable

- .....


12

Why DB Monitoring in the 21st Century?(2)

• Usually, thousands of SQL statements are running concurrently in a database

• Cho & Lee’s rule

- “When database performance is low, less than 5% SQLs are using more than 95% database resource”

• A key issue in DB monitoring

- To rapidly identify the problematic SQL statements!

- Without tool, this task is extremely difficult


13

Why Self-Tuning Databases?

• War of TCO(Total Cost of Ownership) in 21st century

- Major costs of IT infra. ownership

- H/W, S/W, Human Cost

- Moore’s law: H/W, S/W cost ↓

- BUT! Human cost ↑

- cf. 1950s: telephone industry

- manual switching by operators automatic switching system


14

Why Self-Tuning Databases?(2)


15



16


• Commercial DBMSs are expontially complicated

- Even 10-year veteran Oracle DBAs do not know 50% of all the new features of Oracle’s latest release

- Worsely, it would take more than 1 week for them to understand the performance behavior of a single new feature

- The Gap between human DB learning curve and new features becomes larger

- “Crossing this chasm” seems to be impossible!!


17

Industry Cases: IBM DB2

• SMART project: Self-Managing And Resource Tuning

- database part of IBM’s Autonomic Computing Initiatives: IBM Autonomic Computing

- Features

1. Up and Running: “configuration advisor”

- 98.4% of experts; outperform average DBAs

2. Design: “index/design advisor”

3. Maintenance: “LEO(LEarning Optimizer)”

4. Problem Determination and Correction: “health center”

5. Availability and Disaster Recovery: “recovery expert”


18

Industry Cases: IBM DB2


19

Industry Cases(2): Oracle9i

• Oralce 9i Manageability: more conservative approach

1. Self-Manageability(more focused)- space, memory, processor, operations

2. Self-Tuning- Dynamic memory management: buffer cache, run-time SQ

L memory(sort/hash area)

Oracle Memory Model

Sort Area, Hash Area, Bitmap Index Creation/Merge

• Pre-Oracle9i

- Fixed PGA size

- Sort(Hash)_Area_Size

• Oracle9i

- PGA_Aggregate_Target

- Dynamically allocated according to the needs


20


050

100150200250300350400450500

1 2 4 6 8 10 12 16 20

Auto

Manual - 5MB

Manual - 15MB

Avg. Response time in seconds

Number of Users


21


0

1,000

2,000

3,000

4,000

5,000

6,000

1 2 4 6 8 10 12 16 20

Manual - 5M

Manual - 15M

Auto

Total Memory Usage in MB

Number of Users


22

Industry Cases(3): MS SQL Server

• AutoAdmin: Self-Tuning and -Administering Databases

“We achieve this by enabling databases to track the usage of their systems and to gracefully adapt to application requirements. Thus, instead of applications having to track and tune databases, databases actively auto-tunes itself to be responsive to application needs.”

- See AutoAdmin homepage for details

- http://www.research.microsoft.com/research/dmx/autoadmin/

- “Rethinking Database System Architecture: Toward a Self-Tuning RISC-style Database System”


23

Industry Cases(4): XM

• DBMax for Oracle(http://www.ex-em.com)- Not Self-Tuning Databases

- Domestic Oracle Performance Monitoring Tool- We plan to incorporate intelligent recommendation wizards for D

B tuning- Application logic- Batch program scheduling- Automatic problem tracking down- Configuration setting- Dynamic memory adjustment- SQL tuning: based on “Query Graph” methodology

- remedy for optimizer’s internal “size estimation cost formula” & “attribute value independence assumption”


24

References

• Gerhard Weikum et al., “Self-Tuning Database Technology and Information Services: from Wishful Thinking to Viable Engineering” VLDB 2002, Ten-Year Award

• Gerhard Weikum et al, “The COMFORT Automatic Tuning Project”, Information Systems 19(5), 1994

• Gerhard Weikum et al, “Special Issues on Self-Tuning Database and Applications,” Data Engineering Bulletine, Jun., 1999

• IBM’s Autonomic Computing Perspectives: http://www.resarch.ibm.com/autonomic/manifesto/autonomic_computing.pdf

• Guy Lohman et al, “Toward Automomic computing with DB2 Universal Database,” SIGMOD Record, Sep., 2002

• MS AutoAdmin Homepage: http://www.research.microsoft.com/

• Oracle Manageability: http://technet.oracle.com/products/manageability/content.html


25

Let’s DBMax!

Documents

Towards Self-Tuning Databases Sang-Won Lee ([email protected])[email protected] SKKU VLDB Lab. ( oralab )oralab