Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential
EXADATA HEALTH AND RESOURCE USAGE MONITORINGIncluding the Benefits of Administration Tuning Methods
Kellyn Pot’VinConsulting Member of Enterprise Manager Technical TeamStrategic Customer ProgramMike ChafinPrincipal Member of Technical Staff
Oracle Open World 2014
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Safe Harbor StatementThe following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
Oracle Confidential 3
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 4
Program Agenda
1
2
3
4
5
Oracle Tuning Methodologies
Preparing for Tuning
Working through Tuning Exercise
Digging Deeper
Miscellaneous Tips
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 5
Program Agenda with Highlight
1
2
3
4
5
Oracle Tuning Methodologies
Preparing for Tuning
Working through Tuning Exercise
Digging Deeper
Miscellaneous Tips
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 6
Evolution of Methodologies
Documented in the Oracle Documentation Set:
Database 2 Day + Performance Tuning Guide
Oracle Database Performance Tuning Guide
Exadata Performance Tuning White Paper
Each are similar, in that they offer a structural approach to problem solving using the latest tools and technologies.
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 7
Tuning Level
What level is the Issue?
• Hardware
• Database
• Schema/Design
• SQL
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 8
Important Tuning Concepts
Tuning a database by “DBTime”
DBTime is the term used by Oracle to calculate the time used by active CPU, IO and other foreground processes from the total time elapsed to calculate wait events that are performed within the database.
“Tune for Time or You’re Wasting Time…”
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 9
Program Agenda with Highlight
1
2
3
4
5
Oracle Tuning Methodologies
Preparing for Tuning
Working through Tuning Exercise
Digging Deeper
Miscellaneous Tips
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 10
Tuning Exercise Steps
• Prepare
• Compare
• Review (research, investigate)
• Rule out, Test
• Implement/Resolve
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 11
Prepare
Collect hardware and software healthchecks • exachk, cluvfy, emdiag,• Healthchecks• AWR Baseline • Verify and/or set AWR retention period
Copy configuration files• Automate collection and base-lining (EM jobs)
Create AWR, ASH and ADDM reports for research.
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 12
Stopping the Insanity Flowchart
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 13
The Value of Quick Check Lists
Old School, but VALUABLE.
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 14
Program Agenda with Highlight
1
3
2
4
5
Oracle Tuning Methodologies
Preparing for Tuning
Working through Tuning Exercise
Digging Deeper
Miscellaneous Tips
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 15
Comparison Tools
• Incredible Power in Knowing What’s Changed.
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 16
CompareIf there is a problem what has changed? And who might know?
Considerations• Patch levels (everywhere!)• Schema• Tunable OS parameters • Resource Management Plans• Code Changes• ADDM Comparison Report
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 17
Compare Configurations- Exadata Level
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 18
Compare Configuration- Database Level
• EM Job to compare one ‘reference’ database against one or more other databases
• Job can be scheduled on a repetitive basis, or run ad-hoc
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 19
Compare Configurations- Schema Level
Capture Schema baselines
Compare schema’s
• With a baseline
• Between different databases
Synchronize schema’s
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Compare ADDM
• Full ADDM analysis across two AWR snapshot periods• Detects causes, measure effects, then correlates them
Causes : Workload changes, Configuration changes Effects : Regressed SQL, Reach resource limits (CPU, I/O, memory, interconnect)
• Makes actionable recommendations along with quantified impact
AWR Snapshot Period 1
AWR Snapshot Period 2
Analysis ReportCompare Period ADDM
SQL Commonality
Regressed SQL
I/O Bound
Undersized SGA
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 21
IO Issues
What type of IO Issues are addressed with Exadata features?How can temp tablespace usage be an impact to performance in an Exadata?
• Identify IO Issues- ASH Analytics, ADDM Comparison Report
• Isolate IO Issues- Search SQL, ASH/ADDM Reports,
• Address IO Issues- IO Resource Manager, SQL Tuning Advisor,
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 22
Exadata and SQL Monitor
Offload Percent Efficiency Displayed in Report and UI.• Automatically monitors long running SQL• Enabled out-of-the-box with no performance overhead• Monitors each SQL and PL/SQL execution • Shows global PL/SQL and SQL level statistics
• Drill-down to slow SQL for diagnosing unexpected PL/SQL behavior
• Guides tuning efforts
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
IO Bound Dashboard in ASH Analytics• Base vs. comparison period• Temp reads/writes specified• Single block read latency
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 24
Setting Thresholds
Thresholds with incident rule sets to notify when growth in resource occurs and impacts the environment.
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 25
IO Resource Manager
• IO Resource Manager, (IORM) offers ability to maintain consistent performance levels across an engineered system.
• Allocates and limits IO per settings configured
• Simple configuration access via Enterprise Manager
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 26
Enterprise Manager Exadata Resource Management
• Exadata plugin specifica access to manage IO Resources for db and across cell nodes.
What IO resources on Exadata are shared vs. on a non-Exadata RAC environment?
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 27
IO Resource Management
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 28
Monitoring IORM
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 29
Monitoring Disk Utilization
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 30
Max Cell Disk Limits
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 31
Why Setting Limits are Important
• No Hard Limits equate to IO being freely redistributed
• Setting limits ensures that resources are available for critical databases
• Less high priority databases aren’t able to allocate more than necessary
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 32
Review
• Errors in the system?
• EM Incident Manager
• Alert Log(s)
• Grid Infrastructure
• ASM
• Database
• OS logs / system logs
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 33
AND Rule Out- Hardware
• Compute Node CPU
• I/O Bound / at known limits
• Exadata – Flash cache
• Mixed Workloads creating contention?
• ALL Components up!
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 34
Exachk- See Oracle Support Document 1070954.1: “Oracle Exadata Database Machine Exachk or HealthCheck”
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 35
Exachk and Healthcheck Utility Checks…
• Incorrect configurations
• Parameter Issues
• Recommendations, steps to correct, along with current value.
See Oracle Support Document 1070954.1: “Oracle Exadata Database Machine Exachk or HealthCheck” and Exadata best practices in Oracle Support Document 757552.1: “Oracle Exadata Best Practices
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 36
Automate Exachk via Enterprise ManagerAutomating will populate healthchecks to UI!
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 37
Storage Grid OverviewStorage Cell monitoring and administration support
–Cell Home page and performance pages
–Actions supported: Start/stop Cell, verify connectivity, setup SSH
Automatic discovery of Cells
Management by Cell Group
–All cells used by a database automatically placed in a group
–Cell Group level administration operations (batch job monitoring)
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 38
More Storage Grid Overview
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 39
Hardware- DB Node View
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 40
Hardware- Cell Server View
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 41
Program Agenda with Highlight
1
4
2
3
5
Oracle Tuning Methodologies
Preparing for Tuning
Working through Tuning Exercise
Digging Deeper
Miscellaneous Tips
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 42
AWR Baselines
• Collection of snapshots used for performance comparisons.
• Baselines are retained within the AWR even after the retention time for the data has been reached.
• Exadata should have a moving and a static baseline in place to capture different workloads.
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 43
Details of Baseline Creation in EM
Highlight a snapshot range to use for your static baseline or use a time range to base it from.
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 44
AWR Baselines
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 45
Completing Baseline Creation in EM
Schedule the job to run in a non-impacting schedule and choose not to purge.
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 46
SQL Issues
SQL Tuning – what kind of problem?• Reactive• Transient• Long Running SQL• Proactive• Persistent• Comparative
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 47
Proactive SQL Tuning Choices
• SQL Performance Analyzer• SQL Access advisor• ASH and ADDM to prioritize further work• Group operations (DB12)• Create an SLA / baseline• Use SQL Plan Baselines (SPM)• Validate with SQL Performance Analyzer
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 48
Reactive SQL Tuning Choices
After all changes, including any environments are ruled out…
• Start with AWR, ADDM and ASH
• Identify outliers / worst performing (Pareto)
• Review top wait events
• SQL Tuning Advisor
• If resource constrained consider resource limiting strategies
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 49
Database Performance Page in EM12c
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 50
ASH Analytics
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 51
ASH Analytics
Requirements for 10g and 11g databases:• Additional PL/SQL packages to be installed into the DBSNMP schema• CONNECT, CREATE TYPE and CREATE PROCEDURE privileges• Submit the job to install as SYS from the UI
View information in real-time in the database
Save a snapshot as HTML (or email the report) for offline analysis• Active report, user can visualize data on multiple dimensions, (or run from command line for text version)
Help videos available: Use Active Session History (ASH) Analyticshttps://apex.oracle.com/pls/apex/f?p=44785%3A24%3A0%3A%3ANO%3A24%3AP24_CONTENT_ID%2CP24_PREV_PAGE%3A5451%2C1
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Advisor CentralAll database advisors grouped together on one launch page:
• Advisors, focused on recommendations
• Checkers, focused on diagnostics
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 53
SQL Advisors• SQL Tuning Set
– Capture of SQL in the database
– Ideally run after a representative workload has been run
• Access Advisor– Analyses access patterns of SQL in the cache, or from a defined workload, and gives
recommendations on how to (re)structure the database objects
• Performance Analyzer– Analyses before and after images of SQL in a SQL Tuning Set (Testing scenarios)
– Compares results
• Tuning Advisor– Requires Tuning Sets (Collection of SQL from the database)
– Analyses the SQL from a SQL Tuning Set and gives recommendations
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 54
AWR Warehouse and Exadata
AWR Warehouse
Scheduler jobs with Agent to Agent push to AWR Warehouse Target, scheduled by EM12c EM Job Service.
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 55
Query Across Entire Exadata to Show True Consolidation Value
Adapt Valuable Exadata Specific Queries to AWR Warehouse-
from dba_hist_sys_time_model stm, dba_hist_snapshot s, gv$parameter p, dbsnmp.caw_dbid_mapping m
where stm.stat_name in ('DB CPU','background cpu time')
and LOWER(m.target_name)= '&dbname'
and s.dbid= m.new_dbid and s.snap_id = stm.snap_id
and s.dbid = stm.dbid and s.instance_number = stm.instance_number
and p.name = 'cpu_count' and p.inst_id = s.instance_number)
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential – Internal/Restricted/Highly Restricted 56
IO Information Across Databases on Exadata?
16:5916:5917:2917:2917:5917:5918:3018:3019:0019:0019:3019:3020:0020:000
50000
100000
150000
200000
250000
300000
350000
400000
June 10, 2013
l_reads_s read_iops_maxread_iops_direct read_iops_direct_max
20:3020:30
20:5920:59
21:3021:30
22:0022:00
22:3022:30
23:0023:00
23:3023:30
23:5923:59
0:290:29
0
100000
200000
300000
400000
500000
600000
June 10, 2014
l_reads_s read_iops_maxread_iops_direct read_iops_direct_max
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 57
Answer Specific IT Questions
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 58
Program Agenda with Highlight
1
5
2
3
4
Oracle Tuning Methodologies
Preparing for Tuning
Working through Tuning Exercise
Digging Deeper
Miscellaneous Tips
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 59
Making Copies of Configuration Files
Copy and automate copying of essential database files via Enterprise Manager-
• All instance/database init and spfiles• Database password files• ASM init files• ASM password files
Create an Enterprise Manager job to copy these files on a regular basis.
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 60
A Review of Hardware Network “Red Herrings”• HA can mask problems, reduced capacity even while
meeting service levels
• Hardware Utilization Issues
• CPU is not what it may seem
• I/O Subsystem (Cell screens)
• Network Congestion (IB screens)
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 61
Achieving the tao of Exadata
Exadata is an Engineered System, Powerful and Complex!
• Perform Exachk reviews quarterly, (preferably monthly.)
• Install Enterprise Manager Plug-in for advanced monitoring and enhanced performance impact detection.
• Take advantage of the great performance and tuning features included as part of the diagnostic and tuning packs
• Use IO Resource Manager to deter from impact to critical database systems.
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 62
Want to Learn More?
Whitepapers and Links: http://www.oracle.com/technetwork/database/availability/exadata-health-resource-usage-2021227.pdf
http://www.oracle.com/us/products/enterprise-manager/index.html
http://docs.oracle.com/cd/E11857_01/install.111/e23441/pichk.htm
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | Oracle Confidential 63
Connect with me via Social Media: