Upload
tranphuc
View
215
Download
1
Embed Size (px)
Citation preview
Oracle Business Intelligence 11g
Performance Tuning OBI 11g - All The Options
Antony Heljula December 2012
© Peak Indicators Limited 2
Agenda Performance Tuning OBI 11g – All the Options
• Overview • Identifying Bottlenecks • Oracle BI Optimisation • Database Optimisation • Data Model Optimisation • Multi-Dimensional Cubes • Upgrade Database Hardware • Oracle Engineered Systems • Real-Time Optimisation • In Summary
© Peak Indicators Limited 3
Performance Tuning OBI 11g – All The Options
Overview
© Peak Indicators Limited 4
Overview Performance Issues
• Organisations of all shapes and sizes can be affected by performance issues – it is not just large companies
• As data volumes grow rapidly it is becoming increasingly difficult to achieve consistent and scalable performance in the long-term – especially when attractive new BI features/functions add extra load onto the underlying data sources
• Performance issues are rarely down to a single cause, often it is a combination of factors
• The way performance issues are dealt with will be different in every Organisation and can be dependent on a variety of things such as:
• Urgency Ownership Quick win vs Long Term • Budget Politics Complexity of Data Source • Available Resources Licensing etc
© Peak Indicators Limited 5
Overview Aim of Presentation
• Oracle probably provides more optimisation capabilities than any other vendor (covering database, middleware, hardware etc)
• The aim of this presentation is to outline the key options that exist for tuning Oracle BI 11g, with the objective of achieving major performance gains rather than just “tweeks”
• Some options are low-cost and some are not! The aim is also is to help you maximise “business value” by maximising the benefit from the low-cost options before considering the next alternative
• The main focus of this presentation will be on optimising a Data Warehouse environment, but the final topic will cover “real-time optimisation”
© Peak Indicators Limited 6
Overview Order of Play
• The topics are structured in this order, which is generally how most Organisations would approach the resolution of performance issues against a relational Data Warehouse:
Oracle Engineered System
Optimise Database
Upgrade Database Hardware
Multi-Dimensional Cube Engine
Optimise Oracle BI
Addition DB Licensed Options
Optimise Data Model
© Peak Indicators Limited 7
Overview Before we start….
• It is vital the Organisations give themselves the best possible chance of addressing performance issues in the cheapest/shortest way possible:
• Maintain development environments with realistic “production” data volumes • Perform a suitable stress test before moving into Production • Never assume that performance will be better in the Production environment!
(Production may have bigger servers, but there will be 100x more users and bigger data volumes)
• To minimise costs, it is vital to identify and deal with performance issues early on in the development cycle, rather than wait until UAT/Production
• Try to avoid having 1 week at the end of your development project set aside for “performance tuning”. Developers need to develop with performance in mind – tackle performance issues as and when they occur
© Peak Indicators Limited 12
Performance Tuning OBI 11g – All The Options
Oracle BI Optimisation
Oracle Engineered System
Optimise Database
Upgrade Database Hardware
Multi-Dimensional Cube Engine
Optimise Oracle BI
Addition DB Licensed Options
Optimise Data Model
© Peak Indicators Limited 13
Oracle BI Optimisation Overview
• With any BI implementation, most of the processing is done down at the database level, this means that quite often Oracle BI server will be sitting idle (waiting just like the end user is!)
• Oracle BI itself is configured by default for around 2000 total users per instance (catering for 100 concurrent sessions). These limits can be easily increased to scale for more users – but increasing them does not make the system run faster
• There is one major feature (BI Server “caching”) which can be used for optimisation, but the main focus should be on making sure Oracle BI is generating efficient queries and the dashboards/reports are not causing an unreasonable/unnecessary amount of load on the system
© Peak Indicators Limited 14
Oracle BI Optimisation General Recommendations
• When it comes to tuning the application itself, it is mostly down to being as reasonable/sensible as possible with the front end requirements to avoid performance issues:
• Keep the number of reports on a dashboard down to a minimum • Avoid having large pivot tables where possible • Encourage the use of drill-downs. So you drill-down from a summary report
(rather than just starting with a large detail report) • Minimise the Agents (schedules) being sent during working hours (especially
those which are delivered to many recipients – it is all done concurrently) • Minimise the columns left in the “Exclude” sections within Answers (if columns
aren’t needed then remove them completely from the report) • Make sure query filters make appropriate use of database indexes
© Peak Indicators Limited 15
Oracle BI Optimisation Avoid “Excluded” Columns
• A common mistake is to end up with lots more columns to the report than is actually required
• In the example below, the final report only shows 3 columns with 4 more being “excluded”
• The SQL queries will however involve all 7 columns • Recently we have found this type of issue to cause significant performance
degradation on customer sites ranging from 7 seconds to 3 minutes per report
© Peak Indicators Limited 16
Oracle BI Optimisation Efficient Query Filters
• It is also quite easy for report developers to write inefficient query filters • One common mistake is to apply functions on to columns. This is because applying a function on a
column will render any indexes on that column unusable • In the example below, the “TimeStampAdd” function was applied to the “Order Date” column
rather than to the variable the other side. The database could not use the index on the “Order Date” column so a full table scan was performed instead
• By correcting the logic the query ran 2x faster (7 seconds instead of 16)
TimeStampAdd(“Order Date”, 7) is greater than @{VAR_DATE} “Order Date” is greater than TimeStampAdd(@{VAR_DATE}, -7)
If you want to make use of an index, don’t apply a function
on the column
© Peak Indicators Limited 17
Oracle BI Optimisation BI Server Caching
• Caches the results of “Logical” SQL queries issued to the BI server
• BI Server will fetch results from the server cache if a user issues the same query or a sub-set of a previous query
• Cache is shared amongst users and across multiple BI servers
• Tuned for large volumes of “small” result sets (<10K records)
• Cache can be configured (number of files, total volume, max number of records, max file size etc)
WebLogic
BI Server
BI Presentation Services
© Peak Indicators Limited 18
Oracle BI Optimisation BI Server Caching • BI Server Caching can bring significant gains in reporting performance – queries
returned from BI Server cache are always instant
• Performance issues can sometimes be resolved by “pre-seeding” the BI Server cache with report results prior to the users running their reports
• But BI Server Caching is not your “silver bullet”! • Although it can provide valuable and cost-effective “quick-wins” to resolve
performance issues, it does have limitations • BI Server caching doe not replace the need for “longer-term” solutions (such as
enhancing the physical data-sources to be generally more efficient for reporting)
© Peak Indicators Limited 19
Oracle BI Optimisation Oracle BI Server Caching - Limitations
• To effectively tune a Dashboard Page using BI Server Caching, you have to build reports that will cache all the possible dashboard prompt selections and column selector combinations
• Otherwise performance will dive as soon as the user makes a new selection • Doing this for every dashboard page is not maintainable or scalable in the long term,
each result set could be massive! (well beyond the limits of what is supported)
© Peak Indicators Limited 20
Performance Tuning OBI 11g – All The Options
Database Optimisation
Oracle Engineered System
Optimise Database
Upgrade Database Hardware
Multi-Dimensional Cube Engine
Optimise Oracle BI
Addition DB Licensed Options
Optimise Data Model
© Peak Indicators Limited 21
Database Optimisation Summary
• On an Oracle database, there are many tuning methods which are commonly used on a Data Warehouse. For example:
• Gather statistics • Parallel Query • Star Transformation • Bitmap Indexes
• We will attempt to answer the following questions:
• Do they actually work? • What performance gains do they actually deliver? • In what situations are they most effective?
• Partitioning • Bitmap Join Indexes • Compression
NOTE: Items in red require Oracle
DB Enterprise Edition
Partitioning is an additional licensed option for DB Enterprise Edition
Oracle OLAP is discussed
later on
© Peak Indicators Limited 22
Database Optimisation
Performance Test
• A performance test was conducted based on the following: • Star-schema data-model consisting of 30 million fact records • 4 BI Dashboards with “Summary” reports (multiple years/months) • 2 BI Dashboards with “Drill-Down” reports (single month/organization etc)
• Database Server consisted of:
• 1 x dual core CPU • 1 x Hard Disk • 4 x GB RAM
NOTE: This is very lightweight hardware for a database, but it is useful for exaggerating performance issues.
It is not appropriate hardware for testing Parallel Query, so a proper
test for Parallel Query will be shown later
© Peak Indicators Limited 23
Database Optimisation
Overall Results
• With all these tuning options: • Summary reports are 2x faster (but still too slow!) • Detail reports are 27x faster
© Peak Indicators Limited 24
Database Optimisation
Another Test - Parallel Query
• Another test was performed to highlight the benefits of Parallel Query – especially when it comes to scalability:
• Database Server with 8 x CPU cores, 8 x HDD, 100GB RAM • Tests involved running queries processing 2.5K to 25Million records
Without parallel query enabled, scalability becomes an issue
Parallel Query scales better and is 3x faster when
processing 25 million rows Performance was comparable until you
process >250K records
© Peak Indicators Limited 25
Database Optimisation
Summary
© Peak Indicators Limited 26
Performance Tuning OBI 11g – All The Options
Data-Model Optimisation
Oracle Engineered System
Optimise Database
Upgrade Database Hardware
Multi-Dimensional Cube Engine
Optimise Oracle BI
Addition DB Licensed Options
Optimise Data Model
© Peak Indicators Limited 27
Data-Model Optimisation Overview
• We will now discuss two commonly used methods for improving the performance of database queries: • Removal of Snow-Flakes/Denormalisation • Aggregation
• The first one aims to improve performance by reducing the number of table joins
• Aggregation aims to improve performance by summarising data so there is less data process at report run-time
© Peak Indicators Limited 28
Data-Model Optimisation Removal of Snow-Flakes
• In our test data-model we have 3 snow-flakes, this is where the you get multiple dimension tables hanging off each other
• In the example above, if a SQL query need data from all tables then it would require 8 table joins….and this could hinder performance?
© Peak Indicators Limited 29
Data-Model Optimisation Removal of Snow-Flakes
• As a test, we merged the dimensions together so that now there were only 5 tables in total and a maximum of 4 table joins:
© Peak Indicators Limited 30
Data-Model Optimisation Removal of Snow-Flakes - Result
• The result was very surprising!
• By moving FKs on to the central fact table: • “Summary” queries increased from 58 to 60 seconds on average • “Drill-down” queries increased from 37 to 40 seconds on average
• By merging the tables together we are increasing data volumes (as the data is not as normalised). This increases the number of disk reads – which is our main bottleneck
• With “Star Transformation” and other database optimisations enabled maybe the sensible approach is to focus on reducing data volumes instead of reducing table joins?
© Peak Indicators Limited 31
Data-Model Optimisation Removal of Snow-Flakes – 2nd Approach
• A 2nd approach was tested. This time, instead of merging the Dimension tables together, we moved the FK columns in to the central fact table
• This meant all the necessary dimension columns were linked directly to the central fact
• In theory “Star Transformation” prefers to have pure star-schema diagrams like the one on the right-hand side below:
© Peak Indicators Limited 32
Data-Model Optimisation Removal of Snow-Flakes – 2nd Approach
• The result again was very surprising!
• By moving FKs on to the central fact table: • “Summary” queries increased from 58 to 137 seconds on average • “Drill-down” queries increased from 37 to 70 seconds on average
• So it appears that with modern database optimisation techniques such as “Star Transformation” and “Bitmap Indexes”, it is more important to keep your central fact table small to reduce the total number of physical disk reads (which is commonly the main bottleneck in a BI system)
• Lesson Learnt: Reducing the number of joins is not always effective if it means you have to increase the volume of data
© Peak Indicators Limited 33
Data-Model Optimisation Aggregation
• Earlier on we had managed to tune a set of reports using standard database features, but the response times of the “Summary” reports were still too high
• We will now try out the effectiveness of creating a set of summary tables (Materialized Views) and configure Oracle BI to make use them whenever it was appropriate to do so
By today’s standards, an
average response time of 36 seconds
is too high
© Peak Indicators Limited 34
Data-Model Optimisation Aggregation
• In our test environment, we have managed to build a single aggregate which can: • Reduce the number of records from 31M down to 8M • Support all 4 “Summary” dashboards • Provide at least 2 levels of drill-down across all the hierarchies
© Peak Indicators Limited 35
Data-Model Optimisation
Aggregation - Results • Finally, with just one aggregate all our
response times are now acceptable!
© Peak Indicators Limited 36
Database Optimisation
Summary so Far
© Peak Indicators Limited 37
Performance Tuning OBI 11g – All The Options
Multi-Dimensional Cubes
Oracle Engineered System
Optimise Database
Upgrade Database Hardware
Multi-Dimensional Cube Engine
Optimise Oracle BI
Addition DB Licensed Options
Optimise Data Model
© Peak Indicators Limited 38
Multi-Dimensional Cubes Overview
• Oracle has two well-established “cube” engines in its product line: • Oracle OLAP • Oracle Essbase
• Whilst they are architecturally and functionally quite different, they
both offer good potential for optimising your BI infrastructure
© Peak Indicators Limited 39
Multi-Dimensional Cubes When Do You Need a Cube Engine?
• Sometimes however, data volumes or reporting complexity are such that the level of “optimisation” required goes beyond manageable levels – you have too many relational aggregates:
Oracle BI
MVs
© Peak Indicators Limited 40
Multi-Dimensional Cubes When Do You Need a Cube Engine?
• A Cube Engine’s calculation performance is sometimes so superior you can sometimes replace the complexity with just a single Cube
Oracle BI
OLAP
Essbase
Oracle OLAP is the cube engine inside
the Oracle Database
Essbase is the cube engine used as the
primary data source for all Hyperion EPM applications. It sits
outside of the Oracle Database on its own
dedicated servers
© Peak Indicators Limited 41
Multi-Dimensional Cubes Cube vs Relational
• The benefits of Oracle OLAP and Oracle Essbase are clear to see in this performance chart
• The performance of OLAP/Essbase is far great than an equivalent set of relational MVs – especially as the number of concurrent queries grows
• Based on our own internal tests
• Useful for “Summary” reports only
© Peak Indicators Limited 42
Multi-Dimensional Cubes
Disadvantages
• There are some factors however that reduce the attractiveness of cube engines: • Storage required for cube engines can be far greater • Cube engines do not scale as well (large cubes can be harder to maintain than the equivalent
relational structures) • Typically, the main dimension hierarchy columns are included in the cubes, not the whole
range of dimension attributes • Not so useful for detail / drill-down queries • Data quality rules are stricter than is required for relational structures (so you may not
always be able to build the cubes you want without fixing data first) • Oracle BI does not support “all” features of the cube engines • Not all types of Oracle BI functions supported e.g. Time Series • If the cube structure changes, it has to be re-modelled from scratch (Oracle OLAP)
© Peak Indicators Limited 43
Multi-Dimensional Cubes
Unofficial Recommendation
• Based on the disadvantages and in light of other performance tuning techniques which are now available:
• If you are already licensed for Oracle Essbase or Oracle OLAP then it could be worth trying it out
BUT • If you do not have licenses purchased already then you may want to consider
other options first e.g. relational aggregation, upgrading db hardware or an Oracle Engineered System
© Peak Indicators Limited 44
Performance Tuning OBI 11g – All The Options
Upgrade Database Hardware
Oracle Engineered System
Optimise Database
Upgrade Database Hardware
Multi-Dimensional Cube Engine
Optimise Oracle BI
Addition DB Licensed Options
Optimise Data Model
© Peak Indicators Limited 45
Upgrade Database Hardware Overview
• If you fully understand your existing bottlenecks, then new hardware can provide significant improvements to system performance and scalability
• Remember – hardware costs are always on the decrease, so you might be surprised at what is available!
• “Commodity” servers with large amounts of memory (>100GB) are common nowadays. A lot of companies could easily fit their databases totally inside the DB Cache
• New database hardware can easily make 2-3x performance gains without having to reconfigure the application
© Peak Indicators Limited 46
Upgrade Database Hardware Possibilities and Benefits
Hardware Component
Benefits
Increase CPUs Greater parallel query capability Greater scalability to support more users (Might incur additional license cost)
Increase RAM Increasing memory available for the Database Cache will result in reduced disk reads – easy way to double your performance. For example: A 60 second query coming from disk reduced to 30 seconds when the data was totally cached
Increase Number of Hard Disks Spreading data over more disks increases disk throughput, reducing the bottleneck. Doubling the number of disks can reduce query times by 25-50%
Use Solid State Drives (SSD) Not as fast as RAM, but is cheaper SSDs are much faster than using Hard Disks. Either replace your Hard Disks completely with SSDs or use the Oracle Database “Smart Flash Cache” where data can be cached on SSD, as an extension to the DB Cache (NOTE: SFC available on linux/solaris only)
© Peak Indicators Limited 47
Upgrade Database Hardware But Remember!
• When judging a server’s performance, most people will look at the number of available CPUs (cores)
• But For a database server, memory and disk throughput is more important! There is no point having 16 CPUs if they have to sit idle waiting for the disks
• Also, disk capacity is always increasing, it leads people to buy fewer disks. The number of disks you have is more important than their individual capacity! Stripe data over as many disks as possible Buy as much memory as possible Make use of SSDs to give an extra performance boost (if your data volumes
exceed the total RAM available)
© Peak Indicators Limited 48
Performance Tuning OBI 11g – All The Options
Oracle Engineered Systems
Oracle Engineered System
Optimise Database
Upgrade Database Hardware
Multi-Dimensional Cube Engine
Optimise Oracle BI
Addition DB Licensed Options
Optimise Data Model
© Peak Indicators Limited 49
Oracle Engineered Systems For when only a Ferrari will do?
• Oracle provides its own Engineered Systems which adopt all the latest optimisation techniques e.g. in-memory, SSD, columnar compression
• Three relevant ones are:
• Oracle Exalytics • Oracle Database Appliance • Oracle Exadata
• Significant performance gains 20x, 40x, 50x faster etc
• They come with some features that you don’t get if you build servers yourself!
© Peak Indicators Limited 50
Oracle Engineered Systems Exalytics
One customer found some reports (especially with pivot
tables) ran 20% faster on Exalytics without any tuning or
configuration
© Peak Indicators Limited 51
Oracle TimesTen Customer Example
• With TimesTen 11.1.1.2 in-memory aggregates, each dashboard page was on average 26x faster compared to the Data Warehouse • 25 queries tested across 6 dashboard pages Aggregates reside in-memory
and on the BI Server: No disk bottlenecks
No network bottlenecks
© Peak Indicators Limited 52
Oracle Engineered Systems Oracle Database Appliance
• 2 x Oracle Linux server nodes • A total of:
• 24 x Intel Xeon CPU cores • 192 GB RAM • 12 TB of Hard Disk storage (20 x HDDs) • 946 GB of SSD storage for DB redo logs (4 x SSDs)
• Pay-as-you-grow pricing
• Hybrid Columnar Compression available for 10-15x compression
© Peak Indicators Limited 53
Oracle Engineered Systems Oracle Exadata X3-8
• 160 CPU cores • 4 TB memory • 22.4 TB of Smart Flash Cache (SSD) • 50 TB hard-disk storage • Hybrid Columnar Compression • 168 CPU cores at disk level – to process/filter data on the disks rather than
in the database cache
• Eight, Quarter and Half rack configurations available
© Peak Indicators Limited 54
Performance Tuning OBI 11g – All The Options
Real-Time Optimisation
© Peak Indicators Limited 55
Real-Time Optimisation Overview
• Using a BI tool to report against real-time (OLTP) data sources can provide significant performance headaches:
• Data structures are designed for individual insert/updates not large SQL queries • Queries often are complex, involving many tables • Outer-joins often required • Few options for improving performance (you can’t just enable Star Transformation!) • No scalability, query response times grow exponentially as data volumes increase:
OLTP Data Volumes
Query Response Time
© Peak Indicators Limited 56
Real-Time Optimisation The Solution
• Build an Operational Data Store (ODS) • Data extracted frequently from the OLTP • Data stored in ODS in a format suitable for reporting • ODS only stores the data needed for near-real-time reporting
OLTP OD S
Oracle BI 11g
© Peak Indicators Limited 57
Real-Time Optimisation The Solution
• Because an ODS only ever contains the data needed for “operational” reporting, the data volumes do not grow
• This means an ODS is 100% scalable in the long term • For example, consider a call centre:
• The number of “closed” Service Requests will grow over time • The number of “open” Service Requests will stay constant • For real-time reporting, the ODS only needs to contain “open” Service Requests
OLTP Data Volumes
ODS Data Volumes
© Peak Indicators Limited 58
Real-Time Optimisation Two Examples
• Financial News & Media organisation: • ODS implemented for “real-time” reporting against CRM system • ODS updated with 1 minute cycles • 8000% performance improvement across 450 real-time reports
• Global Corporate Finance organisation: • ODS implemented for “real-time” reporting for planning HR compensation • Previously users could not report directly off OLTP (queries had to run against
15+ tables and were too slow). This meant users could only generate reports once a day
• Data now available for reporting in ODS within 15 seconds, down from 24 hours
© Peak Indicators Limited 59
Performance Tuning OBI 11g – All The Options
In Summary
© Peak Indicators Limited 60
In Summary • There are a wide range of performance optimisation techniques available –
applicable to customers of all sizes and types
• The most effective tuning options are (in no particular order): • Data aggregation • Star Transformation • Parallel Query • Table Partitioning • Oracle Exalytics
• “in-memory” optimisation is becoming the trend. Examples are: • Oracle Exalytics 1 TB RAM • Oracle Exadata 4 TB RAM, 22 TB Flash Cache
Lessons learnt from these 5 options: 1) Summarise data 2) Query efficiently 3) Query in parallel 4) Quickly Exclude irrelevant data 5) Process more in-memory
© Peak Indicators Limited 61
Questions?
Helping Your Business Intelligence Journey