Oracle result cache highload 2017

100500 способовкэширования в Oracle Database или как достичьмаксимальной скоростиобработки запросовминимальной ценой

Токарев Александр

DataArt

Agenda• Database caches

• Result cache

• Result cache in DBMSs different from Oracle

• Hand-made Oracle result cache implementation

• Embedded Oracle result cache implementation

• Performance tests

• Limitations and caveats

• Cases

• Conclusion

Agenda• Retailer case

• Finance case

• Limitations and caveats

• Loyalty case

• Conclusion

Database caches

• Buffer cache – cache for data pages/data blocks

• Statement cache – cache of queries plan

• Result cache – rows from queries

• OS cache

Retailer case

DWH report

Oracle 1120 Tb300 users

20 min350 distinct SKU5000 rows

Select sku_id,Shop_id,sku_detail(sku_id),…..

from dim_saleswhere ….Order by shop_id……..

Create or replace function sku_detail(sku_idnumber) return number isSelect 1If Select 2Else Select 3………Select 30End;

400 lines of SQL+PL/SQL

0.2 second per SKU

5000 * 0.2 = 1000 seconds

Retailer case Hand-made cache

DWHreport

Oracle 1120 Tb300 users

4 min350 distinct SKU5000 rows

Select sku_id,Shop_id,sku_detail(sku_id),…..

from dim_saleswhere ….Order by shop_id……..

Create or replace function

sku_full(sku_id number)

return number isSelect 1If Select 2Else Select 3………Select 30End;

400 lines of SQL+PL/SQL

0.2 second per SKU

350* 0.2 = 70 secondsCREATE PACKAGE BODY cache_sku AS

TYPE sku_cache_aat IS TABLE OF number INDEX BY PLS_INTEGER;

cache sku_cache_aat;

end cache_sku;

cache

FUNCTION sku_detail(skunumber) RETURN number ISBEGINIF NOT cache.EXISTS(sku) THEN

cache(sku) := sku_full(sku);END IF;RETURN cache(sku);

END sku_detail;

Hand-made cache Memory

12X

Hand-made cachePros:- Very fast- Easy to implement- No configuration efforts- No intra-process sync logic burden

Cons:- Cache consumes expensive memory from DB- Memory is allocated per-session basis- PL/SQL or other DB stored logic is required- Vendor specific- No automatic invalidation

Hand-made cache Not OracleNo cache

Hand-made cache Not OracleWith propopulated cache

Case 2 Recommendation engine

Oracle main

OracleDG

Application server

Application server

Load balancer

Client browser

4000users10000 requests per second

In-memory cluster

Case 2 Recommendation engine

Recommendation rules

1. 10 best recommendations by text match 2. Multilanguage capabilities3. Should be taken from 12 previous recognized documents of the client4. If there is no documents – from all clients of the same industry5. If no in same industry – from clients similar by margin and e.t.c

max 100 rows

2-3 columns

max 100 users

Case 2 Recommendation engine

1 week before the Release

1. Recommendations work slow – 5 minutes for 1 document

2. Code freeze

Case 2 Recommendation engine

Solution

1. Use database to cache queries

2. Use Oracle Database Result Cache

Why

1. SQL to get recommendation works 0.5 sec, no options for query tuning – Oracle full text search engine + it is really heavy SQL

2. Same parameters appear at least 5-10 times – cache will be used

3. Data to get recommendations is refreshed 1 hour basis

4. PL/SQL is prohibited

Oracle Result Cache

Oracle result cache

1. Memory area to share query result sets

2. Read consistent – auto invalidation during DML

3. Automatic dependency tracking

4. Minimal changes in the application

5. There is an option how not to change application

6. Could cache PL/SQL logic as well

Oracle Result Cache Way 1

ID of result

First execution Second execution

Oracle Result Cache Table annotations

No query changes but RC works

Oracle Result Cache Table annotations

No annotation on JOBS table – no result cache

Oracle Result Cache Table annotations

2 annotations –result cache works

Oracle Result Cache Dependency TrackingNo in dependency list

Oracle Result Cache Dependency Tracking

Oracle Result Cache Inline Views

Oracle Result Cache Inline Views

Oracle Result Cache Simple Views

Oracle Result Cache Invalidation

Oracle Result Cache Invalidation

Cache is ignored for current session

Good for others sessions

Oracle Result Cache Invalidation

Invalidated after commit for others sessions!

Oracle Result Cache Invalidation

Unexpected cache invalidation

1. SELECT FOR UPDATE statement even there were no changes at all

2. an unindexed foreign key + delete/update/insert a record from parent table

3. Update/delete statements in main table with no records affected + an update to any table where rows were affected.

P.S. Result Cache doesn’t track partitions even if a result cache query works with only 1 partition. Table level tracking always.

Case 2 Recommendation engine

Final solution

1. Do not use annotations – not all queries should be cached

2. Use /*+ result_cache*/ for long-running query

3. Performance is tested. Document recognition –30 seconds.

Time for production

Case 2 Recommendation engine Early morning

Level 3 support

Production incident

Severity 1

Users can’t provide document recognition. Recognition takes 20 minutes

at least. Sessions hangs.

Regards, L2 support team.

Case 2 Recommendation engine

• Active user count: 400

• Database active session count: 1200 = 400* 3

• Row count: 500

• Columns count: 5-8

X5 more!

Monitoring features

View Name DescriptionV$RESULT_CACHE_STATISTICS Lists cache settings and memory usage statisticsV$RESULT_CACHE_MEMORY Lists all the memory blocks and corresponding

statisticsV$RESULT_CACHE_OBJECTS Lists all the objects(cached results and

dependencies) along with their attributesV$RESULT_CACHE_DEPENDENCY Lists the dependency details between the cached

results and dependenciesV$SQLAREA Lists SQL statements issued inside Oracle database

Management features

Procedure Name DescriptionBYPASS Instruction to ignore result cache: for current

session or for all DBFLUSH Clean cacheMEMORY_REPORT Memory detail report

STATUS Checks the status

INVALIDATE Invalidates the specified result-set object

Package: DBMS_RESULT_CACHE

V$RESULT_CACHE_MEMORY

V$RESULT_CACHE_OBJECTS

V$RESULT_CACHE_STATISTICS

DBMS_RESULT_CACHE.MEMORY_REPORT

Locks Result cache prior 12.2 – use very careful!!!

Case 2 Recommendation engine Investigations

Strange queries for 40 small tables each minute:

ETL

Case 2 Recommendation engine Investigations

Result cache annotation

Still 20 minutes per document

Case 2 Recommendation engine

We have received very positive feedback about Oracle Adaptive Statistic feature from customer with respect to adaptive plans. It has proved to be very able at improving system performance for a huge range of workloads. (c) Oracle

20000 queries

10 minutes per document!!!

Via bug? WTF!!!

Result cache latches

Latches are Oracle-internal low-level locks that protect the memory structures of the system global area (SGA) against simultaneous accesses.

Result cache latches

Result cache latches Type 1When setsFirst row of dataset is placed in Result Cache

When releaseLast row of dataset is placed in Result Cache

Who waits

Sessions with same SQL which requested the latch

How much

_RESULT_CACHE_TIMEOUT – 10 seconds. Next - result cache bypassed.

Result cache latches Type 2When setsFirst row of dataset is requested from Result Cache

When releaseLast row of dataset is read from Result Cache

Who waits

Sessions with same SQL which requested the latch

How much

It depends

Result cache latchesLatches not only makes SQL to wait but consumes CPU.

There is no options to get rid of result cache latches – slow for concurrent environment..

Be ready to convince DBA latches wait time saves DB time.

Result cache statistics

NAME VALUEBlock Size (Bytes) 1024Block Count Maximum 4096Block Count Current 4096Result Size Maximum (Blocks) 204Create Count Success 500

Create Count Failure 0Find Count 20000Invalidation Count 10000Delete Count Invalid 155Delete Count Valid 14000Hash Chain Length 1Find Copy Count 1770Latch (Share) 0

They are equal – the cache is full!!!

Proper results are deleted

Memory estimate

Result Cache Size = row width (bytes)* expected row count

NAME VALUEBlock Size (Bytes) 1024Block Count Maximum 4096Block Count Current 4096

Memory allocated by blocks!!!

Result Cache Size = block size (if fits in 1024) * expected row count

Administration23 undocumented parameters6 documented parameters

AdministrationParameter Purpose

RESULT_CACHE_MAX_SIZE memory allocated to the server result cache in bytes. default – 0 bytes

RESULT_CACHE_MAX_RESULT maximum amount of server result cache memory (in percent) that can be used for a single result. The default value is 5%.

RESULT_CACHE_MODE Default is MANUAL which means that the cache should be requested explicitly via the RESULT_CACHE hint

_RESULT_CACHE_TIMEOUT (undocumented)

Maximum time a session request for a latch. Default 10 sec.

6 minutes per document!!!

Case 2 Recommendation engineNAME VALUEBlock Size (Bytes) 1024Block Count Maximum 4096Block Count Current 4096Result Size Maximum (Blocks) 204Create Count Success 500

Create Count Failure 0Find Count 20000Invalidation Count 10000Delete Count Invalid 155Delete Count Valid 14000Hash Chain Length 1Find Copy Count 1770Latch (Share) 0

A lot of updates on source tables

Case 2 Recommendation engine

Final statistics for result cache

40 seconds per document!!!

NAME VALUEBlock Size (Bytes) 1024Block Count Maximum 4096Block Count Current 4096Result Size Maximum (Blocks) 204Create Count Success 500

Create Count Failure 0Find Count 20000Invalidation Count 10000Delete Count Invalid 155Delete Count Valid 14000Hash Chain Length 1Find Copy Count 1770Latch (Share) 0

NAME VALUEBlock Size (Bytes) 1024Block Count Maximum 8192Block Count Current 6000Result Size Maximum (Blocks) 204Create Count Success 1000

Create Count Failure 0Find Count 20000Invalidation Count 30Delete Count Invalid 155Delete Count Valid 0Hash Chain Length 1Find Copy Count 1770Latch (Share) 0

Case 2 Recommendation engine Auto-expiring

SHELFLIFE = read-consistent result life time in seconds

SNAPSHOT = NON-read-consistent result life time in seconds

Restrictions• Dictionary tables/views (sys. schema)

• Temporary and external tables

• Sequences (nextval and curval columns)

• Non-deterministic SQL functions:

current_date, current_time, local_timestamp, sys_guid…

• Non-deterministic PL/SQL function: dbms_random, hand-written, …

• Pipelined functions (returning rowsets)

• Only IN parameter with simple data types: no CLOB, BLOB, records, objects, collections, ref cursors

• The same for return result

Result cache inside OracleWhere in Oracle

Jobs related stuffSELECT /*+ NO_STATEMENT_QUEUING RESULT_CACHE (SYSOBJ=TRUE) */

OBJ#,SCHEDULE_LMT,PRIO,JOB_WEIGHT FROM "SYS"."SCHEDULER$_PROGRAM" WHERE bla-bla-bla

APEXSELECT /*+result_cache*/ NAME, VALUE FROM WWV_FLOW_PLATFORM_PREFS

WHERE NAME IN ( 'QOS_MAX_WORKSPACE_REQUESTS', 'QOS_MAX_SESSION_REQUESTS', bla-bla-bla

select *

from v$sqlarea

where upper(sql_fulltext) like

'%RESULT_CACHE%‘

Result cache inside OracleDynamic samplingSELECT /* OPT_DYN_SAMP */ /*+ ALL_ROWS RESULT_CACHE(SNAPSHOT=3600)

opt_param('parallel_execution_enabled', 'false') NO_PARALLEL(SAMPLESUB)

NO_PARALLEL_INDEX(SAMPLESUB) NO_SQL_TUNE */ NVL(SUM(C1),:"SYS_B_0"),

NVL(SUM(C2),:"SYS_B_1") FROM (bla-bla-bla

Adaptive statisticSELECT /* DS_SVC */ /*+ dynamic_sampling(0) no_sql_tune no_monitoring

optimizer_features_enable(default) no_parallel RESULT_CACHE(SNAPSHOT=3600) */

NVL(SUM(C1),0) FROM (SELECT /*+ qb_name("innerQuery") NO_INDEX_FFS bla-bla-bla

Oracle internals for result cache

1

2

3

Database result cache pros&cons

Pros:- Minimal or no intervention at all into application code- No DB stored logic required- Read consistency - Fast in certain scenarios

Cons:- Cache consumes expensive memory from database- Should be properly set up- Sometimes could lead even to performance degradation- Vendor specific

We are not alone

1. Не расчитан размер кэша Troubleshooting Latch Free (Result Cache: RC Latch) Issues When The Result Cache is Full (Doc ID 2143739.1)

2. Блокировки Patch 14665745: DBMS_RESULT_CACHE.MEMORY_REPORT LOCKS OUT WRITERS TO THE RESULT CACHE

Bug 19846066 : LATCH FREE IN RESULT CACHE WHEN QUERYING V$RESULT_CACHE_OBJECTSPatch 14665745: DBMS_RESULT_CACHE.MEMORY_REPORT LOCKS OUT WRITERS TO THE RESULT CACHE

We are not alone

Result_cache_max_size /*+ result_cache*/ removed ordbms_result_cache.add_to_black_list or/*+ no_result_cache*/

We are not alone: Lessons learned

Best approach to roll out updates:

1. Adjust result cache memory

2. Disable cache before bulk loading

dbms_result_cache.bypass;

data ingestion scripts;

Issue dbms_result_cache.bypass(false);

Client side result cache

DBClient cache is ON

Client driver2. Configuration message

Connection thread 1

Connection thread 2

Result cache

3. SQL

Statistics messages

1. connect

1. connect 3. Cached SQL 1

4. Results

4. Results

5. Cached SQL 1

6. Results

CACHE SIZE

Client side result cache Invalidation Case 1

DBClient cache is ON

Client driver

Connection thread 1

Result cache

1. non-cached SQL

2. Invalid resultset list

2. Results

t last cached SQL 1 < Invalidation lag

Invalidation rules = Invalidation rules for Server Side Result Cache

Invalidation

Client side result cache Invalidation Case 1

DBClient cache is ON

Client driver

Result cache

1. Get invalid result set list

2. Invalid result set list

Current time = t Invalidation message + Invalidation lag

Invalidation rules = Invalidation rules for Server Side Result Cache

Invalidation

Client side result cache Configuration

Parameter Purpose

CLIENT_RESULT_CACHE_LAG maximum time in milliseconds that client result cache can lag behind changes in the database that affect its result sets. Default 3000 milliseconds

CLIENT_RESULT_CACHE_SIZE the maximum size of the client result set cache for each client process. Default 0 – not active, min - 32KB, max – 2G

Client side result cache

Client side result cache

Client side result cache

NAME VALUEBlock Size (Bytes) 256Block Count Maximum 256

Block Count Current 3

Create Count Success 1

Create Count Failure 0Find Count 9Invalidation Count 0Delete Count Invalid 0Delete Count Valid 0

= 10

Client side Result cache pros&consPros:

- Cheap client memory

- JDBC and .NET drivers

- Minimal or no intervention at all into application code

- Significant CPU, I/O, network roundtrip reduction

- No extra caching layer/API is required

- No latches

Cons:

- Eventual read consistency with delay

- Oracle OCI client should be installed

- Vendor specific

- 2 Gb per client limitation

- Not enough information about production

Hand-made cache bad scenario• Cache invalidation in case of data changes is a must

• Database stored logic isn’t in favor

• There is strong database developers team

• PL/SQL business logic is already in place

• There are limitations which don’t permit others caching techniques

Hand-made cache good scenario

Server side Result cache bad scenario• SQL populates a large number of distinct result sets

• SQL statement takes more than _RESULT_CACHE_TIMEOUT

• Cached results are requested very often from many sessions

Result cache good scenario• Queries have a limited number of possible result sets

• Result sets are relatively small (200-300 rows)

• SQL statements are relatively expensive

• Queries run against relatively static tables

• There is a strong DBA

Client side Result cache bad scenario• Instant cache invalidation in case of data changes is a must

• Thin drivers are required

• There is fine middle-tier developers team

• Middle tier uses a lot of SQL without any caching layer

• There are DB server hardware limitations

Hand-made cache good scenario

Conclusion1. Estimate memory size properly:

volume (Mb) = (

number of result rows * block size+

avg number of apex usage +

avg number adaptive statistic usage

)/1024

2. Add auto-cleaning capabilities with (snapshot + shelflife) options

3. Bypass the cache during bulk data changes

4. Adjust _result_cache_timeout to expected queries duration

5. Never use FORCE mode for all database

6. Check does FORCE used as expected in table annotations

7. Decide about adaptive statistics: _optimizer_ads_use_result_cache = false

Q&A

Alexander Tokarev

Database expert

DataArt

Alexander.Tokarev@dataart.com

THANK YOU.WE ARE HIRING!