Using End-Of-Time Date Semantics to Improve Performance

1HS2 Solutions confidential and proprietary.

9 / 1 5 / 2 0 1 6

By Donald Bales, Rails Practice Lead

USING END-OF-TIME ACTIVE DATE SEMANTICS TO IMPROVE PERFORMANCE

Underlying almost every Rails application is a relational database management system. Let me show you how important it is to apply some fundamental time rules to your application's database in order to get the best response times possible.

ABSTRACT

For 20 years, we have helped some of the best brand and eCommerce companies leverage the internet and digital marketing.

We value smart engineering and team members that collaborate well internally as well as with our clients and their agencies.

Leveraging Technology & Intelligence to Drive Results

WHO WE ARE

HS2 HISTORY

1 9 9 4

Software DevelopmentHollyer & Schwartz (H&S)

1 9 9 9

eBusinessH&S acquired by XOR, Inc.

2 0 0 1

Precision Marketing

XOR merges with Seurot

2 0 0 3

HS2 FormedSeurat acquired by Fair Isaac. HS2

Solutions formed.

Hollyer & Schwartz was founded in 1994 as a software development and systems integration company. The core team has evolved and grown together for over 15 years into a full-service eBusiness and Precision Marketing company.

WHO WE WORK WITH

HS2 Solutions confidential and proprietary. 5

WHAT WE DO

ECOMMERCE, WEB & MOBILE DEVELOPMENT

ANALYTICS & INSIGHTSEXPERIENCE DESIGN (UX/UI)

INTERACTIVE MARKETING

“D O N A L D B A L E S

The obvious is always illusive

• There’s not a bit of Ruby or Rails code

• Fundamentals

• Structured Query Language (SQL)

• Data Definition Language (DDL)

• Data Manipulation Language (DML)

DISCLAIMER

PROGRAMMER

EXPLICIT CONTENTADVISORY

don=# create table test_integer (an_integer integer);

CREATE TABLE

don=# insert into test_integer (an_integer) values (1.5);

INSERT 0 1

don=# select * from test_integer;

an_integer

------------

WARM-UP EXERCISE QUESTION

don=# select * from test_integer;

an_integer

------------

(1 row)

WARM-UP EXERCISE ANSWER

where start_date <= CURRENT_DATE

and end_date >= CURRENT_DATE

and (end_date >= CURRENT_DATE or end_date is NULL)

WHAT’S THE DIFFERENCE BETWEEN THESE?

where CURRENT_DATE between start_date and end_date

and (end_date >= CURRENT_DATE or end_date is NULL)

WHAT’S THE DIFFERENCE BETWEEN THESE?

ANSWER: EFFICIENCY AND PERFORMANCE

In this context we are talking about determining if something is active at some point in time by comparing that point in time against an item's start and end dates.

So if I have a time line:

WHAT DOES IT MEAN TO BE ACTIVE?

Mar 31

Active Period Inactive Period

PointIn Time

We can say that at this point in time, that is, when the point in time is between the start and end dates, the item is active.

Mar 31

PointIn Time

Mar 31

PointIn Time

where CURRENT_DATE between start_date and end_dateIn SQL:

But what do we do if we don't know when an item will become inactive?

The typical and intuitive programming solution is not to specify an end date:

WHAT IF WE DON’T KNOW THE END DATE?

Active Period

PointIn Time

Name Null? Type

------------------------------- -------- ----------------------

ID NOT NULL NUMBER(38)

CODE NOT NULL VARCHAR2(30)

DESCRIPTION VARCHAR2(4000)

START_DATE NOT NULL DATE

END_DATE DATE

Active Period

PointIn Time

where start_date <= CURRENT_DATEand (end_date >= CURRENT_DATE or end_date is NULL)

In SQL:

But using this semantic for “active” is wholly inefficient for queries against a database. Is there a more efficient yet equivalent way to represent no end date?

Active Period

PointIn Time

Yes! We can substitute a code-able notion of the end of time: 12/31/9999

Now the item is still active, at this moment, and through the end-of-time, but it’s no longer NULL! And, that, makes all the difference.

USE A KNOWN VALUE TO REPRESENT THE END-OF-TIME

Active Period

PointIn Time

Dec 31, 9999

end-of-time

• How about December 31, 9999 or 12/31/9999

• It works for all these:• DB2

• MariaDB/MySQL

• Microsoft SQL server

• Oracle

• PostgreSQL

• Sysbase

DEFINE AN END-OF-TIME FOR UNKNOWN END DATES

Name Null? Type

------------------------------- -------- ----------------------

ID NOT NULL NUMBER(38)

CODE NOT NULL VARCHAR2(30)

DESCRIPTION VARCHAR2(4000)

START_DATE NOT NULL DATE

END_DATE NOT NULL DATE

Active Period

PointIn Time

Dec 31, 9999

end-of-time

In SQL:

In the future, when someone wants to make the item truly inactive, they update the end date to a non-end-of-time value:

Active Period

PointIn Time

Dec 31, 9999

end-of-time

Mar 31

Inactive Period

Using end-of-time semantics for end date instead of no end date is extremely important if one is concerned about efficiency and performance.

That's what we will discuss in the remainder of this presentation.

WANT EFFICIENCY AND PERFORMANCE?

LET’S REVIEW

Determining if an entry is active is done by testing if the start date is in the past or the current moment and the end date, if it exists, is in the current moment or the future, and if it does not exist, that it is NULL. In SQL:

where CURRENT_DATE >= start_date

and (CURRENT_DATE <= end_date or end_date is NULL)

WITH A NULLABLE END DATE

This is not an optimal situation, because a NULL value cannot be indexed, and accordingly, the database will have to do a full table scan, or index scan against the start date if it is indexed.

WITH A NULLABLE END DATE

An optimal way to state that something is active is to populate both the start and end date. By setting the end date to the end of code-able time, say 12/31/9999, determining if an entry is active is done by testing if the start date is in the past or the current moment and the end date is in the current moment or the future. In SQL:

where CURRENT_DATE >= start_date

and CURRENT_DATE <= end_date

AN OPTIMAL APPROACH

Another way to write it in SQL:

AN OPTIMAL APPROACH

• NULL values typically can’t be indexed

• This leads to full table scans or

• This leads to full index scans or

• This leads to partial index scans with partial table scans

• Full and partial table scans flush the database’s buffers (cache)

• Un-necessary work consumes capacity that may be needed elsewhere

• As table and index size grow, queries slow

WHY ARE NULL VALUES A PROBLEM?

• An example using a dictionary• Find by Definition (full scan) – “free from legal or discriminatory restrictions”

• Start at the first page

• Read definitions

• Compare

• Match? You’ve found it, so you’re done, else go to next page…

FULL TABLE SCANS

Found It!

• Find by Word (index scan) - “source”

• Start anywhere

• Read word

• Compare

• Match? You’ve found it! You’re done

• Less than, jump forward and compare again

• Greater than, jump backward and compare again

INDEX SCANS

Found It!

PERFORMANCE AS A FACTOR OF SIZE

- 5,000,000 10,000,000 -

5,000 10,000 15,000 20,000 25,000 30,000 35,000

FULL TABLE SCAN

Full Scan

“As tables grow, queries slow.”

milliseconds

PERFORMANCE AS A FACTOR OF ORDER

- 5,000,000 10,000,000 -

5,000 10,000 15,000 20,000 25,000 30,000 35,000

UNCACHED INDEX RANGE SCANS

Full ScanStart-End UncachedStart-Null End Uncached

Look at how poorly that null-able end

date performs

That is, using memory I/O instead of physical disk I/O

• NOTE: Full table scans flush cache, making it useless

PERFORMANCE AS A FACTOR OF CACHE

- 5,000,000 10,000,000 -

5,000 10,000 15,000 20,000 25,000 30,000 35,000

CACHED INDEX RANGE SCANS

Full ScanStart-End UncachedStart-Null End UncachedStart-End CachedStart-Null End Cached

A full table scan is faster than that

null-able end date

LET’S TAKE A CLOSER LOOK

PERFORMANCE AS A FACTOR OF SIZE

- 200 400 600 800 1,000 - 1 2 3 4 5 6 7

FULL TABLE SCAN

Full Scan

PERFORMANCE AS A FACTOR OF ORDER

- 200 400 600 800 1,000 - 1 2 3 4 5 6 7

UNCACHED INDEX RANGE SCANS

Full ScanStart-End UncachedStart-Null End Uncached

- 200 400 600 800 1,000 - 1 2 3 4 5 6 7

AN EVEN CLOSER LOOK!

- 10 20 30 40 50 60 70 80 90 100 - 1 2 3 4 5 6 7

- 5 10 15 20 25 - 1 2 3 4 5 6 7

PERFORMANCE CHARACTERISTICS AND TIME

• In the beginning• Database is small

• Tables are small

• Indexes are small

• Temporal width is small

• Queries are fast

PERFORMANCE CHARACTERISTICS CHANGE OVER TIME

“YO U R A P P L I C AT I O N U S E R S

Boy! This application is great!

• As time goes by• Database gets larger

• Tables get larger

• Indexes get larger

• Temporal width gets larger

• Queries take longer

PERFORMANCE CHARACTERISTICS CHANGE OVER TIME

“A N O N Y M O U S

*Sigh* This application sucks!

• Now that you have index-able data, you need optimal indexes• Indexes speed up queries

• Always on the Primary Key

• Almost always on Foreign Keys

• As needed on temporal columns

• You can’t index every column• Indexes slow down inserts and affected updates

• An indexes value is proportional to its selectivity

MORE ON THAT OPTIMAL APPROACH

• start_date, end_date? – the intuitive choice!

• end_date, start_date?

• How much history will you keep around?• A whole-heck-of-a-lot? Then end_date, start_date

• Only Recent? Then start_date, end_date

SELECTIVITY

- 5,000,000 10,000,000 -

5,000 10,000 15,000 20,000 25,000 30,000 35,000

Full ScanStart-End CachedStart-Null End CachedEnd-Start CachedNull End-Start Cached

The combination of start date and null end date is off the chart!

1,000,000 5,500,000 10,000,000 -

300 CACHED INDEX RANGE SCANS

The combination of end date and start date performs the best!

NULL END DATE, START DATE

-------------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | -------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | 16 | 10104 (1)| 00:00:01 | | 1 | SORT AGGREGATE | | 1 | 16 | | | | 2 | CONCATENATION | | | | | | |* 3 | INDEX RANGE SCAN| T10_000_000_NIA | 487K| 7621K| 3607 (1)| 00:00:01 | |* 4 | INDEX RANGE SCAN| T10_000_000_NIA | 849K| 12M| 6497 (1)| 00:00:01 | --------------------------------------------------------------------------------------

END-OF-TIME END DATE, START DATE

------------------------------------------------------------------------------------ | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | ------------------------------------------------------------------------------------ | 0 | SELECT STATEMENT | | 1 | 16 | 5224 (1)| 00:00:01 | | 1 | SORT AGGREGATE | | 1 | 16 | | | |* 2 | INDEX RANGE SCAN| T10_000_000_IA | 1413K| 21M| 5224 (1)| 00:00:01 | ------------------------------------------------------------------------------------

WHAT’S HAPPENING HERE (AT TEN MILLION ROWS)?

end-of-time is ½ the cost

- 200 400 600 800 1,000 - 1 2 3 4 5 6 7

--------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | --------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | 16 | 4 (0)| 00:00:01 | | 1 | SORT AGGREGATE | | 1 | 16 | | | | 2 | CONCATENATION | | | | | | |* 3 | INDEX RANGE SCAN| T1_000_NIA | 45 | 720 | 2 (0)| 00:00:01 | |* 4 | INDEX RANGE SCAN| T1_000_NIA | 85 | 1360 | 2 (0)| 00:00:01 | ---------------------------------------------------------------------------------

------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | ------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | 16 | 2 (0)| 00:00:01 | | 1 | SORT AGGREGATE | | 1 | 16 | | | |* 2 | INDEX RANGE SCAN| T1_000_IA | 134 | 2144 | 2 (0)| 00:00:01 | -------------------------------------------------------------------------------

WHAT’S HAPPENING AT ONE THOUSAND ROWS?

end-of-time is still ½ the cost

- 10 20 30 40 50 60 70 80 90 100 - 1 2 3 4 5 6 7

----------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | ----------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | 16 | 1 (0)| 00:00:01 | | 1 | SORT AGGREGATE | | 1 | 16 | | | |* 2 | INDEX SKIP SCAN| T100_NIA | 11 | 176 | 1 (0)| 00:00:01 | -----------------------------------------------------------------------------

----------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | ----------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | 16 | 1 (0)| 00:00:01 | | 1 | SORT AGGREGATE | | 1 | 16 | | | |* 2 | INDEX RANGE SCAN| T100_IA | 11 | 176 | 1 (0)| 00:00:01 | -----------------------------------------------------------------------------

WHAT’S HAPPENING AT ONE HUNDRED ROWS?

end-of-time is the same cost

- 1 2 3 4 5 6 7 8 9 10 -

USING COST INSTEAD OF ELAPSED TIME

A LOOK AT EXPLAIN PLAN COSTS

- 5,000,000 10,000,000 -

10,000 20,000 30,000 40,000 50,000 60,000 70,000 80,000

Full ScanStart-Null EndEnd-Start

A LOOK BACK AT ELAPSED TIMES

1,000,000 5,500,000 10,000,000 -

300 CACHED INDEX RANGE SCANS

Full ScanStart-EndStart-Null EndEnd-StartNull End-Start

The combination of end date and start date performs the best!

START DATE, NULL END DATE

------------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | ------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | 16 | 78992 (1)| 00:00:04 | | 1 | SORT AGGREGATE | | 1 | 16 | | | |* 2 | INDEX RANGE SCAN| T10_000_000_ANI | 1337K| 20M| 78992 (1)| 00:00:04 | -------------------------------------------------------------------------------------

------------------------------------------------------------------------------------ | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | ------------------------------------------------------------------------------------ | 0 | SELECT STATEMENT | | 1 | 16 | 5224 (1)| 00:00:01 | | 1 | SORT AGGREGATE | | 1 | 16 | | | |* 2 | INDEX RANGE SCAN| T10_000_000_IA | 1413K| 21M| 5224 (1)| 00:00:01 | ------------------------------------------------------------------------------------

WHAT’S HAPPENING HERE (AT TEN MILLION ROWS)?

end-of-time is 15 TIMES less costly!

-------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | -------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | 16 | 10 (0)| 00:00:01 | | 1 | SORT AGGREGATE | | 1 | 16 | | | |* 2 | INDEX RANGE SCAN| T1_000_ANI | 130 | 2080 | 10 (0)| 00:00:01 | --------------------------------------------------------------------------------

------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | ------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | 16 | 2 (0)| 00:00:01 | | 1 | SORT AGGREGATE | | 1 | 16 | | | |* 2 | INDEX RANGE SCAN| T1_000_IA | 134 | 2144 | 2 (0)| 00:00:01 | -------------------------------------------------------------------------------

WHAT’S HAPPENING AT ONE THOUSAND ROWS?

end-of-time is 5 TIMES less costly

------------------------------------------------------------------------------ | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | ------------------------------------------------------------------------------ | 0 | SELECT STATEMENT | | 1 | 16 | 1 (0)| 00:00:01 | | 1 | SORT AGGREGATE | | 1 | 16 | | | |* 2 | INDEX RANGE SCAN| T100_ANI | 11 | 176 | 1 (0)| 00:00:01 | ------------------------------------------------------------------------------

----------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | ----------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | 1 | 16 | 1 (0)| 00:00:01 | | 1 | SORT AGGREGATE | | 1 | 16 | | | |* 2 | INDEX RANGE SCAN| T100_IA | 11 | 176 | 1 (0)| 00:00:01 | -----------------------------------------------------------------------------

WHAT’S HAPPENING AT ONE HUNDRED ROWS?

end-of-time is the same cost

• Make end dates not null

• Define an end-of-time value to use everywhere

• Use the end-of-time value when you don’t know the end date

• Index in end date, start date order

• Eliminate full table scans

• Performance tests under load

• Performance tests with “real-life” data and table sizes

• Retest periodically

CONCLUSIONS

• Move non-transactional data to another database

• Brush your teeth twice a day, and don’t forget to floss

• Listen to your spouse, I didn’t say obey did I?

OTHER GOOD HABITS

The use of null values in a column that is queried often is a common performance problem that Oracle addressed years ago by allowing the creation of functional indexes. Using a functional index, one can work-around the null values by having the database calculate a replacement value using the nvl() function and storing the calculated value in the index instead. This means that every query against the table must use the same replacement value syntax if it is going to take advantage of the functional index. One caveat, you must use HINTS.

AN ORACLE-ENABLED WORKAROUND

• You can use hints /*+ INDEX() */

• You can pin tables in cache (don’t do this with large tables)

ANOTHER ORACLE ENABLED WORKAROUNDS

NULL VALUES ARE THE ZERO OF THE 21ST CENTURY

Known Unknown

Unknowable

NULL Values

CONTACT:Phone: (773) 296-2600

Email: don.bales@hs2solutions.com

THANK YOU!

Using End-Of-Time Date Semantics to Improve Performance

Documents

Improve Your Front-End Project Workflow With Grunt

Lexical Semantics Computational Semantics

LT-TCP: End-to-End Framework to Improve TCP Performance over

Improve business insights & decision making with - …Improve business insights & decision making with is a comprehensive real time system and a seamlessly integrated end to end E

Understanding roles - working together to improve end of ... · Understanding roles: working together to improve end of life care Introduction It is only by understanding the roles,

Commissioning end of life care to improve patient outcomes

02 - Syntax and semantics - uni-bonn.de · Prolog Syntax and Semantics Syntax Model-theoretic semantics (“Logical Consequence”) Operational semantics (Operational semantics (

Using semantics to improve interactive information access

Comp 205: Comparative Programming Languages Semantics of Imperative Programming Languages denotational semantics operational semantics logical semantics

Learn to improve Period-End Close Process with effective ...fulcrumway.com/images/resources/Learn to improve Period-End Close... · Learn to improve Period-End Close Process with

Dynamic semantics, discourse semantics and continuations › SPogodalla › files › cauld › documents › publ… · Dynamic semantics, discourse semantics and continuations Nicholas

Improve Front End Performance (How to Make More Quality Cans) · Improve Front End Performance. About IPS Integrated Packaging Solutions Services •Equipment Support •Training

Cognitive Semantics and Structural Semantics

Semiconductors: EDA Front end design drastically improve

Integrated Strategies to Improve the Quality of Wheat End-Products… · 2019-11-12 · Integrated strategies to improve the quality of wheat end-products: optimization of processes

End-to-End Optimization of Scene Layout3dsln.csail.mit.edu/papers/3dsln_cvpr.pdfa depth and a semantics map, the differentiable render- ... coder described above, we optimize L

345. structural semantics and cognitive semantics

Evasion, Traffic Normalization, and End-to-End Protocol Semantics

Using Semantics to Improve Corporate Online Communities

Operational Semantics - USTCstaff.ustc.edu.cn/~xyfeng/teaching/TOPL/lectureNotes/06...Operational semantics Small-step operational semantics Structural operational semantics (SOS)