How PostgreSQL’s SQL dialect stays ahead of its competitors · GROUPING SETS are multiple GROUP...

How PostgreSQL’sSQL dialect stays aheadof its competitors

@MarkusWinand

I’m sorry!

I couldn’t make it to PgConf.EU in 2014!

Don’t repeat my mistake.

Come toPgConf.EU 2018

in Lisbon!Oct 23-26.

How PostgreSQL’sSQL dialect stays aheadof its competitors

@MarkusWinand

PostgreSQL 9.5 2016-01-07

GROUPINGSETS

Only one GROUPBY operation at a time:

GROUPINGSETS Before SQL:1999

SELECTyear,month,sum(revenue)FROMtblGROUPBYyear,month

Monthly revenue Yearly revenue

SELECTyear,sum(revenue)FROMtblGROUPBYyear

GROUPINGSETS Before SQL:1999SELECTyear,month,sum(revenue)FROMtblGROUPBYyear,month

UNIONALL

GROUPINGSETS Since SQL:1999SELECTyear,month,sum(revenue)FROMtblGROUPBYyear,month

UNIONALL

SELECTyear,month,sum(revenue)FROMtblGROUPBYGROUPINGSETS((year,month),(year))

GROUPINGSETS are multiple GROUPBYs in one go

() (empty parenthesis) build a group over all rows

GROUPING (function) disambiguates the meaning of NULL(was the grouped data NULL or is this column not currently grouped?)

Permutations can be created using ROLLUP and CUBE(ROLLUP(a,b,c) = GROUPINGSETS((a,b,c),(a,b),(a),())

GROUPINGSETS In a Nutshell

GROUPINGSETS Availability19

5.1[0] MariaDB5.0[1] MySQL

9.5 PostgreSQLSQLite

5 DB2 LUW9iR1 Oracle

2008 SQL Server[0]Only ROLLUP (properitery syntax).[1]Only ROLLUP (properitery syntax). GROUPING function since MySQL 8.0.

TABLESAMPLE

TABLESAMPLE Since SQL:2003

SELECT*FROMtblTABLESAMPLEsystem(10)REPEATABLE(0)

bernoulli(10)better sampling,

way slower.

User provided seed

TABLESAMPLE Since SQL:2003

SELECT*FROMtblTABLESAMPLEsystem(10)REPEATABLE(0)

bernoulli(10)better sampling,

way slower.

User provided seed

TABLESAMPLE Availability19

5.1 MariaDBMySQL

9.5[0] PostgreSQLSQLite

8.2[0] DB2 LUW8i[0] Oracle

2005[0] SQL Server[0]Not for derived tables

PostgreSQL 10 2017-10-05

XMLTABLE

SELECTid,c1,nFROMtbl,XMLTABLE('/d/e'PASSINGxCOLUMNSidINTPATH'@id',c1VARCHAR(255)PATH'c1',nFORORDINALITY)r

XMLTABLE Since SQL:2006Stored in tbl.x:

XPath* expression to identify rows

*Standard SQL allows XQuery

XPath* expressions to extract data

Row number (like for unnest)

Result id|c1|n----+----+---42|…|1

XMLTABLE Availability19

MariaDBMySQL

10[0] PostgreSQLSQLite

9.7 DB2 LUW11gR1 Oracle

SQL Server[0]No XQuery (only XPath). No default namespace declaration.

IDENTITYCOLUMNS

IDENTITY column Since SQL:2003Similar to serial columns, but standard and more powerful.

CREATETABLEtbl(idINTEGERGENERATEDBYDEFAULTASIDENTITY(STARTWITH10INCREMENTBY10)

,PRIMARYKEY(id))

IDENTITY column Since SQL:2003Similar to serial columns, but standard and more powerful.

CREATETABLEtbl(idINTEGERGENERATEDBYDEFAULTASIDENTITY(STARTWITH10INCREMENTBY10)

,PRIMARYKEY(id))

More (esp. vs. serial):https://blog.2ndquadrant.com/postgresql-10-identity-columns/

IDENTITY column Availability19

5.1 MariaDBMySQL

10 PostgreSQLSQLite

7.0 DB2 LUW12cR1 Oracle

SQL Server

PostgreSQL 11 201?-??-??

OVER and

PARTITIONBY

Not new, butbear with me

OVER (PARTITION BY) The ProblemTwo distinct concepts could not be used independently:

‣Merge rows with the same key properties

‣ GROUPBY to specify key properties

‣ DISTINCT to use full row as key

‣ Aggregate data from related rows ‣ Requires GROUPBY to segregate the rows

‣ COUNT, SUM, AVG, MIN, MAX to aggregate grouped rows

SELECTc1,SUM(c2)totFROMtGROUPBYc1

OVER (PARTITION BY) The Problem

Yes ⇠

ws ⇢

No ⇠ Aggregate ⇢ Yes

SELECTc1,c2FROMt

SELECTDISTINCTc1,c2FROMt

SELECTc1,c2FROMt

Yes ⇠

ws ⇢

SELECTc1,c2FROMt

SELECTc1,c2FROMtJOIN()taON(t.c1=ta.c1)

Yes ⇠

ws ⇢

SELECTc1,c2FROMt

SELECTc1,c2FROMtJOIN()taON(t.c1=ta.c1)

OVER (PARTITION BY) Since SQL:2003

Yes ⇠

ws ⇢

SELECTc1,c2FROMt

,SUM(c2)OVER(PARTITIONBYc1)

SELECTdep,salary,SUM(salary)OVER()FROMemp

dep salary ts1 1000 600022 1000 600022 1000 6000333 1000 6000333 1000 6000333 1000 6000

OVER (PARTITION BY) How it works

dep salary ts1 1000 600022 1000 600022 1000 6000333 1000 6000333 1000 6000333 1000 6000

dep salary ts1 1000 100022 1000 200022 1000 2000333 1000 3000333 1000 3000333 1000 3000

)PARTITIONBYdep

OVER and

ORDERBY(Framing & Ranking)

Still not new, butbear with me

acnt id value balance

1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20

OVER (ORDER BY) The Problem

SELECTid,value,

(SELECTSUM(value)FROMtransactionst2WHEREt2.id<=t.id)

FROMtransactionst

1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20

OVER (ORDER BY) The Problem

SELECTid,value,

(SELECTSUM(value)FROMtransactionst2WHEREt2.id<=t.id)

FROMtransactionst

Range segregation (<=)not possible with

GROUP BY orPARTITION BY

OVER (ORDER BY) Since SQL:2003

SELECTid,value,

FROMtransactionst

SUM(value)OVER(

1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20

ORDERBYid

SELECTid,value,

FROMtransactionst

SUM(value)OVER(

1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20

ORDERBYidROWSBETWEENUNBOUNDEDPRECEDINGANDCURRENTROW

SELECTid,value,

FROMtransactionst

SUM(value)OVER(

1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20

SELECTid,value,

FROMtransactionst

SUM(value)OVER(

1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20

SELECTid,value,

FROMtransactionst

SUM(value)OVER(

1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20

SELECTid,value,

FROMtransactionst

SUM(value)OVER(

1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20

SELECTid,value,

FROMtransactionst

SUM(value)OVER(

1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20

SELECTid,value,

FROMtransactionst

SUM(value)OVER(

1 1 +10

22 2 +20

22 3 -10

333 4 +50

333 5 -30

333 6 -20

SELECTid,value,

FROMtransactionst

SUM(value)OVER(

1 1 +10 +10

22 2 +20 +20

22 3 -10 +10

333 4 +50 +50

333 5 -30 +20

333 6 -20 .0

PARTITIONBYacnt

OVER (ORDER BY) Since SQL:2003With OVER(ORDERBYn) a new type of functions make sense:

n ROW_NUMBER RANK DENSE_RANK PERCENT_RANK CUME_DIST1 1 1 1 0 0.252 2 2 2 0.33… 0.753 3 2 2 0.33… 0.754 4 4 3 1 1

‣ Aggregates without GROUPBY

‣ Running totals, moving averages

‣ Ranking‣ Top-N per Group

‣ Avoiding self-joins

[… many more …]

Use Cases

SELECT*FROM(SELECTROW_NUMBER()OVER(PARTITIONBY…ORDERBY…)rn,t.*FROMt)numbered_tWHERErn<=3

AVG(…)OVER(ORDERBY…ROWSBETWEEN3PRECEDINGAND3FOLLOWING)moving_avg

OVER (SQL:2003)

OVER may follow any aggregate function

OVER defines which rows are visible at each row

OVER() makes all rows visible at every row

OVER(PARTITIONBY …) segregates like GROUPBY

OVER(ORDERBY…BETWEEN) segregates using <, >

In a NutshellOVER (SQL:2003)

OVER (SQL:2003) Availability19

5.1 10.2 MariaDB8.0 MySQL

8.4 PostgreSQLSQLite

7.0 DB2 LUW8i Oracle

2005 SQL Server

ImpalaSpark

8 years

OVERSQL:2011 functions

WITHnumbered_tAS(SELECT*)

SELECTcurr.*,curr.balance-COALESCE(prev.balance,0)FROMnumbered_tcurrLEFTJOINnumbered_tprevON(curr.rn=prev.rn+1)

OVER (SQL:2011) The ProblemDirect access of other rows of the same window is not possible.

(E.g., calculate the difference to the previous rows)

currbalance … rn

50 … 190 … 270 … 330 … 4

currbalance … rn

50 … 190 … 270 … 330 … 4

FROMt,ROW_NUMBER()OVER(ORDERBYx)rn

currbalance … rn

50 … 190 … 270 … 330 … 4

currbalance … rn

50 … 190 … 270 … 330 … 4

currbalance … rn

50 … 190 … 270 … 330 … 4

prevbalance … rn

50 … 190 … 270 … 330 … 4

currbalance … rn

50 … 190 … 270 … 330 … 4

prevbalance … rn

50 … 190 … 270 … 330 … 4

currbalance … rn

50 … 190 … 270 … 330 … 4

prevbalance … rn

50 … 190 … 270 … 330 … 4

+50+40-20-40

SQL:2011 can access other rows directly:SELECT*,balance-LAG(balance,1,0)OVER(ORDERBYx)FROMdata

OVER Since SQL:2011

Available functions: LEAD/LAGFIRST_VALUE/LAST_VALUENTH_VALUE(col,n)FROMFIRST/LASTRESPECT/IGNORENULLS

OVER Since SQL:2011

Available functions: LEAD/LAGFIRST_VALUE/LAST_VALUENTH_VALUE(col,n)FROMFIRST/LASTRESPECT/IGNORENULLS

OVER Since SQL:2011

Not supported by PostgreSQL

(as of 11)

OVER (LEAD, LAG, …) Since SQL:201119

5.1 10.2[0] MariaDB8.0[0] MySQL

8.4[0] PostgreSQLSQLite

9.5[1] 11.1 DB2 LUW8i[1] 11gR2 Oracle

2012[1] SQL Server[0]No IGNORENULLS and FROMLAST[1]No NTH_VALUE

OVERSQL:2011 groups option

OVER (groups option) Since SQL:2011ORDERBYx<frameunit>between1precedingand1following

rows,range

rangexbetweencurrent_row-1

andcurrent_row+1

rowscount(*)

3.53.54

CURRENT ROW

groupscount(distinctx)

rows,rangeNew in

SQL:2011

andPostgreSQL 11

groups

rangexbetweencurrent_row-1

andcurrent_row+1

rowscount(*)

3.53.54

CURRENT ROW

Since SQL:201119

5.1 MariaDBMySQL

11 PostgreSQLSQLite

DB2 LUWOracleSQL Server

OVER (groups option)

OVERSQL:2003 frame exclusion

Since SQL:2003OVER(ORDERBY…BETWEEN…exclude[noothers|currentrow|group|ties])

default

New inPostgreSQL 11

OVER (frame exclusion)

noothers

currentrow

Since SQL:2003

x12223

OVER(ORDERBY…BETWEEN…exclude[noothers|currentrow|group|ties])

default

New inPostgreSQL 11

currentrow

groupx=current_row

currentrow

Since SQL:2003

x12223

default

New inPostgreSQL 11

tiesgroupx=current_row

currentrow

Since SQL:2003

x12223

default

New inPostgreSQL 11

OVER (frame exclusion) Since SQL:201119

5.1 MariaDBMySQL

11 PostgreSQLSQLite

DB2 LUWOracleSQL Server

PostgreSQL 12+ 20??-??-??

Copying rows from another table is easy:

INSERTINTO<target>SELECT…FROM<source>WHERENOTEXISTS(SELECT*FROM<target>WHERE…)

MERGE The Problem

Both, <target> and <source>

are in scope here.

Deleting rows that exist in another table is also possible:

DELETEFROM<target>WHEREEXISTS(SELECT*FROM<source>WHERE…)

MERGE The Problem

are in scope here.

Updating rows from another table is awkward:

UPDATE<target>SET…WHERE…

MERGE The Problem

Subqueries are a more common choice

Sometimes, updatable views can help.

Bringing another tables rows into scope of the SET clause is tricky

Updating rows from another table is awkward:

UPDATE<target>SET…WHERE…SET(col1,col2)=(SELECTcol1,col2FROM<source>WHERE…)WHERE…

MERGE The Problem

are in scope here.

SQL:2008 introduced merge to improve this situation two-fold:

‣It has always two tables in scope,the source can be a derived table (subquery). ‣It can do insert, update, or delete in one go.

MERGEINTO<target>USING<source>ON<joincondition>WHENMATCHED[AND<cond>]THEN[UPDATE…|DELETE…]WHENNOTMATCHED[AND<cond>]THENINSERT…

MERGE Since SQL:2008

WHEN/THENcan appear many times

MERGE Since SQL:200819

5.1 MariaDBMySQLPostgreSQLSQLite

9.1 DB2 LUW9iR1[0] Oracle

2008 SQL Server[0]No AND condition.

(standard) JSON

JSONPostgreSQL got the JSON data type and

the first JSON related functions with 9.2 (2012)

ISO added SQL JSON functions (but no type)with SQL:2016

Both approaches follow a different mantra. The following slides only reflects the standards perspective.

JSON — Preliminary testing of patchesUsed a06e56b24 as basis, applied those patches on top:

0001-strict-do_to_timestamp-v16.patch 0002-pass-cstring-to-do_to_timestamp-v16.patch 0003-add-to_datetime-v16.patch 0004-jsonpath-v16.patch 0005-jsonpath-gin-v16.patch 0006-jsonpath-json-v16.patch 0007-remove-PG_TRY-in-jsonpath-arithmetics-v16.patch 0008-jsonpath-extensions-v16.patch 0009-jsonpath-extensions-tests-for-json-v16.patch 0010-add-invisible-coercion-form-v16.patch 0011-add-function-formats-v16.patch 0012-sqljson-v16.patch 0013-sqljson-json-v16.patch 0014-optimize-sqljson-subtransactions-v16.patch 0015-json_table-v16.patch 0016-json_table-json-v16.patch }SQL/JSON: JSON_TABLE

}}SQL/JSON: jsonpath

SQL/JSON: functions

JSON — Preliminary testing of patches

@ModernSQL modern-sql.comMy other website:

https://use-the-index-luke.com

Training & co: https://winand.at/

5-day intensive training — Sept 17-21Visit Vienna!SQL Performance Kick-Start • Indexing, Indexing, Indexing… • Joining • Ordering, grouping, • …

Analysis and Aggregation • OVER, GROUPING SETS • Avoiding self-joins • Grouping consecutive events • …

SQL Reloaded • Type safety, NULL an 3VL • Modern interpretation of

the relational model • Keeping historic data • Design guidelines

Recursive Queries • WITH • WITH RECURSIVE • Use cases and examples

https://winand.at/sql-training/5-day-training/aug-sept-2018

How PostgreSQL’s SQL dialect stays ahead of its competitors · GROUPING SETS are multiple GROUP...

Documents

Cornwall/Kernow Kernow bys vyken! (Cornwall For Ever)

Blood grouping grouping/blood... · Web viewBlood grouping Blood grouping This report includes about the introduction of how blood groups classified and also about different types

Grouping Angles11

Moodle Grouping

Grouping Students

Grouping Symbols

Winter visit with the Flutter-bys

ABO Grouping (Reverse Grouping )-LAB-062

Musical Grouping Structure - titanmusic.com · done on musical grouping structure. ... theory of grouping structure • Musical grouping is an “auditory analog of the partitioning

grouping exercise

Revista bys

Use this chart to practice your 6s count-bys

Flexible Grouping

APPENDIX 11.8: PROPOSED LAY-BYS ECOLOGY REPORT...PROPOSED LAY-BYS ECOLOGY REPORT . Page 2 MYG Appendix 11. 8: Lay-Bys Ecology Report Mynydd y Gwynt Wind Farm: Environmental Statement

Uav Grouping

Stratos Grouping

10- DPM - DPM Bys Pass - 0-1.2

GROUPING LEARNERS

BYS Report

Grouping data