Combining SQL and NoSQL with MySQL · •In 5.7: Updating a JSON document alwayscauses the entire document to be written •In 8.0: Automatically detect potential in-place updates

Copyright © 2016, Oracle and/or its affiliates. All rights reserved. Copyright © 2014, Oracle and/or its affiliates. All rights reserved.

Manyi Lu

DirectorMySQL Optimizer Team, OracleApril, 2018

Combining SQL and NoSQL withMySQL

Copyright © 2016, Oracle and/or its affiliates. All rights reserved.

Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

2


Agenda

3

JSON data type

JSON functions

Indexing JSON data

Partial update of JSON


CREATE TABLE employees (data JSON);INSERT INTO employees VALUES('{"id": 1, "name": "Jane"}'), ('{"id": 2, "name": "Joe"}');SELECT * FROM employees;+-------------------------------------+| data |+-------------------------------------+| {"id": 1, "name": "Jane"} || {"id": 2, "name": "Joe"} |+-------------------------------------+

• Validation on INSERT

• No reparsing on SELECT

• Optimized for read

• Dictionary of sorted keys

• Can compare JSON/SQL

• Can convert JSON/SQL

• Supports all native JSON datatypes

• Also supports date, time, timestamp etc.

4

The New JSON Datatype


Agenda

5

JSON data type

JSON functions

Indexing JSON data



JSON FunctionsMySQL 5.7 and 8.0

6

JSON_ARRAY_APPEND()

JSON_ARRAY_INSERT()

JSON_ARRAY()

JSON_CONTAINS_PATH()

JSON_CONTAINS()

JSON_DEPTH()

JSON_EXTRACT()

JSON_INSERT()

JSON_KEYS()

JSON_LENGTH()

JSON_MERGE[_PRESERVE]()

JSON_OBJECT()

JSON_QUOTE()

JSON_REMOVE()

JSON_REPLACE()

JSON_SEARCH()

JSON_SET()

JSON_TYPE()

JSON_UNQUOTE()

JSON_VALID()

JSON_PRETTY()

JSON_STORAGE_SIZE()

JSON_STORAGE_FREE()

JSON_ARRAYAGG()

JSON_OBJECTAGG()

JSON_MERGE_PATCH()

JSON_TABLE()


Inlined JSON Path Expressions

• SELECT * FROM employees WHERE data->'$.id'= 2;

• ALTER … ADD COLUMN id INT AS (data->'$.id') …

• CREATE VIEW .. AS SELECT data->'$.id', data->'$.name' FROM …

SELECT * FROM employees WHERE data->”$.name” = “Jane";

Is a short hand for

SELECT * FROM employees WHERE JSON_EXTRACT(data, “$.name” ) = “Jane”;

[[database.]table.]column->”$<path spec>”

7


JSON_TABLE

8

• JSON is not limited to CRUD, can also be used in complex queries

• JSON_TABLE creates a relational view of JSON data

– Each object in a JSON array as a row

– JSON values within an JSON object as column values

• Query the result of JSON_TABLE() as a relational table using SQL

• Leverage existing framework for aggregation


JSON_TABLE

CREATE TABLE t1(json_col JSON);

INSERT INTO t1 VALUES ('{ "people": [

{ "name":"John Smith", "address":"780 Mission St, San Francisco, CA 94103"}, { "name":"Sally Brown", "address":"75 37th Ave S, St Cloud, MN 94103"}, { "name":"John Johnson", "address":"1262 Roosevelt Trail, Raymond, ME 04071"}

] }');

9


JSON_TABLE

SELECT people.* FROM t1,

JSON_TABLE(json_col, '$.people[*]' COLUMNS (name VARCHAR(40) PATH '$.name',address VARCHAR(100) PATH '$.address')) people;

10

Convert JSON documents to relational tables

name address

John Smith 780 Mission St, San Francisco, CA 94103

Sally Brown 75 37th Ave S, St Cloud, MN 9410

John Johnson 1262 Roosevelt Trail, Raymond, ME 04071


JSON_TABLE

SELECT people.* FROM t1,

JSON_TABLE(json_col, '$.people[*]' COLUMNS (name VARCHAR(40) PATH '$.name',address VARCHAR(100) PATH '$.address')) people;

WHERE people.name LIKE 'John%';

11

Filter JSON data

name address

John Smith 780 Mission St, San Francisco, CA 94103

John Johnson 1262 Roosevelt Trail, Raymond, ME 04071

Copyright © 2016, Oracle and/or its affiliates. All rights reserved. 12

JSON_TABLE – Nested Arrays

id father married child_id child age

1 John 1 1 Eric 12

1 John 1 2 Beth 10

2 Paul 0 1 Sarah 9

2 Paul 0 2 Noah 3

2 Paul 0 3 Peter 1

[ { "father":"John", "mother":"Mary",

"marriage_date":"2003-12-05", "children": [

{ "name":"Eric", "age":12 },{ "name":"Beth", "age":10 } ] },

{ "father":"Paul", "mother":"Laura", "children": [

{ "name":"Sarah", "age":9},{ "name":"Noah", "age":3} ,{ "name":"Peter", "age":1} ] }

]


JSON_TABLE – Nested Arrays

id father married child_id child age

1 John 1 1 Eric 12

1 John 1 2 Beth 10

2 Paul 0 1 Sarah 9

2 Paul 0 2 Noah 3

2 Paul 0 3 Peter 1

JSON_TABLE (families, '$[*]' COLUMNS (id FOR ORDINALITY,father VARCHAR(30) PATH '$.father',married INTEGER EXISTS PATH

'$.marriage_date',NESTED PATH '$.children[*]' COLUMNS (

child_id FOR ORDINALITY,child VARCHAR(30) PATH '$.name',age INTEGER PATH '$.age‘ ) )


JSON_TABLE

SELECT father, COUNT(*) "#children", AVG(age) "age average"FROM t, JSON_TABLE (families, '$[*]' COLUMNS (

id FOR ORDINALITY,father VARCHAR(30) PATH '$.father',NESTED PATH '$.children[*]' COLUMNS (age INTEGER PATH '$.age‘ ) ) AS fam

GROUP BY id, father;

14

SQL aggregation on JSON data

father #children age average

John 2 11.0000

Paul 3 4.3333


JSON Aggregation

15

Combine JSON documents in multiple rows into a JSON array

CREATE TABLE t1 (id INT, grp INT, jsoncol JSON);

INSERT INTO t1 VALUES (1, 1, '{"key1":"value1","key2":"value2"}');

INSERT INTO t1 VALUES (2, 1, '{"keyA":"valueA","keyB":"valueB"}');

INSERT INTO t1 VALUES (3, 2, '{"keyX":"valueX","keyY":"valueY"}');

SELECT JSON_ARRAYAGG(jsoncol) FROM t1;

[{"key1":"value1","key2":"value2"}, {"keyA":"valueA","keyB":"valueB"}, {"keyX":"valueX","keyY":"valueY"}]


JSON Aggregation

16

Combine JSON documents in multiple rows into a JSON object

CREATE TABLE t1 (id INT, grp INT, jsoncol JSON);

INSERT INTO t1 VALUES (1, 1, '{"key1":"value1","key2":"value2"}');

INSERT INTO t1 VALUES (2, 1, '{"keyA":"valueA","keyB":"valueB"}');

INSERT INTO t1 VALUES (3, 2, '{"keyX":"valueX","keyY":"valueY"}');

SELECT grp, JSON_OBJECTAGG(id, jsoncol)FROM t1GROUP BY grp;

1 | {"1":{"key1":"value1","key2":"value2"}, "2":{"keyA":"valueA","keyB":"valueB"}} 2 | {"3":{"keyX":"valueX","keyY":"valueY"}}


Agenda

17

JSON data type

JSON functions

Indexing JSON data



Indexing JSON data

• Use Functional Indexes,

• STORED and VIRTUAL types are supported

18

CREATE TABLE employees (data JSON);

ALTER TABLE employees ADD COLUMN name VARCHAR(30) AS (data->>”$.name”) VIRTUAL, ADD INDEX name_idx (name);

• Functional index approach

• Use inlined JSON path or JSON_EXTRACT to specify field to be indexed

• Support both virtual and stored generated columns

• data->>”$.name” is a shorthand for JSON_UNQUOTE(data->”$.name”)


Simplified Functional Index Syntax Upcoming!

19

• In 5.7: Create a virtual column, then add an index on this column

• In 8.0: Add a functional index directly, hide creation of virtual column

ALTER TABLE employeesADD COLUMN name VARCHAR(30) AS (data->>”$.name”) VIRTUAL, ADD INDEX name_idx (name);

ALTER TABLE employeesADD INDEX name_idx ((CAST(data->>"$.name" AS VARCHAR(30))));


Indexing JSON Array Upcoming!

20

• 5.7: Single value index, one index record <-> one data record

• 8.0: Multi value index, multiple index records <-> one data record

• Multi-value index for efficient queries against array fields

• Use CAST( <expr> AS <type> ARRAY) to create multi-valued index

– Expr is any expression that returns JSON array

– Type is any SQL type supported by CAST()

ALTER TABLE employeesADD INDEX name_idx ((CAST(data->>"$.address" AS VARCHAR(30) ARRAY));



id Families

1 {"father":"John", "mother":"Mary","marriage_date":"2003-12-05",

"children": [ {"name":"Eric","age":12,"eyes":"blue"},{"name":"Beth","age":10, "eyes": "green"}]}

2 {"father":"Paul", "mother": "Laura", "children": [

{"name":"Sarah","age":9,"eyes":"brown"},{"name":"Noah","age":3,"eyes": "green"} ,{"name":"Peter","age":1,"eyes": "green"}]}

CREATE TABLE t (id int primary key, families JSON);ALTER TABLE t ADD INDEX eye_color((CAST(families->>"$.children[*].eyes" AS CHAR(10) ARRAY)))

eye_color id

blue 1

brown 2

green 1

green 2

Index



22

id father mother

2 Paul Laura

Find parents with children having brown eyes:

SELECT id, families->>"$.father", families->>"$.mother" FROM tWHERE "brown" IN (families->>"$.children[*].eyes")

eye_color id

blue 1

brown 2

green 1

green 2

Index



23

Find parents with children having either blue or brown eyes.

SELECT id, families->>"$.father", families->>"$.mother" FROM t WHERE JSON_OVERLAPS(families->>"$.children[*].eyes",

'["blue", "brown"]')

eye_color id

blue 1

brown 2

green 1

green 2

id father mother

1 John Mary

2 Paul Laura

Index


Agenda

24

JSON data type

JSON functions

Indexing JSON data



Partial Update of JSON

25

• In 5.7: Updating a JSON document always causes the entire document to be written

• In 8.0: Automatically detect potential in-place updates

– JSON_SET, JSON_REPLACE, JSON_REMOVE

– Input and target columns are the same

– Updating existing values, not adding new ones

– New value is equal to or smaller than the original value, or previous partial update has left sufficient space

• In 8.0: Only diff is shipped through binlog (default full image)

• In 8.0: InnoDB supports partial update of BLOB



Agenda

27

JSON data type

JSON functions

Indexing JSON data


JSON or column?


Storing as JSON

• More flexible way to represent data that is hard to model in schema

• No painful schema changes

• Easier prototyping–Fewer types to consider

–No enforced schema

28


Storing as Column

• Easier to apply schema to your application

• Schema may make applications easier to maintain over time as change is controlled;–Do not have to expect as many permutations

–Allows some constraints over data

29


Hybrid: Schema + Schemaless

30

CREATE TABLE pc_components (id INT NOT NULL PRIMARY KEY,description VARCHAR (60) NOT NULL,vendor VARCHAR(30) NOT NULL,serial_number VARCHAR(30) NOT NULL,attributes JSON NOT NULL

);

Documents

Combining SQL and NoSQL with MySQL · •In 5.7: Updating a JSON document alwayscauses the entire document to be written •In 8.0: Automatically detect potential in-place updates