Embed Size (px)
DESCRIPTION
The advantage of RDBMS It separates logical level (schema) from physical level (implementation). Physical data independence – Users do not worry about how their data is stored and processes on the physical devices. – It is all SQL! – Their queries work over (almost) all RDBMS deployments. 3
Citation preview
1
CS 440 Database Management Systems
Lecture 6: Data storage & access methods
2
Database System Implementation
Conceptual Design
Physical Storage Schema
Entity Relationship(ER)
Model
Relational Model Files and Indexes
User Requirements
3
The advantage of RDBMS• It separates logical level (schema) from physical
level (implementation). • Physical data independence– Users do not worry about how their data is stored and
processes on the physical devices.– It is all SQL!– Their queries work over (almost) all RDBMS
deployments.
4
DBMS Architecture
Query Executor
Buffer Manager
Storage Manager
Storage
Transaction Manager
Logging & Recovery
Lock Manager
Buffers Lock Tables
Main Memory
User/Web Forms/Applications/DBA
query transaction
Query Optimizer
Query Rewriter
Query Parser
Files & Access Methods
Process manager
5
Challenges in physical level• Processor: 10000 – 100000 MIPS• Main memory: around 10 Gb/ sec.• Secondary storage: higher capacity and durability• Disk random access – Seek time + rotational latency + transfer time– Seek time: 4 ms - 15 ms!– Rotational latency: 2 ms – 7 ms!– Transfer time: at most 1000 Mb/ sec– Read, write in blocks.
6
Gloomy future: Moor’s law• Speed of processors and cost and maximum
capacity of storage increase exponentially over time.
• But storage (main and secondary) access time grows much more slowly.
7
Random access versus sequential access
• Disk random access : Seek time + rotational latency + transfer time.
• Disk sequential access: reading blocks next to each other– No seek time or rotational latency –Much faster than random access
8
Units of data on physical device• Fields: data items• Records• Blocks• Files
9
Fields• Fixed size– Integer, Boolean, …
• Variable length– Varchar, …– Null terminated– Size at the beginning of the string
10
Records: sets of fields• Schema– Number of fields, types of fields, order, …
• Fixed format and length– Record holds only the data items
• Variable format and length– Record holds fields and their size, type, …
information• Range of formats in between
11
Record header• Pointer to the record schema ( record type)• Record size• Timestamp• Other info …
12
Blocks• Collection of records• Reduces number of I/O access• Different from OS blocks–Why should RDBMS manage its own blocks?• It knows the access pattern better than OS.
• Separating records in a block– Fixed size records: no worry!–Markers between records– Keep record size information in records or block
header.
13
Spanned versus un-spanned• Un-spanned– Each records belongs to only one block
• Spanned– Records may be stored across multiple blocks– Saves space– The only way to deal with large records and fields:
blob, image, …
14
Block header• Data about block• File, relation, DB IDs • Block ID and type• Record directory• Pointer to free space• Timestamp• Other info …
15
Heap versus sorted files• Heap files– There is not any order in the file– New blocks are inserted at the end of the file.
• Sorted files– Order blocks (and records) based on some key.– Physically contiguous or using links to the next
blocks.
16
Average cost of data operations• Insertion – Heap files are more efficient.– Overflow areas for sorted files.
• Search for a record or a range of records– Sorted files are more efficient.
• Deletion– Heap files are more efficient – Although we find the record faster in the sorted file.
17
Row versus column stores• We have talked about row store– All fields of a record are stored together.
SSN1 Name1 Age1 Salary1SSN2 Name2 Age2 Salary2SSN3 Name3 Age3 Salary3
18
Row versus column stores• We can store the fields in columns.–We can store SSNs implicitly.
SSN1 Name1SSN2 Name2SSN3 Name3
SSN1 Age1SSN2 Age2SSN3 Age3
SSN1 Salary1SSN2 Salary2SSN3 Salary3
19
Row versus column store• Column store– Compact storage– Faster reads on data analysis and mining operations
• Row store– Faster writes – Faster reads for record access (OLTP)
• Further reading–Mike Stonebreaker, et al, “C-Store, a column oriented
DBMS”, VLDB’05.
20
Access paths• The methods that RDBMS uses to retrieve the
data.• Attribute value(s) Tuple(s)
21
Types of search queries• Point query over Beers(name, manf) Select *
From BeersWhere name = ‘Bud’;
• Range query over Sells(bar, beer, price) Select *
From SellsWhere price > 2 AND price <
10;
22
Types of access paths• Full table scan– Heap files– Inefficient for both point and range queries.
• Sequential access– Sorted files– Efficient for both point and range queries. – Inefficient to maintain
• Middle ground?
23
Indexing• An old idea
24
Index• A data structure that speeds up selecting tuples in
a relation based on some search keys.• Search key– A subset of the attributes in a relation–May not be the same as the (primary) key
• Entries in an index– (k, r)– k is the search key.– r is the pointer to a record (record id).
25
Index• Data file stores the table data. • Index file stores the index data structure.
• Index file is smaller than the data file. • Ideally, the index should fit in the main memory.
10
20
30
40
50
60
70
80
10
20
30
40
50
60
Data File Index File
26
Index categorizations• Clustered vs. unclustered – Records are stored according to the index order.– Records are stored in another order, or not any order.
• Dense vs. sparse – Each record is pointed by an entry in the index.– Each block has an entry in the index.– Size versus time tradeoff.
• Primary vs. secondary – Primary key is the search key– Other attributes.
27
Index categorizations• Clustered and dense
10
20
30
40
50
60
70
80
10
20
30
40
50
60
DATAINDEX
28
Index categorizations• Clustered and sparse
10
30
50
70
90
110
10
20
30
40
50
60
DATAINDEX
70
80
29
Duplicate search keys • Clustered and dense
10
20
30
40
50
60
10
10
10
20
20
30
DATAINDEX
40
50
30
Duplicate search keys • Clustered and sparse:
– Any problem?
10
10
20
40
50
60
10
10
10
20
20
30
DATAINDEX
40
50
31
Duplicate search keys • Clustered and sparse: – Point to the lowest new search key in every block
10
20
30
40
50
10
10
10
20
20
30
DATAINDEX
40
50
32
Unclustered Index• Dense / sparse?
10
10
10
20
20
30
30
40
30
10
20
30
10
20
DATAINDEX
10
40
33
Well known index structures• B+ trees:– very popular
• Hash tables: – Not frequently used
34
B+ trees• The index of a very large data file gets too large.
• How about building an index for the index file?
• A multi-level index, or a tree
35
B+ trees• Degree (order) of the tree: d• Each node (except root) stores [d, 2d] keys:
10 32 94
[A , 10) [10, 32) [32, 94) [94, B)
Non-leaf nodes
12 28 32
12 28 32
39 41 65Leaf nodes
Records
36
Example
60
19 50 80 90 110
12 13 17 19 21 30 40 50 52 60 65 72
12 13 17 19 21 30 40 50 52 60 65 72
d = 2
37
B+ tree tuning• How to choose the value of d?– Each node should fit in a block.
• Example– Key value: 8 byte– Record pointer: 16 bytes– Block size: 4096 bytes– 2d * 8 + (2d + 1) * 16 <= 4096– d <= 85
38
Retrieving tuples using B+ tree • Point queries– Start from the root and follow the links to the leaf.
• Range queries– Find the lowest point in the range.– Then, follow the links between the nodes.
• The top levels are kept in the buffer pool.
39
B+ tree and index categories• B+ tree index could be – Dense / sparse?– Clustered/ unclustered?
40
Inserting a new key• Pick the proper leaf node and insert the key.• If the node contains more than 2d keys, split the
node and insert the extra node in the parent.
– If leaf level, add K3 to the right node
K1 K2 K3 K4 K5
R0 R1 R2 R3 R4 R5
K1 K2
R0 R1 R2
K4 K5
R3 R4 R5
(K3, ) parent
41
Insertion
60
19 50 80 90 110
12 13 17 19 21 30 40 50 52 60 65 72
12 13 17 19 21 30 40 50 52 60 65 72
Insert K = 18
42
Insertion
60
19 50 80 90 110
12 13 17 18 19 21 30 40 50 52 60 65 72
12 13 17 19 21 30 40 50 52 60 65 72
Insert K = 18
18
43
Insertion
60
19 50 80 90 110
12 13 17 18 50 52 60 65 72
12 13 17 19 21 30 40 50 52 60 65 72
Insert K= 20
19 20 21 30 40
2018
44
Insertion
60
19 50 80 90 110
12 13 17 18 50 52 60 65 72
12 13 17 19 21 30 40 50 52 60 65 72
Need to split the node
19 20 21 30 40
2018
45
Insertion
60
19 21 50 80 90 110
12 13 17 18 50 52 60 65 72
12 13 17 19 21 30 40 50 52 60 65 72
Split and update the parent node.What if we need to split the root?
20
19 20 21 30 40
18
46
Deletion
60
19 21 50 80 90 110
12 13 17 18 50 52 60 65 72
12 13 17 19 21 30 40 50 52 60 65 72
Delete K = 21
20
19 20 21 30 40
18
47
Deletion
60
19 21 50 80 90 110
12 13 17 18 50 52 60 65 72
12 13 17 19 30 40 50 52 60 65 72
Note: K = 21 may still remain in the internal levels
20
19 20 30 40
18
48
Deletion
60
19 21 50 80 90 110
12 13 17 18 50 52 60 65 72
12 13 17 19 30 40 50 52 60 65 72
Delete K = 20
20
19 20 30 40
18
49
Deletion
60
19 21 50 80 90 110
12 13 17 18 50 52 60 65 72
12 13 17 19 30 40 50 52 60 65 72
We need to update the number of keys on the node: Borrow from siblings: rotate
19 30 40
18
50
Deletion
60
19 21 50 80 90 110
12 13 17 50 52 60 65 72
12 13 17 19 30 40 50 52 60 65 72
We need to update the number of keys on the node: Borrow from siblings: rotate
18 19 30 40
18
51
Deletion
60
18 21 50 80 90 110
12 13 17 50 52 60 65 72
12 13 17 19 30 40 50 52 60 65 72
We need to update the number of keys on the node: Borrow from siblings: rotate
18 19 30 40
18
52
Deletion
60
18 21 50 80 90 110
12 13 17 50 52 60 65 72
12 13 17 19 30 40 50 52 60 65 72
What if we cannot borrow from siblings?Example: delete K = 30
18 19 30 40
18
53
Deletion
60
18 21 50 80 90 110
12 13 17 50 52 60 65 72
12 13 17 19 40 50 52 60 65 72
What if we cannot borrow from siblings?Merge with a sibling.
18 19 40
18
54
Deletion
60
18 21 50 80 90 110
12 13 17 50 52 60 65 72
12 13 17 19 40 50 52 60 65 72
What if we cannot borrow from siblings?Merge siblings!
18 19 40
18
55
Deletion
60
18 21 50 80 90 110
12 13 17 50 52 60 65 72
12 13 17 19 40 50 52 60 65 72
What to do with the dangling key and pointer? simply remove them
18 19 40
18
56
Deletion
60
18 50 80 90 110
12 13 17 50 52 60 65 72
12 13 17 19 40 50 52 60 65 72
Final tree
18 19 40
18
57
Index creationCREATE TABLE Person(Name varchar(50), Pos int, Age int);CREATE INDEX Person_ID ON Person(ID);
CLUSTER Person USING ON Person_ID;
CREATE INDEX Pos_Age ON Person(Pos, Age);
Default is normally B-tree.
Cluster Person_ID index
Multi-attribute index
58
Index selection• Let’s index every attribute on every table to speed
up all queries!
• Indexes generally slow down data manipulation– INSERT, DELETE, UPDATE.
59
Index selection• Given a query workload and a schema, find the
set of indexes that optimize the execution.• The query workload:– Queries and their frequencies.– Queries are both data retrieval (SELECT) and data
manipulation (INSERT, UPDATE, DELETE).
60
Index selection• Part of physical database design– File structure, indexing, tuning queries,…
• Physical database design may affect logical design!– Change the schema to run the queries faster
61
Index selection• Generally a hard problem.• RDBMS vendors provide wizards:– Started with AutoAdmin project for SQL Server– SQL Server/ Oracle Index Tuning Wizard– DB2 Index Advisor
• They try many configurations and pick the one that minimizes the time and overheads.
