CS 4604: Introduction to Database Management...

CS4604:IntroductiontoDatabaseManagementSystems

B.AdityaPrakashLecture#8:StoringdataandIndexes

Annoucements

§  Extraofficehourstillmidterm– CheckPiazzapost

Prakash2018 VTCS4604 2

STORINGDATA

Prakash2018 VTCS4604

DBMSLayers:

Query Optimization and Execution

Relational Operators

Files and Access Methods

Buffer Management

Disk Space Management

Queries

TODAYà

LeverageOSfordisk/filemanagement?

§  Layersofabstractionaregood…but:

§  Layersofabstractionaregood…but:– Unfortunately,OSoftengetsinthewayofDBMS

§  DBMSwants/needstodothings“itsownway”– Specializedprefetching– Controloverbufferreplacementpolicy

•  LRUnotalwaysbest(sometimesworst!!)– Controloverthread/processscheduling

•  “Convoyproblem”– AriseswhenOSschedulingconflictswithDBMSlocking

– Controloverflushingdatatodisk• WALprotocolrequiresflushinglogentriestodisk

DisksandFiles

§  DBMSstoresinformationondisks.– but:disksare(relatively)VERYslow!

§ MajorimplicationsforDBMSdesign!

DisksandFiles

§ MajorimplicationsforDBMSdesign:– READ:disk->mainmemory(RAM).– WRITE:reverse– Botharehigh-costoperations,relativetoin-memoryoperations,somustbeplannedcarefully!

WhyNotStoreItAllinMainMemory?

§  Coststoomuch.– disk:~$1/Gb;memory:~$100/Gb– High-endDatabasestodayinthe10-100TBrange.

– Approx60%ofthecostofaproductionsystemisinthedisks.

§ Mainmemoryisvolatile.§  Note:somespecializedsystemsdostoreentiredatabaseinmainmemory.

TheStorageHierarchySmaller, Faster

Bigger, Slower

TheStorageHierarchy

– Main memory (RAM) for currently used data.

– Disk for the main database (secondary storage).

– Tapes for archiving older versions of the data (tertiary storage).

Smaller, Faster

Bigger, Slower

Registers

L1 Cache

Main Memory

Magnetic Disk

Magnetic Tape

JimGray’sStorageLatencyAnalogy:HowFarAwayistheData?

Registers On Chip Cache On Board Cache

Memory

Boston

This Building

This Room My Head

10 min

1.5 hr

2 Years

2,000 Years

The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.

The image cannot be displayed. Your computer may not have

The image cannot be displayed. Your computer may not have enough

Andromeda

Disks§  Secondarystoragedeviceofchoice.§ Mainadvantageovertapes:randomaccessvs.sequential.

§  Dataisstoredandretrievedinunitscalleddiskblocksorpages.

§  UnlikeRAM,timetoretrieveadiskpagevariesdependinguponlocationondisk.–  relativeplacementofpagesondiskisimportant!

AnatomyofaDisk

Platters

Spindle

•  Sector •  Track •  Cylinder •  Platter •  Block size = multiple of sector size (which is fixed)

Disk head

Arm movement

Arm assembly

Tracks

Sector

AccessingaDiskPage

§  Timetoaccess(read/write)adiskblock:–  .–  .–  .

AccessingaDiskPage

§  Timetoaccess(read/write)adiskblock:– seektime:movingarmstopositiondiskheadontrack

–  rotationaldelay:waitingforblocktorotateunderhead

–  transfertime:actuallymovingdatato/fromdisksurface

AccessingaDiskPage

§  Relativetimes?– seektime:–  rotationaldelay:–  transfertime:

AccessingaDiskPage

§  Relativetimes?– seektime:about1to20msec–  rotationaldelay:0to10msec–  transfertime:<1msecper4KBpage

Transfer

Rotate

transfer

Seektime&rotationaldelaydominate

§  KeytolowerI/Ocost:reduceseek/rotationdelays!

§  Alsonote:Forshareddisks,muchtimespentwaitinginqueueforaccesstoarm/controller

Rotate

transfer

ArrangingPagesonDisk

§  “Next” blockconcept:– blocksonsametrack,followedby– blocksonsamecylinder,followedby– blocksonadjacentcylinder

§  Accesing‘next’blockischeap§  Ausefuloptimization:pre-fetching

– Seetextbookpage323

Rulesofthumb…

1. MemoryaccessmuchfasterthandiskI/O(~1000x)

§  “Sequential”I/Ofasterthan“random”I/O(~10x)

Conclusions---Storing

§ Memoryhierarchy§  Disks:(>1000xslower)-thus

– packinfoinblocks–  trytofetchnearbyblocks(sequentially)

TREEINDEXES

DeclaringIndexes

§  Nostandard!§  Typicalsyntax:CREATE INDEX StudentsInd ON Students(ID);

CREATE INDEX CoursesInd ON Courses(Number, DeptName);

TypesofIndexes

§  Primary:indexonakey– Usedtoenforceconstraints

§  Secondary:indexonnon-keyattribute§  Clustering:orderoftherowsinthedatapagescorrespondtotheorderoftherowsintheindex– Onlyoneclusteredindexcanexistinagiventable– Usefulforrangepredicates

§  Non-clustering:physicalordernotthesameasindexorder

UsingIndexes(1):EqualitySearches

§  Givenavaluev,theindextakesustoonlythosetuplesthathavevintheattribute(s)oftheindex.

§  E.g.(useCourseIndindex)SELECT Enrollment FROM Courses WHERE Number = “4604” and DeptName = “CS”

UsingIndexes(1):EqualitySearches

§  Givenavaluev,theindextakesustoonlythosetuplesthathavevintheattribute(s)oftheindex.

§  CanuseHashes,butseenext

UsingIndexes(2):RangeSearches

§  ``Findallstudentswithgpa>3.0’’§  maybeslow,evenonsortedfile§  Hashesnotagoodidea!§ Whattodo?

Page 2 Page N Page 3 Data File

RangeSearches

§  ``Findallstudentswithgpa>3.0’’§  maybeslow,evenonsortedfile§  Solution:Createan`index’file.

k2 kN k1 Index File

RangeSearches

§ Moredetails:§  ifindexfileissmall,dobinarysearchthere§  Otherwise??

k2 kN k1 Index File

B-trees

§  themostsuccessfulfamilyofindexschemes(B-trees,B+-trees,B*-trees)

§  Canbeusedforprimary/secondary,clustering/non-clusteringindex.

§  balanced“n-way”searchtrees§  OriginalPaper:RudolfBayerandMcCreight,E.M.OrganizationandMaintenanceofLargeOrderedIndexes.ActaInformatica1,173-189,1972.

B-trees

§  Eg.,B-treeoforderd=1:

>6 <9>9

B-treeproperties:

§  eachnode,inaB-treeoforderd:– Keyorder– atmostn=2dkeys– atleastdkeys(exceptroot,whichmayhavejust1key)

– allleavesatthesamelevel–  ifnumberofpointersisk,thennodehasexactlyk-1keys

–  (leavesareempty)

v1 v2 … vn-1

Properties

§  “blockaware”nodes:eachnodeisadiskpage§  O(log(N))foreverything!(ins/del/search)§  typically,ifd=50-100,then2-3levels§  utilization>=50%,guaranteed;onaverage69%

Queries

§  Algoforexactmatchquery?(eg.,ssn=8?)

>6 <9>9

JAVAanimation

§  http://slady.net/java/bt/

Queries

>6 <9>9

Queries

>6 <9>9

Queries

>6 <9>9

Queries

>6 <9>9

Hsteps(=diskaccesses)

Queries

§  whataboutrangequeries?(eg.,5<salary<8)§  Proximity/nearestneighborsearches?(eg.,salary~8)

Queries

>6 <9>9

Queries

>6 <9>9

Queries

>6 <9>9

Queries

>6 <9>9

Variations

§  HowcouldwedoevenbetterthantheB-treesabove?

B+trees-Motivation

§  B-tree–printkeysinsortedorder:

>6 <9>9

B+trees-Motivation

§  B-treeneedsback-tracking–howtoavoidit?

>6 <9>9

B+trees-Motivation

§  Strongerreason:forclusteringindex,datarecordsarescattered:

>6 <9>9

Solution:B+-trees

§  facilitatesequentialops§  Theystringallleafnodestogether§  AND§  replicatekeysfromnon-leafnodes,tomakesureeverykeyappearsattheleaflevel

§  (vital,forclusteringindex!)

B+trees

>=6 <9>=9

B+trees

>=6 <9>=9

IndexPages

DataPages

B+trees

§ Moredetails:next(andtextbook)§  Inshort:onsplit

– atleaflevel:COPYmiddlekeyupstairs– atnon-leaflevel:pushmiddlekeyupstairs(asinplainB-tree)

ExampleB+Tree

§  Searchbeginsatroot,andkeycomparisonsdirectittoaleaf

§  Searchfor5*,15*,alldataentries>=24*...

Based on the search for 15*, we know it is not in the tree!

17 24 30

2* 3* 5* 7* 14* 16* 19* 20* 22* 24* 27* 29* 33* 34* 38* 39*

InsertingaDataEntryintoaB+Tree

§  FindcorrectleafL.§  PutdataentryontoL.

–  IfLhasenoughspace,done!– Else,mustsplitL(intoLandanewnodeL2)

•  Redistributeentriesevenly,copyupmiddlekey.

§  parentnodemayoverflow– butthen:pushupmiddlekey.Splits“grow”tree;rootsplitincreasesheight.

ExampleB+Tree–Inserting30*

2* 3* 5* 7* 14* 16* 19* 20* 22* 24* 27* 29*

ExampleB+Tree–Inserting30*

2* 3* 5* 7* 14* 16* 19* 20* 22* 24* 27* 29*

23* 30*

ExampleB+Tree-Inserting8*

2* 3* 5* 7* 14* 16* 19* 20* 22* 24* 27* 29*

NoSpace

ExampleB+Tree-Inserting8*Root

2* 3* 5* 7* 14* 16* 19* 20* 22* 24* 27* 29*

2* 3* 5* 14* 16* 19* 20* 22* 24* 27* 29* 23* 7* 8*

13 17 24

SoSplit!

2* 3* 5* 7* 14* 16* 19* 20* 22* 24* 27* 29*

2* 3* 5* 14* 16* 19* 20* 22* 24* 27* 29* 23* 7* 8*

13 17 24

SoSplit!

AndthenpushmiddleUP

2* 3* 5* 7* 14* 16* 19* 20* 22* 24* 27* 29*

2* 3* 14* 16* 19* 20* 22* 24* 27* 29* 23* 7* 8*

5 13 17 24

<5 >=5

FinalState

14* 16* 19* 20* 22* 24* 27* 29* 7* 5* 8*

13 17 24

2* 3* 14* 16* 19* 20* 22* 24* 27* 29* 7* 5* 8* 23*

14* 16* 19* 20* 22* 24* 27* 29* 7* 5* 8*

13 17 24

2* 3* 14* 16* 19* 20* 24* 27* 29* 7* 5* 8* 21* 22* 23*

17 21 24 13 5 RootisFull,sosplitrecursively

ExampleB+Tree:Recursivesplit

•  Notice that root was also split, increasing height.

14* 16* 19* 20* 21* 22* 23* 24* 27* 29*

7* 5* 8*

Example:Datavs.IndexPageSplit

§  leaf:‘copy’§  non-leaf:‘push’

§  whynot‘copy’@non-leaves?

2* 3* 5* 7* 8*

5 21 24

… 2* 3* 5* 7*

17 21 24 13

Data Page Split

Index Page Split

SameInserting21*:TheDeferredSplit

14* 16* 19* 20* 22* 24* 27* 29* 7* 5* 8*

13 17 24

Notethishasfreespace.So…

Inserting21*:TheDeferredSplit

14* 16* 19* 20* 22* 24* 27* 29* 7* 5* 8*

13 17 24

LENDkeystosibling,throughPARENT!

14* 16* 19* 20* 21* 23* 24* 27* 7* 5* 8*

13 17 23

22* 29*

Inserting21*:TheDeferredSplit

14* 16* 19* 20* 22* 24* 27* 29* 7* 5* 8*

13 17 24

Shorter,morepacked,fastertree

14* 16* 19* 20* 21* 23* 24* 27* 7* 5* 8*

13 17 23

22* 29*

Insertionexamplesforyoutotry

14* 16* 21* 22* 23*

7* 5* 8*

20 … (not shown)

Insert the following data entries (in order): 28*, 6*, 25*

Answer…

7* 8* 14* 16*

13 …

After inserting 28*, 6*

After inserting 25*

21* 22* 23* 28*

Answer…

7* 8* 14* 16* 21* 22* 23* 25* 28*

After inserting 25*

DeletingaDataEntryfromaB+Tree

§  Startatroot,findleafLwhereentrybelongs.§  Removetheentry.

–  IfLisatleasthalf-full,done!–  IfLunderflows

•  Trytore-distribute,borrowingfromsibling(adjacentnodewithsameparentasL).

•  Ifre-distributionfails,mergeLandsibling.–  updateparent–  andpossiblymerge,recursively

DeletionfromB+Tree

Root 17

14* 16* 19* 20* 22* 24* 27* 29* 33* 34* 38* 39*

7* 5* 8*

Example:Delete19*&20*

Deleting19*iseasy:

Root 17

14* 16* 19* 20* 22* 24* 27* 29* 33* 34* 38* 39*

7* 5* 8*

Root 17

14* 16* 33* 34* 38* 39*

7* 5* 8* 22* 24*

27* 29*

20* 22*

•  Deleting20*->re-distribution(notice:27copiedup)

Root 17

14* 16* 33* 34* 38* 39*

7* 5* 8* 22* 24*

27* 29*

...AndThenDeleting24*

Root 17

14* 16* 33* 34* 38* 39*

7* 5* 8* 22* 27*

•  Mustmergeleaves:OPPOSITEofinsert

Root 17

14* 16* 33* 34* 38* 39*

7* 5* 8* 22* 24*

27* 29*

...AndThenDeleting24*

Root 17

14* 16* 33* 34* 38* 39*

7* 5* 8* 22* 27*

•  Mustmergeleaves:OPPOSITEofinsert

…butarewedone??

...MergeNon-LeafNodes,ShrinkTree

Root 17

14* 16* 33* 34* 38* 39*

7* 5* 8* 22* 27*

2* 3* 7* 14* 16* 22* 27* 29* 33* 34* 38* 39* 5* 8*

Root 30 13 5 17

ExampleofNon-leafRe-distribution

§  Treeisshownbelowduringdeletionof24*.§  Now,wecanre-distributekeys

13 5 17 20

14* 16* 17* 18* 20* 33* 34* 38* 39* 22* 27* 29* 21* 7* 5* 8* 3* 2*

AfterRe-distribution

§  needonlyre-distribute‘20’;did‘17’,too§  whywouldwewanttore-distributemorekeys?Ans:reduceslikelihoodofsplit(seeBook,pg.356)

14* 16* 33* 34* 38* 39* 22* 27* 29* 17* 18* 20* 21* 7* 5* 8* 2* 3*

30 20 22

Mainobservationsfordeletion

§  Ifakeyvalueappearstwice(leaf+nonleaf),theabovealgorithmsdeleteitfromtheleaf,only

§  whynotnon-leaf,too?

Mainobservationsfordeletion

§  Ifakeyvalueappearstwice(leaf+nonleaf),theabovealgorithmsdeleteitfromtheleaf,only

§  whynotnon-leaf,too?§  ‘lazydeletions’-infact,somevendorsjustmarkentriesasdeleted(~underflow),– andreorganize/compactlater

Recap:mainideas

§  onoverflow,split(and‘push’,or‘copy’)– orconsiderdeferredsplit

§  onunderflow,borrowkeys;ormerge– orletitunderflow...

B+TreesinPractice

§  Typicalorder:100.Typicalfill-factor:67%.– averagefanout=2*100*0.67=134

§  Typicalcapacities:– Height4:1334=312,900,721entries– Height3:1333=2,406,104entries

B+TreesinPractice

§  Canoftenkeeptoplevelsinbufferpool:– Level1=1page=8KB– Level2=134pages=1MB– Level3=17,956pages=140MB

BulkLoadingofaB+Tree

§  Inanemptytree,insertmanykeys§ Whynotone-at-a-time?

– Tooslow!

§  Initialization:Sortalldataentries§  scanlist;wheneverenoughforapage,pack§  <repeatforupperlevel>

3* 4* 6* 9* 10* 11* 12* 13* 20* 22* 23* 31* 35* 36* 38* 41* 44*

Sorted pages of data entries; not yet in B+ tree Root

3* 4* 6* 9* 10* 11* 12* 13* 20* 22* 23* 31* 35* 36* 38* 41* 44*

Data entry pages not yet in B+ tree 35 23 12 6

3* 4* 6* 9* 10* 11* 12* 13* 20* 22* 23* 31* 35* 36* 38* 41* 44*

not yet in B+ tree Data entry pages

ANoteon`Order’

§  Order(d)conceptreplacedbyphysicalspacecriterioninpractice(`atleasthalf-full’).

§ Manyrealsystemsareevensloppierthanthis:theyallowunderflow,andonlyreclaimspacewhenapageiscompletelyempty.

§  (whatarethebenefitsofsuch‘slopiness’?)

Conclusions

§  B+treeistheprevailingindexingmethod§  Excellent,O(logN)worst-caseperformanceforins/del/search;(~3-4diskaccessesinpractice)

§  guaranteed50%spaceutilization;avg69%

Conclusions

§  Canbeusedforanytypeofindex:primary/secondary,sparse(clustering),ordense(non-clustering)

§  Severalfine-extensionsonthebasicalgorithm– deferredsplit;– bulk-loading

CS 4604: Introduction to Database Management...

Documents

OUTLINE — August 29, 2005olney/fall18/econ1/lect12052018.pdf · Externalities Monopoly & Monopolistic Competition Asymmetric Information Distribution Counterfactual To properly

Paging an Virtual Memory Systemscourses.cs.vt.edu/~cs3204/spring2001/cstruble/notes/chapter12.pdf · Thrashing occurs when a job continuously references pages which are not in main

CS 4604: Introduction to Database Management Systemscourses.cs.vt.edu/~cs4604/Fall18/lectures/lecture-2.pdf · 2018-12-07 · • Conceptual model: In databases, structures are at

CS4604:Introduconto# Database#Management#Systems#courses.cs.vt.edu/~cs4604/Spring15/lectures/lecture-12.pdf · The’predominantchoice’in’storing’data ... VertexDB,Neo4j ’

CS 4604: Introducon to Database Management Systemscourses.cs.vt.edu/~cs4604/Spring16/lectures/lecture-1.pdf– Silberschatz, Korth and Sudarshan, 6th Ed. Prakash 2016 VT CS 4604 3

CS 4604: Introduction to Database Management Systemscourses.cs.vt.edu/~cs4604/Fall18/lectures/lecture-1.pdf§ Project is spread over 3 deliverables ... – Search engines, banking

CS 4284 Operating Systemscourses.cs.vt.edu/cs4284/spring2015/lectures/Lecture-FileSystems-Advanced.pdf · –Some filesystem reallocate blocks to improve spatial locality • Preallocation

CS4604:Introduconto# Database#Management#Systems#courses.cs.vt.edu/~cs4604/Spring15/lectures/lecture-1.pdf · CourseOutline! Weeks’1–4:’Query/ Manipulaon’Languages’ and’DataModeling’

Chapter 17 – Introduction to Distributed Systemscourses.cs.vt.edu/~cs3204/spring2004/Notes/OS3e_17.pdf · Chapter 17 – Introduction to ... 17.8 Case Study: The Amoeba Distributed

Introduction to CS 4604courses.cs.vt.edu/~cs4604/Fall09/lectures/lecture-01-introduction.pdf · About the Course Introduction to Databases Course Information I Instructor I T. M

CS 4604: Introducon to Database Management Systemscourses.cs.vt.edu/~cs4604/Spring16/lectures/lecture-12.pdf · CS 4604: Introducon to Database Management Systems B. Aditya Prakash

CS4604:Introduconto# Database#Management#Systems#courses.cs.vt.edu/~cs4604/Spring15/lectures/lecture-8.pdf · ﬁle’scan’retrieving’all’records.’ – Sorted’Files:’

Braveheart Fall18 2 - flatheadcountysheriff.comflatheadcountysheriff.com/wp-content/uploads/2018/11/Braveheart-Fall18-2.pdf · Braveheart Chaplain Ministry Fall 2018 Give thanks to

Syllabus Biol5425 Fall18 FINAL - student.apps.utah.edu

CS4604:Introduconto# Database#Management#Systems#courses.cs.vt.edu/~cs4604/Spring15/lectures/lecture-2.pdf · How’do’we’generalize’this’schemato’handle’data over’more’than’one’semester?

CS4604:Introduconto# Database#Management#Systems#

CS4604:Introduconto# …courses.cs.vt.edu/~cs4604/Spring15/lectures/lecture-5.pdfE/R:#NOT#IN#BOOK!#! IMPORTANT:’ – Follow’only’lecture’slides’for’this’topic!’ –

CS4604:Introduconto# Database#Management#Systems#cs4604/Spring15/lectures/... · 2015. 5. 12. · CS4604:Introduconto# Database#Management#Systems# B.#Aditya#Prakash# Lecture’#17:’Transac0ons’2:’2PL’and’

CS 4604: Introduction to Database Management Systemscourses.cs.vt.edu/~cs4604/Fall18/lectures/lecture-10.pdf§ Issue is removing duplicates. § Basic approach: sorting – 1. Scan

CS 4604: Introducon to Database Management Systemscourses.cs.vt.edu/~cs4604/Spring16/lectures/lecture-17.pdfTwo-Phase Locking (2PL), cont. 2PL on its own is suﬃcient to guarantee