Dremel: Interactive Analysis of Web-Scale Datasets CS 744 BIG DATA

PHIL MARTINKUS

Motivation  LargeScaleDatamustbeaccessibletoanalystsandengineers

 Interactivequeriesareimportantfordataexploration,monitoring,onlinecustomersupport,rapidprototyping,debuggingofdata,andothertasks

 Manydatabasesrequireacostlyloadingphase

 Webdataisoftennon-relational

The Solution: Dremel  Dremelisasystemthatsupportsinteractiveanalysisofverylargedatasetsoversharedclustersofcommoditymachines.

 HasbeeninproductionatGooglesince2006 Canoperateondatainplaceusingadistributedstoragesystem

 Usesanovelcolumnarstorageformatfornesteddata

 Providesahigh-levelSQL-likelanguageforinteractivequeries

Data Model  Strongly-typednestedrecords Recordsconsistofoneormorefields

 Fieldscanberequired,optionalorrepeated

Data Model Example  EachRecordrepresentsadocument

 RequiredDocIdfield Linksisanoptionalgroupwithtwonestedrepeatedfields.

 NameisarepeatedgroupwithanestedLanguagegroup.

Nested Columnar Storage  Allvaluesofafieldarestoredconsecutivelyinblocks

 Goalsforthestoragesystem:◦ Losslessrepresentationofrecordstructureincolumnarformat

◦ Fastencodings◦ Efficientrecordassembly

 Repeatedrecordsarehandledwithrepetitionanddefinitionlevels

Repetition Levels  Usedtodisambiguateoccurrencesofthesamefieldwithinthesamerecord

 Tellusatwhatrepeatedfieldinthefield'spaththevaluehasrepeated

Repetition Level Example

Definition Levels  Wheneveranoptionalorrepeatedfieldisnotpresentinarecord,thesystemstoresaNULL.

 Tellushowmanyfieldsinthefield'spaththatcouldbeundefined(becausetheyareoptionalorrepeated)areactuallypresentintherecord.

 MostlyusefulfordistinguishingNULLvalues.

Definition Level Example

Splitting Records into Columns  Recursivealgorithmcomputeslevelsforeachfield

 Atreeoffieldwritersmatchthestructureofthefieldschema

 ManydatasetsatGooglearesparse

Record Assembly  Goalistoreconstructrecordsgivenasubsetoffields

 Finitestatemachine(FSM)readsvaluesandappendstooutputrecords

 AnFSMstatecorrespondstoafieldreader

 TheFSMistraversedfromthestartstatetotheendstateforeachrecord

Query Language  BasedonSQL Designedforcolumnarnestedstorage

Query Execution  UsesaTreearchitecture Rootreceivesincomingqueries

 Intermediateserversrewritethequery

 Leafserversaccessdata Eachserverhasaninternaltreecorrespondingtoaphysicalqueryexecutionplan.

Query Execution Example

Queryisrewritten

Querysenttoroot

Querysenttoleafnodes

Asetofiteratorsscantheinputcolumninlockstepandemitresultswithannotatedrepetitionanddefinitionlevelswithoutactuallyassemblingtherecords

Query Dispatcher  Dremelisamulti-usersystemthatexecutesqueriessimultaneously

 Thequerydispatcherschedulesqueries Dealingwithstragglers◦ Disproportionallyslowprocessesarerescheduledonanotherserver◦ Aparameterspecifiestheminimumpercentageoftabletsthatmustbescannedbeforereturningaresult

Experiments

Questions?