Introducing Kudu and Recordservice

1©Cloudera,Inc.Allrightsreserved.

IntroducingApacheKuduandRecordService(incubating)GuidoOswald|SalesEngineer,SwitzerlandApril2016,SwissBigDataUserGroupMeetup18

@GuidoOswald


CurrentstoragelandscapeinHadoopHDFSexcelsat:• Efficientlyscanninglargeamounts

ofdata• Accumulatingdatawithhigh

throughputHBaseexcelsat:• Efficientlyfindingandwriting

individualrows• Makingdatamutable

Gapsexistwhenthesepropertiesareneededsimultaneously


Managingthegap(today)

CodeComplexity•ManageflowandsyncofdatabetweenHDFSandHbase

MonitoringandSecurity•Managingconsistentbackups,securitypolicies,monitoringandmoreishard

Performance• SignificantlagbetweenarrivalofHbase data“staging”andtimewhendataisavailableforanalytics.


Changinghardwarelandscape

• Spinningdisk->solidstatestorage• NANDflash:Upto450kread250kwriteIOPS,about2GB/secreadand1.5GB/secwritethroughput, atapriceoflessthan$3/GBanddropping• 3DXPoint memory(1000xfasterthanNAND,cheaperthanRAM)

• RAMischeaperandmoreabundant:• 64->128->256GBoverlastfewyears

• Takeaway1: The nextbottleneckisCPU,andcurrentstorageheavyapplicationsweren’tdesignedwithCPUefficiencyinmind• Takeaway2: Columnstoresarefeasibleforrandomaccess


ApacheKudu(Incubating)StorageforFastAnalyticsonFastData

• NewupdatingcolumnstoreforHadoop• Simplifiesthearchitectureforbuildinganalyticapplicationsonchangingdata

• Designedforfastanalyticperformance• NativelyintegratedwithHadoop

• DonatedasincubatingprojectatApacheSoftwareFoundation(November17,2015)

• Betanowavailable

STRUCTUREDSqoop

UNSTRUCTUREDKafka,Flume

PROCESS,ANALYZE,SERVE

UNIFIEDSERVICES

RESOURCEMANAGEMENTYARN

SECURITYSentry,RecordService

FILESYSTEMHDFS

RELATIONALKudu

NoSQLHBase

STORE

INTEGRATE

BATCHSpark,Hive,PigMapReduce

STREAMSpark

SQLImpala

SEARCHSolr

SDKKite


• Highthroughputforbigscans(columnarstorageandreplication)Goal:Within2xofParquet

• Low-latency forshortaccesses(primarykeyindexesandquorumdesign)Goal: 1msread/writeonSSD

• Database-likesemantics(initiallysingle-rowACID)

• Relational datamodel• SQLquery• “NoSQL”stylescan/insert/update(Javaclient)

Kududesigngoals


Kudubasicdesign• Apache-licensedopensourcesoftware

• Structureddatamodel

• Basicconstruct:tables• Tablesbrokendownintotablets(roughlyequivalenttopartitions)

• Architecturesupportsgeographicallydisparate,active/activesystems• Nottheinitialdesigngoal


WhatKuduisnot• Not aSQLinterface• Justthestoragelayer• “BYOSQL”– Bring-your-ownSQL

• Notafilesystem• Datamust havetabularstructure

• Not anapplicationthatrunsonHDFS• Analternative,native Hadoop storageengine

• Not areplacementforHDFSorHBase• Selecttherightstoragefortherightusecase• Clouderawillcontinuetosupportandinvestinallthree


Kududatamodel

• TableshaveaRDBMS-likeschema• Finitenumberofcolumns(unlikeHBase/Cassandra)• Types:BOOL, INT8/16/32/64, FLOAT, DOUBLE, STRING, BINARY, TIMESTAMP• Somesubsetofcolumnsmakeupaprimarykey

• Fastrandomreads/writesbyprimarykey• Nosecondaryindexes(yet)

• Columnar layoutondisk- Parquet• Lazymaterialization• Encodingandcompressionoptions

9


Tablepartitioning

• Hashbucketing• Distributerecordsbyhashofpartitioncolumn(s)• NbucketsleadstoNtablets

• Rangepartitioning• Distributerecordsbyrangesofthepartitioncolumn(s)• NsplitkeysleadstoNtablets

• Canbeamixfordifferentcolumnsoftheprimarykey


Consistencymodel

• ConsistencyandreplicationenforcedbyRaftconsensus(similartoPaxos)• Replicationbyoperationnotdata

• Single-rowtransactionsnow•Multi-rowtransactionslater

• Geo-distributedreplicaswillbepossibleunderstricttimesynchronization

• TechniquesdrawnfromGoogleSpannerandothers


Kuduinterfaces

• NoSQL-styleAPIs• Insert(),Update(),Delete(),Scan()• JavaandC++now• Pythonsoon

• Integrations withMapReduce,Spark,andImpala

• NodirectaccesstounderlyingKudutabletfiles

• Beta does not haveauthentication,authorization,encryption


Impalaintegration

• OpensupKudutoJDBC/ODBCclients

• IntuitivewaytogetdataintoKudu• INSERT INTO kudu_table SELECT * FROM src_table;

• Additionalcommands• UPDATE• DELETE• Efficient INSERT VALUES

• RunsontheKuduC++client


Performancecharacteristics

VeryCPUefficient•WritteninmodernC++,usesspecializedCPUinstructions,JITcompilationwithLLVM

Latencydependentonstoragehardwarecapabilities• Expectsub-millisecondresponseonSSDsandupcomingtechnologies

Nogarbagecollectionallowsverylargememoryfootprintwithnopauses

Bloomfiltersreducetheneedformanydiskaccesses


OperatingKudu

• EasiestthroughClouderaManagerintegration• Separateparcelfornow

• Kuduisalwayscompacting• Nominorvs.majorcompaction• Nocompactionlatencyspikes

•WebUIisfullofmetricsandlogs


Clusterlayout

• Oneormultiplemasters• Onlyoneincurrentbeta• LowCPUandmemoryimpact

• Onetabletserverperworkernode• CansharediskswithHDFS• OneSSDperworkernodejustforKuduWALcanspeedupwrites

• NodependenciesonotherHadoopecosystemcomponents• ButinterfacingcomponentslikeImpalaorSparkdo


Real-timeanalyticsinHadooptodayMerginginnewdata=storagecomplexity

Downsides:

● Multiple storagelayers

● Latestdataishidden

● Filesaremessy

● ComplextodoupdateswithoutbreakingrunningqueriesNewPartition

MostRecentPartition

HistoricData

HBase

ParquetFile

Haveweaccumulatedenoughdata?

ReorganizeHBasefileintoParquet

• Waitforrunning operationstocomplete• DefinenewImpalapartitionreferencingthenewlywrittenParquetfile

IncomingData(MessagingSystem)

ReportingRequest

HDFS+Impala


Real-timeanalyticsinHadoopwithKudu

Improvements:

● Onesystemtooperate

● Noschedulesorbackgroundprocesses

● Handlelatearrivalsordatacorrectionswithease

● Newdataavailableimmediatelyforanalyticsoroperations

HistoricalandReal-timeData

IncomingData(MessagingSystem)

ReportingRequest

Kudu+Impala


Kudufordatawarehousing

• Nearrealtimedatavisibility• BItoolscandisplayeventsthathappenedsecondsearlier

• Excellentforstarschemas• Fastscansofdeepfacttables• Efficientwidefacttables• Simplifiedupdatesofslowlychangingdimensions


NearrealtimedatawarehousingonKudu

Files

RDBMS

Streams

KAFKA

KUDU

IMPALA

HUE

BI tools

User

FLUME

SPARKSTREAMING

Simple

Complex


Resources

Jointhecommunityhttp://[email protected]

Downloadthebetacloudera.com/downloads

Readthewhitepapergetkudu.io/kudu.pdf


CreatingaKudutable

KuduStoragehandler

TablenameinImpaladoesNOTmatchtablenameinKudu.Kuduisitsownstoragelayer.

KuduMasterhostnameandport

Aprimarykeyismandatory


Spark(Scala)code DataFrame Row

KuduMasterhostnameandport

Kudu tablename

Createaclient,sessionandtableobject

Extractvaluesfromtherow, strongtypes

Createaninsertobjectandrow

Setthevaluesbytype,columnnameandcolumnvalule

Performtheactualinsert

Cleanup


Kuducodeexamplesanddocs

https://github.com/cloudera/kudu-examples

http://www.cloudera.com/documentation/betas/kudu/0-7-0/topics/kudu_development.html

http://getkudu.io/docs/developing.html


RecordService


PermissionEnforcementtoday withSentryHive

Server2

Sentry

Enforcem

ent

Impala

HDFS:MR,Pig,Spark,...

Search(Solr)

SentryPermissions

rules

Rule:“Allowfraudanalystsreadaccesstothetransactiontable”

Adminsspecify permissions

Sentry

Enforcem

ent

Sentry

Enforcem

ent

Sentry

Enforcem

ent

Apps:Datameer,Platfora,Zoomdata,

etc

SentryService

Coarsegrained(table)


TheNeedforFine-GrainedAccessControlAcrossallaccesspaths

Columns:Sensitivecolumnvisibilityvaries;Example:creditcardnumbers• Managers:1234567812345678• CallCentre:XXXXXXXXXXXX5678• Analysts:XXXXXXXXXXXXXXXX• Others:Doesnotseecreditcardcolumn

Rows:Differentgroupsofusersneedaccesstodifferentrecords• Europeanprivacylaws• Governmentsecurityclearance• Financialinformationrestrictions


Theworkaround

Date/time Accnt# NationalIdentifier

Asset Trade Broker

09:33:1116-Feb-2015

0234837823 238-23-9876

ABC Sell group1

11:33:0116-Feb-2015

3947848494 329-44-9847

TBT Buy group2

14:12:3416-Feb-2015

4848367383 123-56-2345

DEF Sell group3

09:22:0316-Feb-2015

3485739384 585-11-2345

INTC Buy group1

11:55:3316-Feb-2015

3847598390 234-11-8765

F Buy group1

10:22:5516-Feb-2015

8765432176 344-22-9876

UA Buy group3

13:45:2416-Feb-2015

3456789012 412-22-8765

XYZ Sell group2

09:03:4416-Feb-2015

4857389329 123-44-5678

TMV Buy group1

15:55:5516-Feb-2015

4756983234 234-76-9274

MA Buy group3

Date/time Accnt # NationalIdentifier

Asset Trade Broker

14:12:3416-Feb-2015

4848367383 123-56-2345

DEF Sell group3

10:22:5516-Feb-2015

8765432176 344-22-9876

UA Buy group3

15:55:5516-Feb-2015

4756983234 234-76-9274

MA Buy group3


Asset Trade Broker

11:33:0116-Feb-2015

3947848494 329-44-9847

TBT Buy group2

13:45:2416-Feb-2015

3456789012 412-22-8765

XYZ Sell group2


Asset Trade Broker

09:33:1116-Feb-2015

0234837823 238-23-9876

ABC Sell group1

09:22:0316-Feb-2015

3485739384 585-11-2345

INTC Buy group1

11:55:3316-Feb-2015

3847598390 234-11-8765

F Buy group1

09:03:4416-Feb-2015

4857389329 123-44-5678

TMV Buy group1

Splittheoriginal file;UseHDFSpermissions tolimitaccess

WhatifonlysomebrokersineachgroupareallowedtoseefullIDs?


TheSolution• Applycontrolstothemasterdatafile• Row,column,andsub-column (masking)controls• Ability toenforcetheseacrossaccesspaths

WhatAllGroup1BrokersSee: Date/time Accnt# NationalIdentifier

Asset Trade Broker

09:33:1116-Feb-2015

0234837823 XXX-XX-9876 ABC Sell group1

09:22:0316-Feb-2015

3485739384 XXX-XX-2345 INTC Buy group1

11:55:3316-Feb-2015

3847598390 XXX-XX-8765 F Buy group1

09:03:4416-Feb-2015

4857389329 XXX-XX-5678 TMV Buy group1


RecordService(Beta)toEnforceColumnandRow-levelRules

Hadoopcomponents:MR,Pig,

Spark,Solr,HiveServer2,Impala...

Applications:Datameer,Platfora,etc

SentryPermissions

rules

Rule:AllowmanagerstoseeNationalIDs.

Permissionsspecifiedbyadministrators(top-levelanddelegated)

HDFS

RecordService

Hbase

SentryService

AWSS3


BenefitsofRecordServiceSecurity• Fine-graineddatapermissionsandenforcementacrossHadoop• IntegrationwithSentryforpolicystorage andimplementation

Interoperability• Clientsnolongerneedtobeawareofon-diskformat

• Singledataaccesspathmeanssingleplacetoimplementandtestfileformatrelatedchanges• Transparentlyswapcomponentsaboveorbelow(ex.HDFS->S3)

Performance/Efficiency• PerformanceboostedviaImpala’soptimizedscanner,dynamiccodegeneration,parquetimplementation• Useprojectionsoveroriginalsourcedatasetsinsteadofmakingsomanycopies/subsets


RecordServiceArchitecture

RecordServicePlanner

HDFSNN

SentryService

HiveMetastore

Client

RecordServiceWorker

HDFSDN HBaseRS S3

ClientTask

ClientTask

ClientTask

1

2 3

Request:- Objectstoaccess- UserinfoResponse:- Listofsplits- Delegationtoken

Job launchesasnormal Client tasksreadrecords fromRecordServiceWorker

Notyetsupported


EnforcingSentryPermissionsforMR/Spark

CreateaviewinHMSwiththenecessarycolumn/rowrestrictions

Createaroleandassigntoagroup

Grantaccessprivilegetothatrole

CREATE ROLE demorole;GRANT ROLE demorole to GROUP demogroup;

CREATE VIEW nation_names AS SELECT n_nationkey, n_name FROM tpch.nation;

GRANT SELECT ON TABLE tpch.nation_names TO ROLE demorole;


SparkUsageExample:RDD

ImportRecordServicepackage

ReaddataintoavariableusingRecordServiceAPI

Performanaction

scala> val data = sc.recordServiceRecords("select * from tpch.nation_names");

data: org.apache.spark.rdd.RDD[Array[org.apache.hadoop.io.Writable]] = RecordServiceRDD[0] at RDD at RecordServiceRDDBase.scala:57

scala> import com.cloudera.recordservice.spark._;

import com.cloudera.recordservice.spark._

scala> data.count();

res0: Long = 25


CurrentFeatureAvailabilityCompute:• Support forMR (InputFormat) and Spark (RDDs, SparkSQL DataFrames)

Storage:• Support for reading HDFS or S3 of file format: Parquet, Text, Sequence File, RC, Avro

Data Types:• INT(8-64bits),CHAR/VARCHAR,BOOL,FLOAT,DOUBLE,DECIMAL,STRING,TIMESTAMP• NosupportforLOBsorNestedTypes

Scalability:• Testedupto80large/powerfulnodes• Validatedagainst1trillion row(100TB)TeraSortdataset/workload• Metadataupto1Mblocks(planningonly)• NotethatTPC-DSrunonSparkSQLat500GBscalepointran15%fasterwithRecordService

Security:• Authentication:Kerberos/LDAP/AD• Authorization:Sentrytablelevelprivileges, columnandrow-levelprivileges usingHMSviews.• Delegation token+taskencryptionforsecure taskexecution


CurrentLimitations

SecurityLimitations• Onlysupportssimplesingle-tableviews(nojoinsoraggregations).• SSLsupport hasnotbeentested.• Oozieintegrationhasnotbeentested.• UDFsarenotsupported

Storage/File Format Limitations• Nosupportforwritepath.• UnabletoreadfromKuduorHBase.

Operation andAdministration Limitations• Nodiagnosticbundlesupport.• NometricsavailableinCM.

Application Integration Limitations• Spark DataFrame not well tested.

See in http://cloudera.github.io/RecordServiceClient/ReleaseNotes.html


InstallationandPlatformSupport

Installation• Support CSDinstallationonCDH5.4+• Parcels,viaCM• Packages• QuickStartVM• ClientJARs

Platform/HardwareSupport• Serversupport:RHEL5-7,Ubuntu LTS,SLES,Debian• IntelNehalem(orlater)orAMDBulldozer(orlater)processor• 64GBmemory• Foroptimalperformance,runwith12ormoredisksoruseSSD.

OperationandAdministration• DirectlyfromRecordService:

• MetricsexposedviaaRecordServicewebapp• ProfilesforrequestsviaRecordServicewebapp

• FromCM:• Basicservicemanagement(start/stop/restart)andbasichealthchecksviaCM(processavailability).• AbilitytodeployRecordServicePlanner,Worker,orPlanner+Worker roles.


Resources

• RecordServiceBetaDocs• Featurelist• RecordService SourceCode• RecordServiceClientlibraries


[email protected]

Documents

Introducing Kudu and Recordservice