Migrating from Relational - Data Modeling and Access

©2016CouchbaseInc.

Migratingfromrelationaldatamodelingandaccess

BrantBurnett,LeadDeveloper,CenterEdge

ClarenceTauro,SrTrainer,Couchbase

MarcoGreco,SrEngineer,N1QLR&D,Couchbase1

©2016CouchbaseInc.©2016CouchbaseInc.

Agenda

•  Practicalconsiderationsfordataandapplicationmigration

•  ModelinginCouchbase

•  Reallifeexperience:Centeredge

2

©2016CouchbaseInc. 3

PracticalConsiderations


Inthissection

•  Nomenclature

•  Typeanddatamodelmapping

•  Migratingdata

•  Businesslogic•  Monitoring

4


Nomenclature

5

Oracle Couchbase

Database Bucket

Table Bucket

Row Document

Column Field


TypeMapping

6

Oracle (PL/SQLSynonyms) Couchbase

Number,Binary_real,Binary_integer

Smallint,Int,Dec,Decimal,Float,…

Number

Char,Nchar,Varchar2,Nvarchar2

Character,String String

Boolean Boolean

Date,Timestamp HandledviaString

Interval(yeartomonth,daytofracPon)

Somesupportvia_millis()funcPons


Modelling

7

CustomerID Name DOB

CBC2016 JaneSmith 1990-01-30

CustomerID Type Cardnum Expiry

CBC2016 visa 5827… 2019-03

CBC2016 master 6274… 2018-12

CustomerID ConnId Name

CBC2016 XYZ987 JoeSmith

CBC2016 SKR007 SamSmith

CustomerID item amt

CBC2016 mac 2823.52

CBC2016 ipad2 623.52

CustomerID ConnId Name

CBC2016 XYZ987 JoeSmith

CBC2016 SKR007 SamSmith

Contacts

Customer

Billing

ConnecPonsPurchases

{"Name":"JaneSmith","DOB":"1990-01-30","Billing":[{"type":"visa","cardnum":"5827-2842-2847-3909","expiry":"2019-03"},{"type":"master","cardnum":"6274-2842-2847-3909","expiry":"2019-03"}],"ConnecPons":[{"CustId":"XYZ987","Name":"JoeSmith"},{"CustId":"PQR823","Name":"DylanSmith"}{"CustId":"PQR823","Name":"DylanSmith"}],"Purchases":[{"id":12,item:"mac","amt":2823.52}{"id":19,item:"ipad2","amt":623.52}]}

DocumentKey:CBC2016


Migration

•  Generalizedprocess•  Commercialtools

•  Talend•  Informatica

•  Opensource•  Couchbasejavaimporter•  Oracle2couchbase•  SQSL

•  Importingfromfiles

8


MigrationProcess

•  HighlevelprocesstomigratedatafromRDBMStoCouchbaseusingN1QL

•  Foreachtable•  Determineprimarykeycolumns•  Describetable•  Foreachrow

•  Generatedocumentkeyfromprimarykeycolumns•  Generatedocumentfromprojectionlistdescription,columnvalues•  INSERTINTO<bucket>(key,value)($1,$2)

•  Usekeyanddocumentasplaceholdervalues

9


Talend

•  TalendconnectorforCouchbase--Talend5.3orlater•  http://developer.couchbase.com/documentation/server/4.5/

connectors/talend/talend.html•  Ingestingunstructureddata•  Couchbaseviewsupport•  SeamlessintegrationwithCouchbase

•  tCouchbaseInput•  IncomingdatatransformedintoJSONdocumentsandstoredin

Couchbase.•  UserdefinesthedatafieldstobetransformedintoJSONattributes

•  tCouchbaseOutput:usestheschemamappingtotransformJSONdocumentsintotargetdataformats

•  ODBC/JDBCdrivers(providedbySimbaandCData)

10


Informatica

11

•  InformaticaPowerCenter•  NeedsODBCdriver

•  InformaticaCloud

•  NeedsJDBCdriver

•  ODBC/JDBCdrivers(providedbySimbaandCData)

•  ETL&DataIntegration•  LoaddatafromanyRelationalsystemintoCouchbase•  ExportCouchbasedataintoRDBMS•  SeamlesslyintegrateCouchbaseintorestoftheDatafabric


Couchbasejavaimporter

•  BlogpostbyLaurentDoguindetailingjourneyfromprocesstocode

•  Javabased,butprincipleappliestootherlanguages•  GearedtoPostgresbutprincipleappliestootherengines•  Blog:http://blog.couchbase.com/2016/january/moving-sql-database-content-to-couchbase

•  Sourcecode:https://github.com/ldoguin/couchbase-java-importer

12


Oracle2couchbase

•  Anotherblogpost/opensourcetool•  ByManuelHurtado

•  Javabased•  MigratesfromOracle

•  Blog:http://blog.couchbase.com/2016/february/moving-data-from-oracle-to-couchbase

•  Sourceandbinary:https://github.com/mahurtado/oracle2couchbase

13


SQSL

•  ClientsideSQLlikescriptinglanguagedevelopedbyyourstrulytwodecadesago

•  Severalniftyfeatureslike•  Expansion•  Datadrivenoperation•  Ontheflyaggregationandredirection•  Userdefinedroutines

•  HaverecentlywrittendatasourceforCouchbaseandjsonlibrary

•  Source:http://www.sqsl.org

•  Example:letfromconn="sample";connecttofromconnsourcedb2cli;connectto"couchbase://192.168.1.104:8091"sourcecb;select*fromdb2inst1.deptconnectionfromconninsertintodefault(key,value)values($1,$2)usingjson:key("::",columns),json:row2doc(displaylabels,columns);

14


Import/Exportutilities

•  Upcomingversionincludescbimport&cbexport

•  Filebasedutilities•  NeedtoexportRDBDMSdatatofilefirst

•  LoaddirectlyintodatastorebypassingN1QL

15


Businesslogic

•  DDL•  Views

•  Triggers•  Procedures•  Sequences•  Joins•  Transactions

16


Languagecomparison

17

QueryFeatures SQLonRDBMS N1QL

DML §  SELECT,INSERT,UPDATE,DELETE,MERGE §  SELECT,INSERT,UPDATE,DELETE,MERGE

DDL§  CREATE[INDEX,PROCEDURETABLE,TYPE,VIEW…]§  ALTER[TABLE,TYPE,…]§  DROP[INDEX,PROCEDURETABLE,TYPE,VIEW…]

§  CREATE[PRIMARY]INDEX§  DROP[PRIMARY]INDEX

QueryOperaDons§  Select,Join,Project,Subqueries§  StrictSchema§  StrictTypechecking

§  Select,Join,Project,Subqueriesü  Nest&Unnestü  LookMa!NoTypeMismatchErrors!§  JSONkeysactascolumns

Schema §  PredeterminedColumns ü  FullyaddressableJSONü  Flexibledocumentstructure

DataTypes §  SQLDatatypes§  ConversionFuncPons

§  JSONDatatypes§  ConversionFuncPons

QueryProcessing §  INPUT:SetsofTuples§  OUPUT:SetofTuples

§  INPUT:SetsofJSON§  OUTPUT:SetofJSON


DDL

•  OnlyCreateIndex/DropIndexexistinN1QL

•  Everythingelseshouldberemovedfromtheapplication

•  Temporarytables•  Materializeresults•  Storeinmemory,or•  Insertmaterializeddocumentinakeyspaceusingadesignated“type”:fieldandaUUID()askey

•  DROP<temporarytable>becomesDELETEFROMkeyspaceWHEREtype=…andID=<UUID>

18


Views

•  Accessunderlyingkeyspacesinstead•  Somethingakintoviewscanbeobtainedwith

•  Viewindexes•  Functionalindexes•  CREATEINDEX…WHEREclauses

19


Statementblocks

•  Handledbytheapplication:•  Triggers•  Procedures

20


Sequences

•  Theeventualpersistenceenginehandlesatomicincrements•  Specialdocumentscanbecreatedwithacounterandaccessedatomically•  Canspecifyadeltaoncreation•  MustbedonefromSDK•  Inpython:

•  N1QLdoesnot•  UseUUID()instead

21


Joins

•  Twotypesofjoins•  Lookup

•  Index

•  Joinsusethedocumentkey•  Joiningsidecanbeanexpression•  Joinedsideisdocumentkey•  Fullexpressionjoinsnotsupported

22


Transactions

•  Intwowords:Noneed

•  Documentmodificationisatomic

•  ConsistencycanbespecifiedattheRESTcalllevelorSDK•  RESTexample

•  Addscan_consistency=[not_bounded|at_plus|request_plus|statement_plus]toRESTparameters

•  CSDKexample•  Uselcb_n1p_setconsistency(…,[LCB_N1P_CONSISTENCY_NONE,

LCB_N1P_CONSISTENCY_RYOW,LCB_N1P_CONSISTENCY_REQUEST,LCB_N1P_CONSISTENCY_STATEMENT])whensettinguprequestargs

23


Monitoring

•  Oracle•  ALTER[system|session]SETtimed_statistics=trueturnsontimedstatisticscollection.•  V$SESSTAT,V$SYSSTAT,V$STATNAMEdynamicperformanceviewsreporttimedstatistics.•  EXPLAINPLANexplainsastatement.

•  MySQL•  SETprofiling=1turnsonprofiling•  SHOWPROFILESdisplaysavailablequeryprofiles•  SHOWPROFILEdisplaystheprofileforaspecificquery•  EXPLAIN<statement>producesqueryplan

•  Couchbase•  system:completed_requestsvirtualkeyspacelistscompletedlongrunningquerieswithtimings

andstatistics•  system:active_requestsvirtualkeyspacelistsactivequerieswithtimingsandstatistics•  EXPLAIN<statement>explainsrequestplanasajsondocument

24


Modeling


WhatisDataModeling?

26

•  Adatamodelisaconceptualrepresentationofthedatastructuresthatarerequiredbyadatabase

•  Thedatastructuresincludethedataobjects,theassociationsbetweendataobjects,andtheruleswhichgovernoperationsontheobjects.


ConceptualDataModeling

•  Defineentities,attributesandtheirrelationships•  Entities:Mainobjectsthataretargetsofyourappsoperateson

•  Attributes:propertiesthatyourapplicationskeeptrackoffortheentity•  Relationships:definitionconnectionstootherentities-1-1,1-many,many-many

Airline

Airport

Landmark

Route Passenger

Flight


PhysicalDataModel

•  PhaseII-Mapentities,attributesandtheirrelationshipstocontainersprovidedbytheunderlyingdatabasesolution

RelaDonalDatabases CouchbaseServer

Databases Buckets

Tables DocumentswithtypedesignatorapributeORCompoundKeys

Rows Items(Key-ValueorKey-Document)

Columns AMributes

Index Index


[email protected]{“name”:…,“flights”:[{“_id”:“route_1000”,“flight”:…,},{“_id”:”route_6421”,“flight”:…,}…],…}

PhysicalDataModeling

route_1000{“id”:”1000”,“airline”:“AF”,“sourceairport”:”TLV”,“desPnaPonairport”:”MRW”,…}

airport_TLV{“id”:”126701”,“airportname”:“TLV”,“geo”:{“lat”:…,“long”:…},…}

Flights


DataModelingApproaches

30

NoSQLRelaxedNormalizaPonschemaimpliedbystructurefieldsmaybeempty,duplicate,ormissing

RelaDonalRequiredNormalizaPon

schemaenforcedbydbsamefieldsinallrecords

•  Minimizedatainconsistencies(oneitem=onelocaPon)•  Reducedupdatecost(noduplicateddata)•  Preservestorageresources

•  OpPmizedtoplanned/actualaccesspaperns•  Flexiblywithsoxwarearchitecture•  Supportsclusteredarchitecture•  Reducedserveroverhead


JSONDesignChoices

31

•  CouchbaseServerneitherenforcesnorvalidatesforanyparticulardocumentstructure

•  ChoicesthatimpactJSONdocumentdesign:–  SingleRootAttributes–  Objectsvs.Arrays–  ArrayElementTypes–  TimestampFormats–  EmptyandNullPropertyValues–  JSONSchema


RootAttributesvs.EmbeddedAttributes

32

•  Thechoiceofhavingasinglerootattributeorthe“type”attributeembedded.


Objectsvs.Arrays

33

•  Thechoiceofhavinganobjecttype,oranarraytype


ArrayElementTypes

Arrayofstrings

Arrayofobjects

34

•  Arrayelementscanbesimpletypes,objectsorarrays:


TimestampFormats

ArrayofPmecomponents

String(ISO8601)

Number(Unixstyle)(Epoch)

•  Workinganddealingwithtimestampshasbeenchallengingeversince

•  Whenstoringtimestamps,youhaveatleast3options:

16


EmptyandNullPropertyValues

36

•  KeepinmindthatJSONsupportsoptionalproperties

•  Ifapropertyhasanullvalue,considerdroppingitfromtheJSON,unlessthere'sagoodreasonnotto

•  N1QLmakesiteasytotestformissingornullpropertyvalues

•  Besureyourapplicationcodehandlesthecasewhereapropertyvalueismissing

SELECT * FROM couchmusic1 WHERE userprofile.address IS NULL; SELECT * FROM couchmusic1 WHERE userprofile.gender IS MISSING;


JSONSchema

37

•  CouchbaseServerpaysabsolutelynoattentiontotheshapeofyourJSONdocumentssolongastheyarewell-formed

•  TherearetimeswhenitisusefultovalidatethataJSONdocumentconformstosomeexpectedshape

•  JSONSchemaisaJSON-basedformatfordefiningthestructureofJSONdata

•  Thereareimplementationsformostpopularprogramminglanguages

•  Learnmorehere:http://json-schema.org


DataNesting(akaDenormalization)

38

•  Relationaldatabasedesignpromotesseparatingdatausingnormalization,whichdoesn’tscale

•  ForNoSQLsystems,weoftenavoidnormalizationsothatwecanscale

•  Nestingallowsrelatedobjectstobeorganizedintoahierarchicaltreestructurewhereyoucanhavemultiplelevelsofgrouping

•  Ruleofthumbistonestnomorethan3levelsdeepunlessthereisaverygoodreasontodoso

•  Youwilloftenwanttoincludeatimestampinthenesteddata


ExampleofDataNesting

•  Playlistwithownerattributecontainingusernameofcorrespondinguserprofile

39

DocumentKey:copilotmarks61569


ExampleofDataNesting

•  Playlistwithownerattributecontainingasubsetofthecorrespondinguserprofile

40

*Notetheinclusionoftheupdatedapribute


ChoiceswithJSONKeyDesign

41

•  Akeyformedofattributesthatexistintherealworld:–  Phonenumbers–  Usernames–  Socialsecuritynumbers–  Accountnumbers–  SKU,UPCorQRcodes–  DeviceIDs


SurrogateKeys

42

•  Weoftenusesurrogatekeyswhennoobviousnaturalkeyexist

•  Theyarenotderivedfromapplicationdata

•  Theycanbegeneratedvalues–  3305311F4A0FAAFEABD001D324906748B18FB24A(SHA-1)–  003C6F65-641A-4CGA-8E5E-41C947086CAE(UUID)

•  Theycanbesequentialnumbers(oftenimplementedusingtheCounterfeatureofCouchbaseServer)–  456789,456790,456791,…


MakingToughChoices

43

•  Wemustalsomaketrade-offsindatamodeling:–  Documentsize–  Atomicity–  Complexity–  Speed


Embedvs.Refer

44

•  Alloftheprevioustrade-offsareusuallyrolledintoasingledecision–whethertoembedorrefer

•  Whentoembed?

•  Whentorefer?


WanttoknowmoreonDataModeling?

45

•  Sessiontomorrow–“AgileDocumentModelsandDataStructures”at1:00PM

©2016CouchbaseInc.©2016CouchbaseInc. 46

Brant Burnett Software Development Team Lead

Couchbase Community Expert


AboutCenterEdgeSoftware

ü  Point of Sale

ü  Admissions & Ticketing

ü  Party, Group & Event Bookings

ü  Online Sales & Party Reservations

ü  Time Clock & Labor Management

ü  & More!

47


AboutCenterEdgeSoftware

•  Celebrating 12 Year Anniversary

•  Team of 50 in Roxboro, NC

•  Sister company is Palace Pointe, a 100k sq. ft. Entertainment Venue for which we were developed as an in-house system

•  Over 600 facilties using our platform across the US and abroad

•  FEC’s, Waterparks, Trampoline Parks, Amusement Parks, Skating Rinks, Bowling Centers, Zoos & Museums

48


WhyCouchbaseForCenterEdge’sNewestCloudPlatform?

•  More scalable and performant than traditional SQL in the cloud

•  Previous online store system uses 19 SQL servers, each hosting 30 stores

•  As each store is only on a single server, it doesn’t handle spikes in load efficiently

•  Servers can’t be scaled vertically without downtime for all 30 stores on that server

•  Schema-less JSON increases flexibility as your system evolves, leaving schema enforcement in your data access layer

•  Schema changes to large tables can result in downtime as data structure is updated across all records

•  We were already using Couchbase for our shopping carts as well as a SQL caching layer, with great success. Now we can simplify the architecture with a single data layer.

49


CouchbaseCloudDataFlowArchitecture

50

Data Data Data

Index Query

Web Servers

Remote Application

Servers


EnforcingSchema

•  SinceCouchbasedoesn’tenforceschemalikeSQL,yourdataaccesslayershoulddosoinstead

•  AtCenterEdge,eachdocumenttypeisonlyupdatedbyasingleservice

•  Withinthatservice,schemaisenforcedbyserializingdatafromconsistentPOCOs

•  SchemachangescanbesupportedusingcustomizedJSONconvertersduringdeserialization

•  ISMISSINGisagoodwaytorecognizethedifferenceinattributesthatweren’tstoredbecausethedocumentwassavedusingtheoldschema

•  Wherepossible,trytopredictpossibleschemaneedsinadvance

51


PayAttentionToDocumentModelingUpFront

•  Watchoutfordocumentsthatgettoolarge•  Mighthit20MBdocumentsizelimit•  Highserialization/deserialization/networkingperformancepenalties•  Documentcontentionastoomanyactionsattempttomodifythedocumentsimultaneously

•  Watchoutfordataspreadacrosstoomanyrelateddocuments•  Lackofatomictransactionsacrossmultiplewritescanresultinpartialupdates•  Canaddlatencyifdocumentsmustbereadinachainedmanner(i.e.eachdocumentcontainsthekey

tothenextdocument)

•  Besuretoincludedocumentkeys,orawaytoconstructthem,whereyoumaywanttouseN1QLJOINorNESToperations

•  Shouldthedocumentkeybestoredinsidethedocument,too?•  Increasesdatasize,asthekeyisinthedocumentandinthemetadata•  Requiresthatthedatalayermaintainconsistency•  Canmakequerieseasiersinceyoudon’tneedtouseMETA()functiontogetthekey

52


ViewIndexesvs.GlobalSecondaryIndexes

•  Besuretoanalyzewhattypeofindexisbestforeachworkload

•  Viewsaregreatwherepre-aggregatingnumbersisuseful,suchasreports,graphs,etc

•  GSIisusuallythebestoptionformoregenericqueries,especiallyifwhenyou’rejusttryingtocollectasetofdocuments

•  Viewsdon’tscaleascleanly,theycan’tbescaledindependentlyviaMultiDimensionalScaling•  Viewsliveonthedatanodes,sotheyonlyscaleasyouaddmoredatanodes

•  AtCenterEdge,ournewplatformstartedonCouchbaseServer3.0,beforeGlobalSecondaryIndexeswereanoption

•  Weusedviewsandlookupdocumentsformostofourindexingneeds•  Wehaverunintoproblemswithtoomanyviewsperbucketcausingperformancebottlenecks•  We’recurrentlytransitioningmanyoftheseviewsintoGlobalSecondaryIndexes

53


EfficientIndexingIsEspeciallyImportantForCouchbase

54

•  PrimarykeyscansinSQLhavealwaysbeeninefficient•  EveryrecordinthetablewouldbereadandcheckedforamatchtotheWHEREpredicate•  Forsmalltables,theperformancepenaltywasnegligible,andwouldusuallygounnoticed

•  PrimarykeyscansinCouchbaseareusuallymuchworse•  Inourexperiencewithproduction-scaledata,almostinvariablyresultsinqueriestimingout•  EveryrecordinthebucketisbeingreadandcheckedforamatchtotheWHEREpredicate•  CaneasilyresultinreadingandparsingmillionsofJSONdocuments•  Willalsobustthein-memorycacheonthedatanodesifthereismoredatainthebucketthanallocated

memory

•  DesigneveryquerytobesupportedbyaGlobalSecondaryIndex•  Helpseveniftheindexisn’tanexactmatch•  Agooddesigncanvastlyreducethenumberofdocumentsscanned,makingitmorelikeaSQL

primarykeyscan


EfficientIndexingIsEspeciallyImportantForCouchbase

55

/*Usepredicatetoonlyindexdocumentsofacertaintype*/CREATEINDEXàirport_sourceairportÒN`travel-sample`(`sourceairport`)WHERE`type`='airport'

/*Toindexthesameattributeacrossmultipledocumenttypes,includetypeattributefirst*/CREATEINDEX`def_type_idÒN`travel-sample`(`type`,ìd`)

/*Agoodpracticeistocreateafallbackincaseotherindexesaren'tused*/CREATEINDEX`def_typeÒN`travel-sample`(`type`)

Ifyou’reusingthe“type”attributeasthelogicalequivalentofatableinSQL,mostindexeswillincludethisattribute.


HowToStoreandIndexDate/Times

•  Date/TimesareusuallystoredasISO8601stringsinJSON

•  UseSTR_TO_MILLIS(x)inindexesandqueriestoworkwithISO8601strings

56

/*STR_TO_MILLISconvertsanISO8601stringtoaUnixnumericrepresentation*//*Italsohandlesthetimezonespecifier*/SELECTÈxtent1`.*FROM`beer-sampleàsÈxtent1`WHERE(`type`='beer')AND(STR_TO_MILLIS(Èxtent1`.ùpdated`)<=STR_TO_MILLIS("2010-01-01T00:00:00Z"))

/*STR_TO_MILLISmustalsobeusedintheindex,ortheindexcannotbeused*/CREATEINDEX`beer_updatedÒN`beer-sample`(STR_TO_MILLIS(ùpdated`))WHERE`type`='beer'


IndexPerformanceDuringMutations

57

AirlineSQLTable

AirportSQLTable

travel-sampleBucket

AirlineIndexes

AirportIndexes

BucketIndexes

Remember that GSI indexes are similar to SQL indexes, but not the same


Training!

•  Don’tjustassumeyoucanswitchtoanyNoSQLplatformwithoutsometraining

•  Performanceprofileisdifferent,andthepenaltiescanappearindifferentplaces•  Developerswhoknowthepitfallsinadvancecansaveyoualotofrefactoringheadacheslater

•  N1QLdoeshelpreducethelearningcurvesignificantly•  For.Netdevelopmentshops,lookatLinq2Couchbasetomakeiteveneasier!

•  Theoperationsdepartmentneedstraining,too!

58


[email protected]

ClarenceJMTauro,[email protected]

[email protected]


Share your opinion on Couchbase

1. Go here: http://gtnr.it/2eRxYWn 2.  Create a profile 3.  Provide feedback (~15 minutes)


The Couchbase Connect16 mobile app Take our in-app survey!

©2016CouchbaseInc.

ThankYou!

62