Embedded Database Performance Report · database on a server. With Couchbase, you must utilize a Sync Gateway utility to serve as the proxy between Couchbase Lite on the remote client

Embedded Database Performance Report

Actian Zen more than 6x faster than Couchbase

MCG Global Services Benchmark Results

September 2018

Key insights

This benchmark did a head-to-head comparison of Actian Zen Core and Couchbase Lite, both running on a Nokia 2, Android 7 handset (ARM-based) with 1GB DRAM

Actian Zen Core outperformed Couchbase Lite for Indexed and Non-Indexed data management by:

– more than 5x on inserts, up to 2x on deletes, and over 2x on updates

Actian Zen Core plus Actian Zen Edge was also faster as a client-server combination than Couchbase Lite plus Couchbase Sync Gateway plus Couchbase server, replicating IoT Device or Smartphone to gateway data sharing —the area where it tends to really matter in embedded Edge applications.

Checkout the Actian Zen performance advantage today!

Visit https://www.actian.com/zen

EmbeddedNoSQLDatabasePerformanceBenchmark

ProductProfileandEvaluation:ActianZenandCouchbaseByWilliamMcKnightandJakeDolezalMcKnightConsultingGroupSeptember2018Sponsoredby

McKnightConsultingGroup EmbeddedDatabaseBenchmark

©McKnightConsultingGroup2018 http://www.mcknightcg.com Page2

ExecutiveOverview

Embeddeddatabasesarebuiltintosoftware,transparenttotheapplication’senduserandrequirelittleornoongoingmaintenance.Embeddeddatabasesaregrowinginubiquitywiththeriseofmobileapplicationsandinternetofthings(IoT)givinginnumerabledevicesrobustcapabilitiesviatheirownlocaldatabasemanagementsystem(DBMS).Developerscancreatesophisticatedapplicationsrightontheremotedevice.Fortheseuses,theembeddedarchitectureispreferredoverclient-serverapproacheswhichrelyondatabaseserversaccessedbyclientapplicationsviainterfaces.Today,tofullyharnessdatatogainacompetitiveadvantage,embeddeddatabasesneedahighlevelofperformancetoprovidereal-timeprocessingatscale.Alltheseweb,mobile,andIoTapplicationshavegeneratedanewsetoftechnologyrequirements.Embeddeddatabasearchitectureneedstobefarmoreagilethaneverbefore,andrequiresanapproachtoreal-timedatamanagementthatcanaccommodateunprecedentedlevelsofscale,speed,anddataflexibility.Sometimesrelationaldatabasesareunabletomeetthesenewrequirements,anddevelopersarethereforeturningtoNoSQLdatabasetechnology.NoSQLusecasesaboundwheretheneedforflexibleschemaorschema-lessdatawouldtripupconventional,relationaldatabases.Toquantifyembeddeddatabaseperformance,weconductedthisbenchmarkstudy,whichfocusesontheperformanceofmobileapplication-ready,NoSQL,embeddeddatabasesolutionsActianZenandCouchbase.Theintentofthebenchmark’sdesignwastorepresentasetofbasicdatabasetransactionsthatanorganizationdevelopingedgeapplicationsmightencounter.ThetestmethodologywasbasedonandlargelyfollowedtheBenchmarkofEmbeddedDatabaseson.NETconductedin2017byChristopheDiericxofrelationaldatabasetechnologies;however,ourownbenchmarkharnesswasdevelopedandadaptedtoaNoSQLusecase.WeconductedthebenchmarkonZenandCouchbaseLiteinstalledonthesameAndroiddevice,andZenandCouchbaseServerinstalledonaserver.Inourexperience,performanceisaveryimportantaspectofanembeddeddatabaseselection,butitisonlyoneaspectandmanyfactorsshouldbeconsidered.Overall,thebenchmarkresultswereinsightfulinrevealingthequeryexecutionperformanceofActianZenandCouchbaserevealingsomeofthedifferentiatorsinthetwoproducts.ActianZenEdgewasfasteracrosstheboardincludingtheareawhereittendstoreallymatterinembeddeddatabases—writespeed.



ThisistheessentialperformancemetricforIoTdata.Withoutsynchronizationofdatatotheserver,ActianZenEdgeoutperformedCouchbaseby5xoninserts,7xonqueriesof10,000documentsonanindexedkey,6xonqueriesof5,000documentsonanon-indexedkey,2xondeletesof10,000documentsonanindexedkeyandondeletesof5,000documentsonanon-indexedkey,2xonupdatesontheindexedkeyand3xonnon-indexedkeyupdates.Withsynchronizationofdatatotheserver,ActianZenEdgeoutperformedCouchbaseby6xoninserts,2xondeletesof10,000documentsonanindexedkeyandondeletesof5,000documentsonanon-indexedkey,2xonupdatesontheindexedkeyand3xonnon-indexedkeyupdates.ActianZenisamatureplatformforembeddeddatabaseapplicationswithover30yearsofengineeringanddevelopmentbehindit.Featuresthatcontributedtoitsextremelyfastperformanceinclude,butarenotlimitedto,theBtrieveAPIandTurboWriteAccelerator.



EmbeddedNoSQLDatabaseSelection

Organizationsthatutilizeapplication-ladensmartdevicesrelyonembeddeddatabaseplatformstoprocessedgedataathighspeedandbringitinwithconsistencytoharmonizeanecosystemofactivity.Volumesfordatathatcanbeutilizedattheedgeisrapidlyexpanding—placingsignificantperformancedemandsonembeddedarchitectures.Thus,akeydifferentiatoristhedepthbywhichadatabasemaintainsperformancetoscalewithsimplequeriesrepresentativeofrealworldusecasesofembeddeddatabases—SQLandNoSQLalike. Whileperformanceisveryimportant,itisnottheonlyconsideration.DeveloperschoosingNoSQLmustconsiderdataaccess,scalability,andavailability.BothCouchbaseandActianZenweredesignedto“setitandforgetit,”withlittle-to-noongoingdatabaseadministration.However,ActianZenwasengineeredpurposefullytoparedownanenterpriseNoSQLplatformtobeembeddedwithinOEMenvironments.Therefore,ActianZenhasfeaturesthatCouchbasedoesnot—includingauto-reconnectnetworking,automateddefragmentation,multi-usersupport,andconcurrentwritecapabilities.BothplatformsofferNoSQLsupport.ZenisnativelyNoSQLandisflexibleenoughtobedocument-basedorakey-valuestore.CouchbaseisJSONdocument-oriented.Additionally,ZenexclusivelyoffersthehighperformanceBtrieve2API(whichistestedinthisbenchmark.)TheBtrieve2APIsupportsNoSQLandnativedevelopmentsupportforJavaandC/C++baseddevicesandSWIGforPython,Perl,andPHP—inadditiontoitsSQLsupport.CouchbaseisexclusivelyNoSQL,andonlyofferssoftwaredevelopmentkitsformobiledevices,suchasiOS,Android,and.NET.Whilethesubjectofthisbenchmarkisembeddedapplications,ActianZenedgeispartoftheoverallZenfamilyofZenCore,ZenEnterprise,andZenReportingEngine.Whencombined,thissuiteofproductsenablesnotonlyembeddedapplications,butclient-server(withzeroETL)andclouddeploymentsaswell.Inaclient-serverconfiguration,bothActianZenandCouchbasecomewiththecapabilitytoautomaticallysynchronizeinrealtimebetweenaremotedeviceandaserver—withoutETL.Thiscapabilityiscriticalfortoday’sneedsanduses,becausethepotentialnumberofmobiledevicescouldeasilynumberinthethousands,andallthatinformationmayneedtofunnelintoacoredatabaseonaserver.WithCouchbase,youmustutilizeaSyncGatewayutilitytoserveastheproxybetweenCouchbaseLiteontheremoteclientandCouchbaseServer.ActianhasrealtimesynchronizationcapabilityofActianZenEdgetoCoreviatheBtrieveAPIwithoutanintermediary,whichcanallowyoutoachievescalewithsimplicity.Platformmaturityisalsoaconsideration.Couchbasewasinitiallyreleasedin2010.ActianZenwasinitiallydesignedasBtrieve(andlaterPSQL)andhasbeeninproductionwithmanymulti-nationalorganizationswithover30yearsofengineeringandenhancement.



ThisreportsfocusesontheperformanceoftwoembeddedNoSQLdatabaseoptions.Itisimportanttogetintotherightembeddeddatabaseearlyinthedevelopmentcyclewhenthestakesarelesscritical.Oneisaspecialtyapproachwithenterprisesoftwareoptimizedfortheembeddedarchitecture,andthelatteranopensource,multi-purposedatabaseplatform.



BenchmarkSetup

Thebenchmarkwasexecutedusingthefollowingsetup,environment,standards,andconfigurations.

DataPreparation

Anaimofthebenchmarkistosimulateatypicalreal-worldscenarioandusecaseforNoSQLembeddeddatabases.Inourbenchmark,wechoseasimple“schema”foranapplicationthatstorespeoples’contactinformationintheembeddeddatabase.Themodelconsistsofmultipledocumentsthatlooksimilartothefollowing:

{ "contact": { "id": 1, "lastname": "Rogers", "firstname": "Fred", "address": "381 Willinghelm Dr", "city": "Pittsburgh", "state": "PA", "zip": "15106", "country": "USA", "phone": "412-875-0921" } }

ThedatausedinthebenchmarkwasgeneratedrandomlyinrealtimebytheAndroidapplicationduringthebenchmarkexecution.Thekeyscity,state,andzipwereusedasselectioncriteriaintheSelect,Update,andDeletetests(describedbelow).Therefore,aparticularvaluewasrandomlyseededintothiskeyduringdatagenerationtoensuretherewouldbeenoughinstancesofthatvaluetoachievethedocumentcountsrequiredduringtheSelect,Update,andDeletetests.

Configuration

OurbenchmarkincludedtwodifferentembeddedRDBMS—ActianZenandCouchbase—installedonthesameAndroiddevice.Wealsotestedaconfigurationwithreal-timesynchronization.TheserverhadboththelatestversionsofActianZenCoreandCouchbaseServerinstalledonthesamemachine.Also,Couchbase’sSyncGatewaywasinstalledtoserveastheintermediarybetweenCouchbaseLiteandServer.Allcomponentsweredeployedonalocalareanetwork.

NoSQLDBMS

EmbeddedRDBMS ActianZen CouchbaseVersion 13.10.030 Server5.1.1Community

Lite2.1CommunitySyncGateway2.1Community



AndroidDevice

Hardware Nokia2TA-1035DSProcessor 1.3GHz64-bitquad-coreARMCortexA7RAM 1GB(8GBStorage)OS Android7.1.1Nougat

Server

Hardware LenovoThinkPadX1CarbonG620BS006UUSx64-basedPCProcessor [email protected] 8GBOS MicrosoftWindows10Enterprise10.0.16299

TestUseCases

Asaforementioned,thetestmethodologywasbasedonandlargelyfollowedtheBenchmarkofEmbeddedDatabaseson.NETconductedin2017byChristopheDiericx.ThetestinvolvessimpleusescasesofthemostbasicdatabaseCRUDoperations:selecting,updating,anddeletingrowsbasedonindexedandnon-indexedcolumns.Weconsideredotherbenchmarkframeworks,suchastheTransactionPerformanceCouncil(TPC).While,theirtestusecaseshavebeenappliedtoNoSQLtechnologiesinthepast,theyarenotveryapplicabletotypicalmobiledeviceapplications.MostIoTdevicesandmobileapplicationswillnotrequirethesophisticatedoperationsdemonstratedbythosebenchmarkframeworks.Therefore,weoptedforteststhatwoulddemonstraterawperformancethatcouldbefoundinmostembeddeddatabaseimplementations.BothplatformssupportarobustsetofNoSQLcapabilities.ForbothActianZenandCouchbaseLiteweusedthenativeAPIstoexecutethedatabasetransactionsinordertotestitsfunctionalityandperformance,ratherthanSQL(Zen)orN1QL(CouchbaseSQLforJSON).

UseCase1:OpenandCloseConnectionsinRapidSuccession

NOTE:WedidnotdothisrunfortheAndroiddevicesinceitisstandardpracticeformobiledeveloperstoopenadatabaseconnectionandleaveitopenwhiletheappisrunning.Also,wedidnotusethistestforthesynchronizationbenchmark,havingnoapplicableuseforthatworkload.

UseCase2:InsertPerformance

Mobiledevicesandtheirapplicationswillundoubtedlyneedexcellentinsertperformance.Thismaythesinglemostimportantmetricformanyusecases.Forexample,consideranIoTdeviceisasensortakingreadingsatregularintervals.Inthecaseofreal-timeorrapidsensorreadings,insertperformanceiscritical.



Test2 Insert25,000documents

NOTE:Atthebeginningofthetest,thedatabasecontainsanemptydatabase.TheInserttestprovidedthetestdatafortheremainingbenchmarks.

UseCase3:SelectPerformance

Certainly,wemustconsiderbothplatforms’abilitytoretrievedata.Ourtestcasesinvolveselectingbulkdocuments,ratherthansingledocumentsviaauniqueidentifier.Thefirstvariationofthetestfiltersonanindexedkey(state).Thesecondtestselectsfewerdocuments,butfiltersonakeythatdoesnothaveanindex(zip).

Test3a Select10,000documentsonanindexedkeyTest3b Select5,000documentsonanon-indexedkey

NOTE:Wedidnotusethistestforthesynchronizationbenchmark,sinceselectingdocumentsbythemselveswouldnotconstitutethecompleteworkload.

UseCase4:UpdatePerformance

WealsotestedtheperformanceofbulkdocumentupdatesusingthesameselectiontestcriteriaasTest3.Ourtestcasesinvolveselectingbulkdocumentsandupdatingasinglekey-value.Thefirstvariationofthetestfiltersonanindexedkey(state)andupdateszip.Thesecondtestselectsfewerdocuments,butfiltersonakeythatdoesnothaveanindex(zip)andupdatesstate.

Test4a Update10,000documentsonanindexedkeyTest4b Update5,000documentsonanon-indexedkey

UseCase5:DeletePerformance

Wealsotestedtheperformanceofbulkdocumentdeletes—again,usingthesameselectiontestcriteriaasTest3.Ourtestcasesinvolveselectingbulkdocumentsanddeletingthem.Thefirstvariationofthetestfiltersonanindexedkey(state)anddeletesthosedocuments.Thesecondtestselectsfewerdocuments,butfiltersonakeythatdoesnothaveanindex(zip)anddeletesthedocuments.

Test5a Delete10,000documentsonanindexedkeyTest5b Delete5,000documentsonanon-indexedkey



BenchmarkResults

ThefollowingfiguresdisplaytheaveragetimeelapsedforeachdatabasetransactionforbothActianZenandCouchbase.Eachtestwasexecuted5timesandthemedianvaluewasused.

LocalOnly(NoSynchronization)

Test2:Insert25,000documents

Belowaretheaveragetimes(inmicroseconds)ittooktoinsertacompletedocumentofrandomly-generateddataintotheContactsdatabaseonActianZenandCouchbase.

Thistestrevealedthefirstmajorperformancedifferentiator.ActianZen’saveragetimetoinsertasingledocument(takingtheaverageofall25,000inserts)was4.6timesfasterthanCouchbaseinserts.

Test3a:Select10,000documentsonanindexedkey

Belowaretheaveragetimesperdocument(inmicroseconds)ittooktobulkselectrecordsfromtheContactsdatabaseapplyingafilteronanindexedkeyforbothActianZenandCouchbase.



Bothplatformsrespondedveryquickly.Couchbase’sfetchrateperdocument(takingtheaverageofall10,000documents)was7timesthatofActianZen’s.

Test3b:Select5,000documentsonanon-indexedkey

Belowaretheaveragetimesperdocument(inmicroseconds)ittooktobulkselectrecordsfromtheContactsdatabaseapplyingafilteronanon-indexedkeyforbothActianZenandCouchbase.

Againbothplatformsrespondedveryquickly.Couchbase’sfetchrateperdocument(takingtheaverageofall10,000documents)was5.8timesthatofActianZen’s.

Test4a:Update10,000documentsonanindexedkey

Belowaretheaveragetimes(inmicroseconds)ittooktoupdateasinglekeyintheContactsdatabaseapplyingafilteronanindexedkeyforbothActianZenandCouchbase.

Thisonewasaclosetest.Theaveragetimetoupdateasinglekey(takingtheaverageofall10,000updates)wasonly1.9timesfasterthanCouchbaseupdates.

Test4b:Update5,000documentsonanon-indexedkey

Belowaretheaveragetimes(inmicroseconds)ittooktoupdateasinglekeyintheContactsdatabaseapplyingafilteronanon-indexedkeyforbothActianZenandCouchbase.



Thistesthadsimilarresultsastest4a.ActianZen’saveragetimetoupdateasinglekey(takingtheaverageofall5,000updates)was2.9timesfasterthanCouchbaseupdatesusingthesamefilter.

Test5a:Delete10,000documentsonanindexedkey

Belowaretheaveragetimes(inmicroseconds)ittooktodeleteadocumentintheContactsdatabaseapplyingafilteronanindexedkeyforbothActianZenandCouchbase.

Bothwereveryfast.Theaveragetimetodeletearow(takingtheaverageofall10,000deletes)forActianZenwassimply2timesfasterthanCouchbasedeletes.

Test5b:Delete5,000documentsonanon-indexedkey

Belowaretheaveragetimes(inmicroseconds)ittooktodeleteadocumentintheContactsdatabaseapplyingafilteronanon-indexedkeyforbothActianZenandCouchbase.



Deletingdocumentsonanon-indexedkeyproducedresultsconsistentwithbefore.ActianZen’saveragetimetodeleteadocument(takingtheaverageofall5,000deletes)was1.88timesfasterthanCouchbaseupdatesusingthesamefilter.

Synchronization

Test2:Insert25,000documentsandsync

Belowaretheaveragetimes(inmicroseconds)ittooktoinsertacompletedocumentofrandomly-generateddataintotheContactsdatabaseontheActianZenandCouchbasedatabases.

Thistestrevealedthefirstmajorperformancedifferentiator.ActianZen’saveragetimetoinsertasingledocument(takingtheaverageofall25,000inserts)was5.9timesfasterthanCouchbaseinserts.

Test4a:Update10,000documentsonanindexedkeyandsync

Belowaretheaveragetimes(inmicroseconds)ittooktoupdateasinglekeyintheContactsdatabaseapplyingafilteronanindexedkeyforbothActianZenandCouchbase.



ActianZen’saveragetimetoupdateasinglekey(takingtheaverageofall10,000updates)was2timesfasterthanCouchbaseupdates.

Test4b:Update5,000documentsonanon-indexedkeyandsync

Belowaretheaveragetimes(inmicroseconds)ittooktoupdateasinglekeyintheContactsdatabaseapplyingafilteronanon-indexedkeyforbothActianZenandCouchbase.

Thistesthadsimilarresultsastest4a.ActianZen’saveragetimetoupdateasinglekey(takingtheaverageofall5,000updates)was3.3timesfasterthanCouchbaseupdatesusingthesamefilter.

Test5a:Delete10,000documentsonanindexedkeyandsync

Belowaretheaveragetimes(inmicroseconds)ittooktodeleteadocumentintheContactsdatabaseapplyingafilteronanindexedkeyforbothActianZenandCouchbase.



Bothwereveryfast.Itsaveragetimetodeletearow(takingtheaverageofall10,000deletes)wasalittlelessthanhalfthetimeCouchbasetook.

Test5b:Delete5,000documentsonanon-indexedkeyandsync

Belowaretheaveragetimes(inmicroseconds)ittooktodeleteadocumentintheContactsdatabaseapplyingafilteronanon-indexedkeyforbothActianZenandCouchbase.

Deletingdocumentsonanon-indexedkeyproducedresultsconsistentwithbefore.Couchbase’saveragetimetodeleteadocument(takingtheaverageofall5,000deletes)wasmorethandoublethatofActianZenupdatesusingthesamefilter.



Conclusion

Withoutsynchronizationofdatatotheserver,ActianZenEdgeoutperformedCouchbaseby5xoninserts,7xonqueriesof10,000documentsonanindexedkey,6xonqueriesof5,000documentsonanon-indexedkey,2xondeletesof10,000documentsonanindexedkey,2xondeletesof5,000documentsonanon-indexedkey,2xonupdatesontheindexedkeyand3xonnon-indexedkeyupdates.Withsynchronizationofdatatotheserver,ActianZenEdgeoutperformedCouchbaseby6xoninserts,2xondeletesof10,000documentsonanindexedkey,2xondeletesof5,000documentsonanon-indexedkey,2xonupdatesontheindexedkeyand3xonnon-indexedkeyupdates.ActianZenoutperformedCouchbaseinallofthefundamentaldatabaseoperations.ThesetestedoperationsunderlienearlyalloperationsthatoccuronanembeddeddatabaseforanIoTormobileimplementation,soitisunlikelymorecomplexoperationswouldhaveadifferentresult.ActianZenisamatureplatformforembeddeddatabaseapplicationswithover30yearsofengineeringanddevelopmentbehindit.TheBtrieve2APIhadclearperformanceadvantageswithouttheoverheadofCouchbase.Also,Zen’sTurboWriteAcceleratorcouldalsoshedlightintoitsperformanceadvantages.Sinceitcostsmuchlesstocontinuewritingthantostopandrestart,contiguouswritesaresignificantlyfasterthannon-contiguouswrites.TheTurboWriteAccelerator(TWA)pre-allocatesopenslotswithinthephysicalfilesothatmultiplepagescanbewrittenasasinglecoalescedpage—improvingI/Operformanceandreducingtheoverheadofinteractionwiththeoperatingsystem.Theresultoftheapplicationofthemethodologytothearchitecture,bothexplainedhereinandreplicable,showamarked,andsometimesastonishing,performanceadvantagetoActianZen.Thisisespeciallytrueintheimportantwriteoperationsinsert,updateanddelete.Overall,ActianZenisanexcellentchoiceforIoTormobilecompaniesneedinghighperformanceandascalableembeddeddatabase.



AboutMcKnightConsultingGroup

WilliamMcKnightisPresidentofMcKnightConsultingGroup(MCG)(http://www.mcknightcg.com).Heisaninternationallyrecognizedauthorityininformationmanagement.HisconsultingworkhasincludedmanyoftheGlobal2000andnumerousmidmarketcompanies.Histeamshavewonseveralbestpracticecompetitionsfortheirimplementationsandmanyofhisclientshavegonepublicwiththeirsuccessstories.Hisstrategiesformtheinformationmanagementplanforleadingcompaniesinvariousindustries.JakeDolezalhastwodecadesofexperienceintheInformationManagementfieldwithexpertiseinbusinessintelligence,analytics,datawarehousing,statistics,datamodelingandintegration,datavisualization,masterdatamanagement,anddataquality.Jakehasexperienceacrossabroadarrayofindustries,including:healthcare,education,government,manufacturing,engineering,hospitality,andgaming.HehasadoctorateininformationmanagementfromSyracuseUniversity.MCGservicesspanstrategy,implementation,andtrainingforturninginformationintotheassetitneedstobeforyourorganization.Westrategize,designanddeployinthedisciplinesofMasterDataManagement,BigDataStrategy,DataWarehousing,AnalyticDatabasesandBusinessIntelligence.



AboutActian

Actian,thehybriddatamanagement,analyticsandintegrationcompany,deliversdataasacompetitiveadvantagetothousandsofcustomersworldwide.ThroughthedeploymentofinnovativehybriddatatechnologiesandsolutionsActianensuresthatbusinesscriticalsystemscantransactandintegrateattheirverybest–onpremise,inthecloudorboth.FormoreinformationaboutActianVectorandtheentireActianportfolioofhybriddatamanagement,analyticsandintegrationsolutionson-premiseorinthecloud.FindoutmoreaboutActianVectorforsingleserversandforHadoopclusters,orgetlinkstodownloadsforon-premisedeploymentorcloudinstances.

Documents

Embedded Database Performance Report · database on a server. With Couchbase, you must utilize a Sync Gateway utility to serve as the proxy between Couchbase Lite on the remote client