On Demand HDP Clusters using Cloudbreak and Ambari

On-DemandHDPClustersusingCloudbreakandAmbari

KarthikKaruppaiya NarendraBidariSr.EngineeringManager,CPE Sr.So0wareEngineer,CPE

DublinHadoopSummit2016–KarthikKaruppaiya&NarendraBidari

Agenda


IntroducIon1

BigDataPlaJormChallenges2

WhatisthesoluIon?3

SelfServiceAnalyIcs4

GoingHybridCloudusingCloudbreak5

IngesIngData6

IntroducFon


Symantec-  Symantecistheworldleaderinprovidingsecurityso0wareforbothenterprises

andendusers

-  Thereare1000’sofEnterprisesandmorethan400milliondevices(Pcs,TabletsandPhones)thatrelyonSymantectohelpthemsecuretheirassetsfromaXacks,includingtheirdatacenters,emailsandothersensiIvedata

CloudPlaHormEngineering(CPE)-  BuildconsolidatedcloudinfrastructureandplaJormservicesfornextgeneraIon

datapoweredSymantecapplicaIons

-  AbigdataplaJormforbatchandstreamanalyIcsintegratedwithOpenstack

-  Opensourcecomponentsasbuildingblocks

-  HadoopandOpenstack

-  Bridgefeaturegapsandcontributeback

Agenda


IntroducIon1


WhatisthesoluIon?3



IngesIngData6

BigDataPlaHormChallenge• HundredsofmillionsofusersgeneraIngBillionsofeventseverydayfromacrosstheglobe

• HundredsofBigDataApplicaIonDevelopersdeveloping1000sofapplicaIons

• At12PBand500+nodes,CloudPlaJormEngineeringAnalyIcsteambuiltthelargestsecuritydatalakeatSymantec


BigDataPlaHormChallenge

• Great!NowDeveloperscanstartbuildingapplicaIonsonourBigDataLake• 100sofdevelopersstartbuildingapplicaIonsusingdifferentbigdatatools


BigDataPlaHormChallenge

• Productteamdeveloperswantsquickchanges,latestversions• PlaJormteamwantsstability!

• Soon,frustraIonprevails


Agenda


IntroducIon1


WhatisthesoluIon?3



IngesIngData6

WhatistheSoluFon?

• BuildanduseyourownliXleclusterfordevelopment• Copysubsetofdatafordevelopmentpurposes

• Teardowntheclustera0erdevelopmentiscomplete

• RepeatandRinse


WhatistheSoluFon?

• ButBuildingclustersarehardandImeconsuming• Toomanyservicestoinstallandconfigure

• Developersarenotinterestedinbuildingandmanagingclusters


WhatistheSoluFon?–SelfService

• Whatifwemakeitreallyeasytobuildclusters?• AbstractallthedeploymentcomplexiIesandenabledeveloperstogettheirownclusterinoneclickofabuXon


WhatistheSoluFon?–ExtendAmbari

• WhatabouttheservicesthatarenotsupportedbyAmbarioutofthebox?• WewriteourownAmbaricustomstack


Agenda


IntroducIon1


WhatisthesoluIon?3



IngesIngData6

SelfServiceAnalyFcs(SSA)Clusters

• RESTfulwebservicestoallowcreaIonandmanagementofcustomclusters• Selectfrompre-definedAmbariBlueprints

• SpinsupVMsonourprivateOpenstackcloud

•  InstallsHDPstackspecifiedaspartofAmbariblueprint

• Dashingdashboardtomonitorandmanage(start/stop/kill)clusters


Environment

• PrivatecloudonOpenstack• HDP2.3.2• Ambari2.1.2

• AbilitytoseamlesslysupportpubliccloudlikeAWS



SSAArchitecture


SSAServices


SSADemo

SSAProsandCons

• Pros– Automatesspinningupclusters

– GetsclusterupandrunningwithHDPinminutes

– Userscanspinup/killclustersatwill

– CentralDashboardtomanageclusters

– Customizabletocatertoourprivatecloud

• Cons– Tightlycoupledwithourspecificprivatecloudinfrastructure

– Notportabletoworkwithotherpubliccloudvendors


Agenda


IntroducIon1


WhatisthesoluIon?3



IngesIngData6

NextGenSSA

• Thisisallgreat!But,weareoutofcapacityonourprivateopenstackcloud.• Justusethesameso0waretosetupclustersonAWSandGoogleCloud–thesamecodeshouldworkright?


NextGenSSA–Cloudbreak

• Cloudbreak– CloudbreakhelpstosimplifytheprovisioningofHDPclustersincloudenvironments

– SupportsmulIplecloudsincludingAWS,Google,AzureandOpenstack

– UsesApacheAmbariforHDPinstallaIonandmanagement– HasaniceUItobuildandmanageclusters

– Supportsautomatedclusterscaling


HybridCloudUsingCloudbreak-Gaps

•  Cloudbreak’soutoftheboxsupportforopenstackislimited–  No support for Keystone v3 – private cloud uses keystone v3 –  No support for native Openstack APIs –  Supports only Heat template for cluster provisioning

•  Nowwehavetwodifferentsystems–oneforprivateopenstackcloudandoneforAWS


Symantec’sContribuFontoCloudbreak

• Keystonev3support– Cloudbreak1.2–released03/2016

• NaIveOpenstackAPIsupport(withoutusingHeattemplate)– Codedevelopmentisdone,willbecontributedbacksoon.


Cloudbreak–KeystonreV3Screenshot


Cloudbreak–KeystoneV3ProjectScopeScreenshot


Agenda


IntroducIon1


WhatisthesoluIon?3



IngesIngData6

Whereismydata?

• Great!Inowhaveashinynewcluster–butwhatdoIdowithnodata?


DataIngesFonService

• BuiltasaStorm-TridentTopology• SupportvarioussourceslikeKaja,RabbitMQ,HDFS,Cassandra,etc

• Pluggableinterfacetoaddmoresources

• Samplethedata,fortesIngpurposes.

• GeneralizedtemplateprovidesabilitytoadddatatransformaIon

• Abilitytomonitordatatransferandcontrolrateoftransfer

• TryingtoachieveExactlyoncemessageDelivery


DataIngesFonServiceArchitecture


DataIngesFonServiceDemo


SummaryandFutureWork

•  Ajourneytowardsoneclickclusterdeployment•  Cloudbreak-onetoolforallcloud-  EnableCloudbreaktosupportourversionofOpenstack

-  EnableCloudbreaktosupportBaremetalclusterprovisioning

-  SinglelargeYARNclusterforvarietyofcomputeandstorageloads

•  Opensource–useandcontribute-  Workwithcommunitytoaddressgaps

•  SSAandDataIngesIoncodealreadyopensourced-  hXps://github.com/symantec/


ThankYou!Q&A

KarthikKaruppaiyakarthik_karuppaiya@symantec.comNarendraBidarinarendra_bidari@symantec.comDublinHadoopSummit2016–KarthikKaruppaiya&NarendraBidari

Technology

On Demand HDP Clusters using Cloudbreak and Ambari