Combine Query Language and Data Flow Language for Data Science · What is Apache Pig Ã Apache Pig is a high-level plaorm for creang programs that run on Apache Hadoop. The language

1 ©HortonworksInc.2011–2016.AllRightsReserved

SparkSQL+Pig-La.nCombineQueryLanguageandDataFlowLanguageforDataScience

JeffZhang([email protected])May16,2017


WhoamI

Ã  ASFMember,workinASFforalmost8years

Ã  CommiRerofApacheTez,Pig&Zeppelin

Ã  WorksinHortonworks


DataScience

DataScience,alsoknownasdata-drivenscience,isaninterdisciplinaryfieldaboutscienYficmethods,processesandsystemstoextractknowledgeorinsightsfromdatainvariousforms,eitherstructuredorunstructured.

Ã  Describewhathappens

Ã  Explainwhathappens

Ã  Predictwhatwouldhappen


DataScience

CollectData

DataMunging

DataAnalysisInsight

Product

online offline


DataMunging

§  CollectandTransformServerLogData•  UserAgentNormalizaYon•  RobotDetecYon•  Sessionize

§  MovedatafromDatabasetoHDFS

§  CollectandTransformSocialMediaData


DataMunging

BeforeDataMunging AcerDataMunging


DataAnalysis

Ã  CombinedifferentsourcesofdataandapplystaYsYcs,BItoolstogetinsightfromData–  WebTrafficMetrics–  UserSegmentaYonAnalysis–  A/BTest


DataMungingvsDataAnalysis

DataMunging DataAnalysisDataSource Messy

Structured/UnstructuredUnorganized

Clean,NormalizedStructuredOrganized

Stability Regular,Stable Ad-hoc

Tools Python,Spark,Hadoopandetc.

R,Python,SQLandetc.

Datayouhavetobefullstackbigdataengineertododatascience?

Whatifyouareadataanalystwithoutmuchprogrammingskills?


DataScienceInfrastructure


WhatisSpark

ApacheSparkisafast,in-memorydataprocessingenginewithelegantandexpressivedevelopmentAPIstoallowdataworkerstoefficientlyexecutestreaming,machinelearningorSQLworkloads.


WhatisApachePig

Ã  ApachePigisahigh-levelplajormforcreaYngprogramsthatrunonApacheHadoop.ThelanguageforthisplajormiscalledPigLa.n.PigcanexecuteitsHadoopjobsinMapReduce,ApacheTez,orApacheSpark

•  Easeofprogramming

•  OpYmizaYonopportuniYes

•  Extensibility


WordCount

Load

ForEach Group ForEach Order

StoreUsingSQL?


Pig-La.nvsSQL

SQL Pig-La.nLanguageType QueryLanguage

•  defactorstandard

DataFlowLanguage•  lazyevaluaYon•  supportpipelinesplit

DataSource StructuredData Structured/UnstructuredIntegraYon IntegratedwithmostofBITools VeryfewBItoolsintegratedwith

Pig-LaYn

Conclusion•  Pig-La.nforDataMunging•  SQLforDataAnalysis


IntegrateSparkintoPig

LogicPlan

PhysicalPlan

Execu.onPlan

Execu.onEngine

PigScript


CombineSparkSQL+Pig-La.n

SparkDataFrameTable

SparkSQL

DataMunging

DataAnalysis

SparkScalaAPI

SparkPythonAPI

SparkRAPI

PigLa.n


Pig-Lain+SparkSQL

SparkDataFrameTable

SparkSQL

Load Store

DataMunging

DataAnalysis


SparkTable(bank)

PigLaYn

SQL


WheretorunPig-La.n&SparkSQL(Zeppelin)

ApacheZeppelinisaweb-basednotebookthatenablesinteracYvedataanalyYcs.YoucanmakebeauYfuldata-driven,interacYveandcollaboraYvedocumentswithSQL,Scalaandmore.


JVM

ZeppelinServer

PigInterpreterGroup

Pig-LaYn SparkSQL

JVM

JVM

SparkInterpreterGroup

Scala Python R

Pig-LaYn+SparkSQLinZeppelin


DataScienceInfrastructure(Recap)


Demo


Conclusion

Ã  LeveragethepowerofbothQueryLanguageandDataFlowLanguage

Ã  UseSparkasUnifiedExecuYonEngine.

Ã  ShareDatabetweenDataMunging&DataAnalysis

Ã  UseZeppelinasUnifiedDataSciencePlajorm


Summary

Ã  DataMunging&DataAnalysis

Ã  UsePig-LaYnforDataMunging,UseSQLforDataAnalysis

Ã  RununderSparkEngine

Ã  UseZeppelinasunifiedDataSciencePlajorm


CurrentStatus&What’sNext

Ã Status–  PIG-5080(Supportstorealiasassparktable)–  ZEPPELIN-2232(SupportSparkSQLforPigInterpreter)

Ã Next–  IntegrateSparkMLlibinPig–  UseDataFrameAPIinsteadofRDDAPItointegrateSparkwithPig–  SupporttoIntegratePigwithotherSparkAPIs,likeR,Python


Q&A


ThankYou

Documents

Combine Query Language and Data Flow Language for Data Science · What is Apache Pig Ã Apache Pig is a high-level plaorm for creang programs that run on Apache Hadoop. The language