Spark tutorial py con 2016 part 2

DavidTaiebSTSM-IBMCloudDataServicesDeveloperadvocatedavid_taieb@us.ibm.com

HANDS-ONSESSION:DEVELOPINGANALYTICAPPLICATIONSUSINGAPACHESPARK™ANDPYTHONPart2:AnalyzingcartwiQerdatawithSparkandDashDb PyCon2016,Portland

©2016IBMCorpora6on �

Agenda

•  Provisiontheapplica6onservicesonBluemix:Spark,DashDb,IBMInsightforTwiJer

•  LoadcarrelatedtweetsfromIBMInsightforTwiJerintoDashDbwarehouse

•  RunAnaly6csinPythonNotebookanddiscovernewinsights


Sign up for Bluemix •  AccessIBMBluemixwebsiteonhJps://console.ng.bluemix.net•  ClickonGetStartedforFree

•  CompletetheformandclickCreateaccount•  Lookforconfirma6onemailandclickonconfirmyouaccountlink

CreatenewSpace


Create a new space on Bluemix Inprepara6onforrunningtheproject,wecreateanewspaceonBluemix

CreateaSparkInstance

Op6onal:YoucanskipthisstepifyoualreadyhaveaspacewithSparkinstancethatyouwouldliketoreuse


Create a Spark Instance

Op6onal:YoucanskipthisstepifyoualreadyhaveaspacewithSparkinstancethatyouwouldliketoreuse


Create New Spark Instance Op6onal:YoucanskipthisstepifyoualreadyhaveaspacewithSparkinstancethatyouwouldliketoreuse


Acquiring the data

•  Inthenextsec6on,weshowhowtoacquirethetwiJerdataandstoreitintoDashDb.

•  WeusetheTwiJerloadingconnectoravailableasamenuinDashDbconsole

CreateaDashDbinstance


Create an instance of IBM Dash DB on Bluemix

CreateanIBMInsightforTwiJerinstance


Create an instance of IBM Insight for Twitter on Bluemix


Agenda





Launch DashDb Console ClickontheDashDbService6letoopenthisdashboard,thenclickonLaunchbuJon

LoadTwiJerData


Load Twitter Data

DashDbConsoleofferedmul6pledataconnectorsincludingaTwiJerconnectorthatautoma6callyconnectstoIBMInsightforTwiJer

ConnecttoTwiJer


Connect to Twitter

ReusingtheTwiJerserviceinstancecreatedinpreviousstep


Select the data to be loaded TwiJerQuerybeingused:posted:2015-01-01,2015-12-31followers_count:2000listed_count:1000(volkswagenORvwORtoyotaORdaimlerORmercedesORbmwORgmOR"generalmotors"ORtesla)

SpecifytwiJerquery

Providepreviewcountofoutputdata


Select the DashDb Table

Nameoftheschemaunderwhichthetableswillbecreated

Prefix(Namespace)forthecreatedtables

Listoftablesthatwillbecreated


Loading data monitoring page

Warning:loading6memayvarybasedonbandwidth.Itmaytakebetween15mnsand1hour


Complete the load: Statistics


Complete the load: explore the data


Get connection information CopytheUserid,passwordandjdbcurl,you’llneedthisinforma6onlater


Agenda





Create new Notebook from URL

ImportrequiredPythonpackages

• CreatenotebookfromURL• UsehJps://github.com/ibm-cds-labs/spark.samples/raw/master/notebook/DashDB%20TwiJer%20Car%202015%20Python%20Notebook.ipynb


Step 1: Import Python Packages •  Installnltkpackage(Naturallanguagetoolkit)• Wewilluseittofilterstopwordslaterinthetutorial


Import Python modules and setup the SQLContext


Step 2: Define global Variables

Setupvariousdatastructureswe’llneedthroughouttheNotebook

ThisistheSCHEMAandPREFIXyouusedinStep3oftheTwiJerconnectorwizard


Set up some global helper functions

JavaScriptGooglemapvisualiza6on

Mischelperthatfillinmissingdates


Step 3: Acquire the data from DashDB

UserIDandpasswordfromConnec6onpage

UserIDandpasswordfromConnec6onpage


Join the Tweets and Sentiment Table Inthisstep,wewanttoaddasen6mentscoreforeachtweetrecord:•  JointheTweetsandSen6mentstable•  Encodethesen6mentintoanumbere.g.POSITIVE=+1,NEGATIVE=-1,AMBIVALENT=0•  Createanaverageforeachsen6mentassociatedwithatweet•  %6meinstrumentsthecodetoprovideprofileexecu6onstats.


Step 4: Transform the data

CreateacleanWorkingdataframethatwillbeeasiertouseinouranaly6cs


Step 5: Geographic distribution of tweets

GroupBycountriesandaggregatethetweetscount

ConvertSparkSQLdataframetoPandasdatastructureforvisualiza6on


Bar chart visualization of Tweet distribution by Geo


Google map visualization of tweet distribution by Geos

CallGeoChartHelperthatsetuptheJavaScriptcode


Clean up memory before next analytics

ResourcesincludingmemoryontheSparkDrivermachinearenotinfinite.Itisgoodprac6cetocleanupwhendataisnotneededanymore


Step 6: Analyzing tweets sentiment

GroupBySen6mentsandaggregatethetweetscount



Sentiment visualization

UseMatplotpiechart


Step 7: Analyze Tweet timeline


GroupByPos6ng6meandsen6menttuplesAggregatethetweetcounts

GroupByPos6ng6meandsen6menttuplesAggregatethesumofthetweetcounts


Prepare the timeline data structures


Time series visualization for all tweets


Deep dive into car manufacturers

CreatenewDataFramethatenrichtweetswithextrametadata:-Booleanforeachcarmanufacturer-Booleanforelectriccar-Booleanforselfdrivingcar


Re-analyze tweeter timeline for each car manufacturer

CreatenewDataFrameforeachcarmanufacturerAggregatethetweetcounts,orderbypos6ng6me


Timeline series visualization

No6cethepeakoftweetsforVWbetweenSeptemberandOctober2015


Explain why the peak of tweets for VW between September and October 2015

FilterforallVWtweetsbetweenSeptandOct2015

Piechartvisualiza6onofthetop10wordsbeingusedinthesetweets

Createmapcountofallnon-stopwordsusedinthetweets

UseNLTKstopwordsmoduletofilteroutstopwords


Peak explained

WecanclearlyseefromthelistofmostusedwordsthatthepeakcorrespondtotheVWscandalaroundfraudulentemissionstes6ng


Follow the notebook for many more interesting analytics


Resource

•  hJps://developer.ibm.com/clouddataservices/•  hJps://github.com/ibm-cds-labs/simple-data-pipe•  hJps://github.com/ibm-cds-labs/pipes-connector-flightstats•  hJp://spark.apache.org/docs/latest/mllib-guide.html•  hJps://console.ng.bluemix.net/data/analy6cs/


Thank You

Data & Analytics

Spark tutorial py con 2016 part 2