REAL-TIME INGESTING AND TRANSFORMING SENSOR DATA & SOCIAL DATA w/ NIFI + TENSORFLOW

1 ©HortonworksInc.2011– 2017.AllRightsReserved

Real-TimeIngestingandTransformingSensorDataandSocialDatawithNiFiandTensorFlowTimothySpannHortonworks@PaaSDev


Agenda

• What do we want to do?• Why?• How?• Apache NiFi• TensorFlow• Natural Language Processing• Demo• Questions


Whatdowewanttodo?

• MiniFi ingestscameraimagesandsensordata

• RunTensorFlow Inceptionv3torecognizeobjectsinimage

• NiFistoresimages,metadataandenricheddatainHadoop

• NiFiingestssocialdataandfeeds

• NiFianalyzessentimentoftextualdata


WhyGatherandAnalyzeSocialMediaStream?

- AutomateprocessestomaximizeSocialMediateam’stime

- Improvedresponsetimetorequests,complaintsandemergenciesinsocialmedia

- Predictiveanalyticstoknowwhenandwhereproblemswillhappen

- Learnwhereunhappycustomersareandaddressinstantly


Aggregatealldatafromsensors,geo-locationdevices,machinesandsocialfeeds

Collect:BringTogether

Mediatepoint-to-pointandbi-directionaldataflows,deliveringdatareliablytoHBase,Hive,SlackandEmail.

Conduct:MediatetheDataFlow

Parse,filter,join,transform,fork, query,sort,dissect;enrichwithweather,location,NLPandTensorFlow.

Curate:GainInsights


WhyApacheNiFi?

• Guaranteeddelivery• Databuffering

- Backpressure- Pressurerelease

• Prioritizedqueuing• FlowspecificQoS

- Latencyvs.throughput- Losstolerance

• Dataprovenance• Supportspushandpull

models

• Hundredsofprocessors• Visualcommandand

control• Overafiftysources• Flowtemplates• Pluggable/multi-role

security• Designedforextension• Clustering


DATAENR ICHMENT

DATAD ISCOVERY

Inceptionv3

PRED ICT IVEANALYT ICS

SentimentAnalysis


WhyTensorFlow?AlsoApacheMXNet,PyTorch andDL4J.

• Google• Multipleplatform

support• Hadoopintegration• Sparkintegration• Keras• LargeCommunity• PythonandJavaAPIs• GPUSupport• MobileSupport

• Inceptionv3• Clustering• Fullyfunctionaldemos• OpenSource• ApacheLicensed• LargeModelLibrary• Buzz• ExtensiveDocumentation• RaspberryPiSupport


• TensorFlow (C++, Python, Java) via ExecuteStreamCommand

• TensorFlow NiFi Java Custom Processor

• TensorFlow Running on Edge Nodes (MiniFi)

ApacheNiFiIntegrationwithTensorFlow Options


• TensorFlow Mobile (iOS, Android, RPi)

• TensorFlow on Spark (Yahoo) via Livy, S2S, Kafka

• TensorFlow Running in Containers in YARN 3.0 on Hadoop

• gRPC Call to TensorFlow Serving

ApacheNiFiIntegrationwithTensorFlow Options


ExecuteStreamCommand To TensorFlow

https://community.hortonworks.com/articles/58265/analyzing-images-in-hdf-20-using-tensorflow.html


pythonclassify_image.py --image_file /dir/solarroofpanel.jpg

solardish,solarcollector,solarfurnace(score=0.98316)windowscreen(score=0.00196)manholecover(score=0.00070)radiator(score=0.00041)doormat,welcomemat(score=0.00041)

TensorFlow via Python


TensorFlow Java Processor in NiFi

https://community.hortonworks.com/content/kbentry/116803/building-a-custom-processor-in-apache-nifi-12-for.html

https://github.com/tspannhw/nifi-tensorflow-processor


TensorFlow Running on Edge Nodes (MiniFi)


pipinstall-Utextblobpython-mtextblob.download_corpora

Installing TextBlob for Python

Installing spaCy for Python

https://community.hortonworks.com/articles/76935/using-sentiment-analysis-and-nlp-tools-with-hdp-25.html

pipinstall-Uspacypython-mspacy.en.download all

Installing NLTK for Python 2.7

http://www.nltk.org/install.html

pip install -U nltkpip install -U numpy


run.shpythonsentiment.py "$@”

sentiment.py

fromnltk.sentiment.vader importSentimentIntensityAnalyzerimportsyssid =SentimentIntensityAnalyzer()ss =sid.polarity_scores(sys.argv[1])print('Compound{0}Negative{1}Neutral{2}Positive{3}'.format(

ss['compound'],ss['neg'],ss['neu'],ss['pos']))

Local Sentiment Analysis via Python


ApacheOpenNLP forEntityResolutionProcessorhttps://github.com/tspannhw/nifi-nlp-processor

RequiresinstallationofNARandApacheOpenNLP BINs

Thisisanon-supportedprocessorthatIwroteandputintothecommunity.

Installing Apache OpenNLP NiFi Processor

https://community.hortonworks.com/articles/80418/open-nlp-example-apache-nifi-processor.html


StanfordCoreNLP Processorhttps://github.com/tspannhw/nifi-corenlp-processor

RequiresinstallofNARandStanfordEnglishModelshttp://nlp.stanford.edu/software/stanford-english-corenlp-2017-06-09-models.jar

Thisisanon-supportedprocessorthatIwroteandputintothecommunity.

Installing Stanford CoreNLP Processor

https://community.hortonworks.com/articles/81270/adding-stanford-corenlp-to-big-data-pipelines-apac-1.html


Contact:

TimothySpann

@PaaSDeV

http://www.meetup.com/futureofdata-princeton

https://dzone.com/users/297029/bunkertor.html

https://github.com/tspannhw/dws2017sydney/blob/master/README.md

http://community.hortonworks.com/users/9304/tspann.html


Ã https://community.hortonworks.com/content/kbentry/116803/building-a-custom-processor-in-apache-nifi-12-for.html

Ã https://community.hortonworks.com/articles/118132/minifi-capturing-converting-tensorflow-inception-t.html

Ã https://community.hortonworks.com/articles/73833/an-example-websocket-application-in-apache-nifi-11.html

Ã https://community.hortonworks.com/articles/81694/extracttext-nifi-custom-processor-powered-by-apach.html

Ã https://community.hortonworks.com/articles/79842/ingesting-osquery-into-apache-phoenix-using-apache.html

Ã https://community.hortonworks.com/articles/67980/using-command-line-security-tools-from-apache-nifi.html

Ã https://community.hortonworks.com/articles/52415/processing-social-media-feeds-in-stream-with-apach.html

Ã https://community.hortonworks.com/articles/121916/controlling-big-data-flows-with-gestures-minifi-ni.html

Ã https://community.hortonworks.com/articles/86570/hosting-and-ingesting-data-from-web-pages-desktop.html

Ã https://community.hortonworks.com/articles/63228/monitoring-your-containers-with-sysdig-from-hdf-20.html

Ã https://community.hortonworks.com/articles/101679/iot-ingesting-gps-data-from-raspberry-pi-zero-wire.html

Ã https://community.hortonworks.com/articles/101904/part-2-iot-augmenting-gps-data-with-weather.html


Ã https://community.hortonworks.com/articles/110475/ingesting-sensor-data-from-raspberry-pis-running-r.html

Ã https://community.hortonworks.com/articles/76240/using-opennlp-for-identifying-names-from-text.html

Ã https://community.hortonworks.com/articles/76935/using-sentiment-analysis-and-nlp-tools-with-hdp-25.html


Ã https://community.hortonworks.com/articles/76924/data-processing-pipeline-parsing-pdfs-and-identify.html

Ã https://community.hortonworks.com/articles/80339/iot-capturing-photos-and-analyzing-the-image-with.htmlh

Ã ttps://community.hortonworks.com/articles/122077/ingesting-csv-data-and-pushing-it-as-avro-to-kafka.html

Ã https://community.hortonworks.com/content/kbentry/116803/building-a-custom-processor-in-apache-nifi-12-for.html

Ã https://github.com/tspannhw/nifi-tensorflow-processor

Ã https://community.hortonworks.com/articles/118148/creating-wordclouds-from-dataflows-with-apache-nif.html

Ã https://community.hortonworks.com/articles/118132/minifi-capturing-converting-tensorflow-inception-t.html

Ã https://community.hortonworks.com/articles/110469/simple-backup-and-restore-of-hdfs-data-via-hdf-30.html

Ã https://github.com/tspannhw/rpi-rainbowhat

Ã https://community.hortonworks.com/articles/110475/ingesting-sensor-data-from-raspberry-pis-running-r.html

Ã https://community.hortonworks.com/articles/108718/ingesting-rdbms-data-as-new-tables-arrive-automagi.html

Ã https://community.hortonworks.com/articles/108947/minifi-for-ble-bluetooth-low-energy-beacon-data-in.html

Ã https://community.hortonworks.com/articles/108966/minifi-for-sensor-data-ingest-from-devices.html

Ã https://github.com/tspannhw/rpi-sensehat-minifi-python

Ã https://community.hortonworks.com/articles/107379/minifi-for-image-capture-and-ingestion-from-raspbe.html

Ã https://community.hortonworks.com/articles/104255/ingesting-and-testing-jms-data-with-nifi.html

Ã https://community.hortonworks.com/articles/103863/using-an-asus-tinkerboard-with-tensorflow-and-pyth.html


Ã https://community.hortonworks.com/articles/104226/simple-backups-of-hadoop-with-apache-nifi-12.html


Ã https://community.hortonworks.com/articles/101679/iot-ingesting-gps-data-from-raspberry-pi-zero-wire.html

Ã https://community.hortonworks.com/articles/99861/ingesting-ibeacon-data-via-ble-to-mqtt-wifi-gatewa.html

Ã https://community.hortonworks.com/articles/92345/store-a-flow-to-disk-and-then-reserialize-it-to-co.html

Ã https://community.hortonworks.com/articles/92495/monitor-apache-nifi-with-apache-nifi.html

Ã https://community.hortonworks.com/articles/92496/qadcdc-our-how-to-ingest-some-database-tables-to-h.html

Ã https://community.hortonworks.com/articles/89455/ingesting-gps-data-from-onion-omega2-devices-with.html

Ã https://community.hortonworks.com/articles/87397/steganography-with-apache-nifi-1.html

Ã https://community.hortonworks.com/articles/87632/ingesting-sql-server-tables-into-hive-via-apache-n.html

Ã https://community.hortonworks.com/articles/88404/adding-and-using-hplsql-and-hivemall-with-hive-mac.html


HortonworksCommunityConnection

Read access for everyone, join to participate and be recognized

• FullQ&APlatform(likeStackOverflow)

• KnowledgeBaseArticles

• CodeSamplesandRepositories


CommunityEngagement

Participate now at: community.hortonworks.com©HortonworksInc.2011– 2015.AllRightsReserved

4,000+RegisteredUsers

10,000+Answers

15,000+TechnicalAssets

One Website!

Technology

REAL-TIME INGESTING AND TRANSFORMING SENSOR DATA & SOCIAL DATA w/ NIFI + TENSORFLOW